Throughputmaxxing: DeepSeek-V4-Flash on Isambard-AI

It turns out inference optimization is still hard for agents:

Across 15 frontier agent configurations, agents reliably improve over a naïve PyTorch baseline (up to 8.08×) and often match or exceed serving engines with default settings (4.05× for vLLM), but still fall below a simple hyperparameter search under the same time budget (up to 11.53×) From this recent paper.

Given that novel maths discoveries are not so hard, we probably shouldn’t assume this will last forever.

This article then is an offering to our coming AI overlords: one last echo of the age of centaurs, before the horses gallop off beyond the horizon into fully automated research.

Doubleword was recently named as one of six companies in the first wave of UK Sovereign AI investmentsOur first investments, UK Sovereign AI., which has given us access to Isambard-AI, the UK’s national AI supercomputing facility.

This post is about how to use a single node of that capacity to push DeepSeek-V4-Flash to about 3× what vLLM gives you off the shelf.

What do we have

Each node in Isambard-AI consists of 4xGH200 chips. Each Hopper chip is equivalent in performance to an H100Not, as you might think, an H200, although there is a GH200 SKU whose GPU is equivalent to an H200., but we have a bit more VRAM (about 96GiB). The GPUs are connected with NVLink 4 P2P connections (no switch), with (on paper) 150GB/s of bandwidth peer to peer. GPUs are connected to the LPDDR5x on the host with NVLink C2C, which interestingly on this machine, is actually higher bandwidth than the connection between GPUs.

A first cut

First, make it work.

We’re focused on throughput at all costs. There’s no vLLM recipe for 4xGH200s, so we pick the shape that vLLM suggests for 8xH200s, tweaked for our node shape, we’ll change it later.

We want tool calling and reasoning to be parsed out properly. DeepSeek has a native FP8 KV cacheDeepSeek-V4 technical report: "Both paths use FP8 storage for most KV entries and BF16 only for the RoPE dimensions." We'll see that KV cache size is a big factor when, as we're doing here, we're optimizing for throughput., so enable that. --numa-bind is important on multi-socket chips like the ones on IsambardThe node is 4 GH200 Grace-Hopper pairs, with all 4 Grace chips connected via a CPU-side interconnect and appearing as a single multi-socket CPU. If we don't bind the engines to their NUMA node, then comms between the host and device go over the interconnect and we lose throughput..

vllm serve deepseek-ai/DeepSeek-V4-Flash \
  --trust-remote-code \
  --tensor-parallel-size 4 \
  --enable-expert-parallel \
  --kv-cache-dtype fp8 \
  --block-size 256 \
  --gpu-memory-utilization 0.92 \
  --max-model-len 128000 \
  --max-num-batched-tokens 8192 \
  --max-num-seqs 2048 \
  --tokenizer-mode deepseek_v4 \
  --tool-call-parser deepseek_v4 \
  --enable-auto-tool-choice \
  --reasoning-parser deepseek_v4 \
  --numa-bind

Benchmark is vllm bench serve, completions, random ISL/OSL = 1024/1024, request rate = inf.

vllm bench serve \
  --backend openai \
  --model deepseek-ai/DeepSeek-V4-Flash \
  --endpoint /v1/completions \
  --dataset-name random \
  --num-prompts <2 × conc> \
  --request-rate inf \
  --max-concurrency <conc> \
  --random-input-len 1024 \
  --random-output-len 1024 \
  --ignore-eos

Concurrency is a free parameter: we want it as high as possible. To reliably determine the maximum throughput that the model can serve, we sweep the concurrency up until the KV cache saturates. For this setup, we get to concurrency $2048$.

The run completes 4,096/4,096 with the KV cache pinned at 100%An aside: agents find this terrifying, and think that the KV cache being full will cause the engine to crash. Why?. 5,856 output tokens/second, 1,464 per GPU.

Changing the shape: Data Parallel Attention

Tensor parallelism is the wrong shape for attention on DeepSeek models.

The reason is Multi-head Latent AttentionDeepSeek-V3 technical report. For some intuition, Jamie's post on tensor network attention., original to DeepSeek, an attention variant that reduces the amount of stored KV cache by compressing the KV heads into a single shared “latent” vector.

The problem is that for Tensor Parallel Attention, the head dimension is the useful dimension to shard over — we usually compute different attention heads on different accelerators. With only one shared K/V latent, naive tensor parallelism for MLA has to replicate the KV cache on each accelerator. The KV cache then has to live in N copies, where N is the parallelism degree.

For DeepSeek, the cleaner shape is data-parallel attention. If you replicate your attention layers on each accelerator, then each can have its own KV cache. One nice side effect: every per-token kernel (elementwise ops, FP8 quant, sampling) now runs on 1/N of the batch per rank, instead of the full global batch on every rank as it would under TP.

The drawback is that the weights get replicated, which claws back a little KV cache. But for high-throughput or long-context inference, for models with relatively small attention weights, that’s much better than replicating the KV cache $N$ times.

The diff, and the results:

- --tensor-parallel-size 4
+ --tensor-parallel-size 1
+ --data-parallel-size 4
- --gpu-memory-utilization 0.92
+ --gpu-memory-utilization 0.95
- --max-num-seqs 2048
+ --max-num-seqs 1024   # per DP rank — raise the cap + graphs to reach the KV knee
+ --compilation-config '{"max_cudagraph_capture_size":1024}'

12,802 output tokens/second at c2752, 3,201 per GPU — 2.2× the baselineMaximum concurrency (that saturates the KV cache) should always give higher throughput, all else being equal. In practice, this often ends up not being the case, because all else is not equal..

A pass through the model

Once the model’s been hammered into shape, the process of inference optimization looks like a back and forth between reasoning, profiling, optimizing, and repeating.

Let’s take a first shot. We picked a benchmark with ISL=1024 and OSL=1024. Because prefill is so parallelisable, the time taken is going to be dominated by decode: that is, generating 1024 output tokens, one by one. So we ought to focus on that steady decode state: wherein all the sequences have finished prefilling, and we’re just working on $N$ sequences in a decode batch.

If you hit the server with the torch profiler, you see that the model’s forward pass is dominated by the MoE expert GEMMs.

Which points at an obvious target. The MoE expert weights — most of the model’s 284B parameters — ship in fp4. If we were running Blackwell GPUs, this would be really nice: these would run natively on the fp4 tensor cores. On the Hopper generation, with only fp8, we’ve got to do it in softwareOr just upcast them to fp8 in VRAM, but that would be a waste of VRAM..

vLLM’s default on Hopper is conservative about how it does this: it picks Marlin, a weight-only kernel that dequantizes the fp4 weights to bf16 and runs the GEMMs on the GPU’s bf16 tensor cores. On Hopper, we’ve got fp8 tensor cores. Can we do fp8?

The fear we should have is correctness: providers offering models to the public ought to want the model’s output to match what it was trained to be, not just some vibe-eval hand-wave. DeepSeek’s technical report is pretty specific about the correct arithmetic for those weights: during quantization-aware training, the FP4 weights are dequantized (losslessly) back to FP8 and the forward pass runs through DeepSeek’s standard FP8 pipeline — e4m3 activations, scaled per token per 128 elementsDeepSeek-V4 technical report, §5.2.1: master weights are "first quantized to FP4, then dequantized back to FP8 for computation", reusing "the existing FP8 training framework" — the DeepSeek-V3 recipe of 1×128 activation tiles with FP32 scales..

So there’s no correctness reason to use Marlin. Is there a performance reason? It depends on the batch size. On the datasheet, the bf16 ridge point on the GH200 arrives at ~250 FLOPs per byte: 990 TF/s of bf16 against 4 TB/s of HBM. But you should always check this stuff. If you run a high intensity kernel on Isambard, you’ll see the clocks come down once the GPU draws ~530–570 W, and sustained bf16 tops out at 583 TF/s — nowhere near the datasheet number. This is a software power cap, not a thermal limit.

So the roofline for bf16 is actually ~146 FLOP/byte. From some pen & paper, this means we can have a ‘critical’ batch size of $1{,}300$. Below that, the bf16 GEMMs are memory bound, and Marlin is as good as anything else. Our batch sizes are larger than $1{,}300$. So we have a very rare thing: a MoE model that is compute bound at decode time.

The HummingOne more property worth having: Humming's FP8 path runs with batch-invariant execution and Marlin's doesn't. kernels from the Ant group are the W4A8 kernels that we’re looking for: they upcast to fp8 in the SMs, and then do fp8xfp8 matmuls in the fp8 tensor cores. The weight transfer stays the same, but the available FLOPs doubles, and the critical batch size doubles with it. Out of the box, didn’t work here, so we had to do some work on the kernels to bring them up, and then to get the maximum performance out of themTo get these running in vLLM took a kernel bug fix and a tuning fix, both in Humming's group-scaled path: the WGMMA consumer never applied the per-128 input scales on the accumulator, and once we fixed that, the extra accumulator it needs halves the per-tile register budget, so the tile heuristic's default BlockM spills catastrophically. Once you fix the scales and make the heuristic better, the performance recovers. Fork: doublewordai/humming..

+ --moe-backend humming
+ VLLM_HUMMING_MOE_GEMM_TYPE=indexed
+ VLLM_HUMMING_INPUT_QUANT_CONFIG='{"dtype":"float8e4m3","input_scale_group_size":128}'

The results:

We speed up our DP attention baseline by about 40%. Not bad!

Where does the time go?

There are many more steps along this path. Before closing, let’s take stock. The profiler breaks down a decode step:

The breakdown of the forward pass, created by parsing the output of the torch profiler at steady-state decode (per rank per step, at c2048) and mapping the kernels to their inferred roofline ceilings. All on the same scale, grayed out means current, solid means the minimal achievable time.

There’s lots of work to do! Even with our fixed-up GEMM kernels, the MoE bucket is still at 30% of its roof. Comms costs way more than it ought to: 10.8 ms against ~3ms at the wire rate. We’re wasting time on overhead everywhere: the act+quant bucket is ~520 tiny kernel launches per step, of which ~2 ms is actual data movement.

A lot more exciting stuff comes onto the table as we scale out beyond a single node. Disaggregated inference (where prefill runs on one node and decode on another) makes this kind of kernel optimization much easier, since you can hammer each phase in isolation. Isambard has an HPE slingshot fabric: over which we’ve built out UCCL-P2P (and therefore NIXL) support. And, wide EP over that fabric pushes down the weights stored per node, and pushes up the available KV cache: support for DeepEP style comms over HPE slingshot also lands with a Slingshot backend for UCCL-EP.

But that’s a multi-node story. We’ll leave it for the next post.