How Many GPUs Do You Need to Serve an LLM?

You have a model and some serving constraints to meet. How many GPUs does it take to serve? You can answer that on a napkin, using numbers you already know. It comes down to two questions.

Does it fit? The weights and the KV cache have to live in GPU memory. This is the easy one, and where most people stop.

Can it keep up? The GPUs have to push tokens fast enough to hit your rate without missing the latency target. This is where the GPU count usually comes from.

The Answer in One Line

Call the three bills memory, prefill, and decode. The model has to fit, and it has to keep up:

For the running example below, Qwen3-8B on H200s at 2 req/s with 4K input and 1K output, the bills come out to about 0.7 cards to fit, 0.3 for prefill, and 0.5 for decode. That rounds to one H200. Push the same workload to 20 req/s and it becomes about eight.

What You Need

You need three groups of inputs:

• Model: total parameters $P$, active parameters $P_a$, precision, and attention architecture.
• Hardware: memory per GPU $M_\text{gpu}$, bandwidth $\text{BW}$, and peak compute $F$.
• Demand: request rate $R$, input length $L_\text{in}$, output length $L_\text{out}$, and TPOT target.

$P_a$ is equal to $P$ for a dense model. For an MoE, $P$ is the total resident parameter count, while $P_a$ is the active parameter count per token. Precision sets bytes per parameter: 2 for bf16, which we use throughout, or 1 for fp8.

For hardware, an H200 has 141 GB of HBM, 4.8 TB/s of bandwidth, and about 990 dense bf16 TFLOPS ^[1]. The datasheet's larger bf16 number assumes sparsity; for dense weights, use half.

TPOT means time per output token. In the examples below, the decode SLO is 50 ms/token. If the model writes 1,000 output tokens, that is a 50 second decode window.

The running anchor is the case I'll carry through the memory, prefill, and decode bills, before varying the parameters at the end:

Two Questions, Three Resources

The two questions turn into three resource bills. Does it fit? is about capacity: the weights and the live KV cache have to sit in GPU memory at the same time. Can it keep up? is about throughput, but generation does not have a single bottleneck from start to finish.

Prefill reads the prompt all at once, runs large matrix multiplies, and is compute-bound ^[8]. Decode writes one token at a time. Serving engines batch many streams together, but each decode step still streams weights and KV cache from HBM, so decode is bandwidth-bound ^[10]. That split is why the final count has one memory bill and two throughput bills.

Step 1: Does It Fit?

Capacity has two parts: the weights, which are fixed, and the KV cache, which grows with the number of live tokens. Start with the fixed part.

The Weights

The weights are the easy part. Multiply parameters by bytes per parameter:

Here, $b$ is bytes per parameter. Qwen3-8B is 16 GB at bf16, and the 30B-A3B MoE is 60 GB. That MoE number uses the total parameter count, not the 3B active count, because every expert has to be resident even though a given token routes through only a few.

The KV Cache

The KV cache is the term you cannot read off the model's parameter count. During generation, each token's keys and values are stored so later tokens can attend to them. The bytes per token depend on the attention design: number of layers, KV heads, head dimension, and precision.

For standard grouped-query attention ^[2], the expression includes both keys and values. The per-token cache is:

The leading 2 is for keys and values. So yes, the KV-per-token number can be calculated, but you need architectural details: layer count, number of KV heads, head dimension, and dtype. If the model uses a hybrid or compressed attention scheme, use its actual cache layout instead of this standard-GQA formula.

Qwen3's dense line is a clean example: 8 KV heads and head dim 128 across sizes. That is 4 KB per layer at bf16 ^[6]. The cache is then 4 KB times the layer count. The 8B has 36 layers, so 144 KB per token; the 0.6B has 28 layers, so 112 KB. Despite a more than 10x parameter gap, their KV caches are in the same ballpark.

However, nobody keeps layer counts in their head, so a rule of thumb helps. For standard dense GQA at bf16, the table below gives a ceiling; use it for any model whose attention you have not checked.

The table is per token; multiply it by average sequence length and requests in flight to get total KV bytes in memory ^[4]:

Concurrency comes from Little's Law: requests in flight equal rate times latency. For generation, napkin latency is output tokens times TPOT; add prefill and queueing if those are meaningful fractions of your SLO:

For the anchor, TPOT is 50 ms and $L_\text{out}=1{,}000$, so each request occupies the system for about 50 seconds. At 2 req/s, Little's Law gives 100 requests in flight. The average sequence length during decode is:

Qwen3-8B has 16 GB of bf16 weights and a standard-GQA cache of 144 KB/token, so:

The memory you need to fit is weights plus KV cache, with about 15% extra for CUDA context, fragmentation, and activation buffers:

Dividing by 141 GB of H200 memory gives the fit bill:

Step 2a: Prefill Is a Compute Bill

Prefill reads the prompt. Every input token is processed together in one large matrix-multiply batch, so compute units stay saturated and the phase is compute-bound ^[8]. That makes the GPU bill a FLOP budget: demand is rate times tokens-per-request times FLOPs-per-token, and supply is MFU times peak compute $F$, with MFU $\approx 0.5$ ^[8].

The per-token cost $C_\text{in}$ has two terms. The weight term ($2P_a$) is one multiply-add per active parameter. The attention term ($4\,n_\text{layers}\,d_\text{hidden}\,L_\text{in}$) comes from scoring each query against every prior key and summing weighted values across all layers:

For Qwen3-8B ($n_\text{layers}=36$, $d_\text{hidden}=4{,}096$), the weight term is 16B FLOPs/token and the attention term adds 2.4B at a 4K prompt:

Below $L_\text{in} \approx 6\,d_\text{hidden}$ the weight term dominates and $C_\text{in} \approx 2P_a$; above it, attention takes over and total prefill compute scales as $L_\text{in}^2$. That crossover is roughly 6K tokens for a 0.6B model, up to 50K for 72B.

Step 2b: Decode Is a Bandwidth Bill

Decode writes the answer one token at a time. Each output token depends on the one before it, so a single request cannot be parallelised across time. Serving systems recover parallelism by batching requests instead, so one decode step advances $B$ active streams by one token.

Each decode step streams bytes from HBM. It reads the weights once across the batch, and it reads the KV cache once per stream. The arithmetic is small compared to this traffic, so decode is usually bandwidth-bound ^[10].

A single GPU has usable bandwidth $\text{MBU}\cdot\text{BW}$, where MBU is model-bandwidth utilization. We use MBU $\approx 0.6$ ^[9]. But the latency target has first claim on that bandwidth: to produce one token every $t$ seconds, the card has to read $W$ bytes of weights, consuming $W/t$ bandwidth. Subtract that fixed weight read from usable bandwidth to get what remains for KV traffic:

With request rate $R$, output length $L_\text{out}$, average context $\bar n = L_\text{in} + L_\text{out}/2$, and per-token cache $M_\text{kv/tok}$, the decode bill is:

For Qwen3-8B, $W=16\text{ GB}$ and $\bar n=4{,}500$. On an H200, usable bandwidth is $0.6 \times 4.8\text{ TB/s}$, and the weight read consumes $16\text{ GB}/50\text{ ms}$:

If $W/t$ approaches $\text{MBU}\cdot\text{BW}$, the GPU cuont explodes. We cannot hit the TPOT target without splitting the model across GPUs to shrink the per-GPU weight read (and each split adding all-reduce network overhead per step).

MoE models trip people up here. A 30B-A3B serves like a 3B model in prefill because only 3B parameters activate per token. Decode does not get the same discount. It is bandwidth-bound: what matters is bytes streamed from HBM per step, not FLOPs. All experts have to live in memory, and under load different requests pull different expert subsets simultaneously. By the time the batch is served, nearly the full pool has been read.

Step 3: Combine the Bills

Now take the larger of the memory bill and the throughput bill.

For our example, Qwen3-8B at 2 req/s, 4K input, 1K output, and 50 ms TPOT is a one-card H200 job before operational margin. The three bills collapse to:

The workload shape determines which compute term leads. Long prompts and short answers, like RAG, coding, multi-turn chat, and agents, push prefill up. Long outputs push $\bar n$ up and shift the cost into decode and the KV cache, as in reasoning models and long-form generation.

Run the Napkin

If you want to adapt this to your own workload, start here. The script below uses the same formulas as the walkthrough. Change the constants at the top for your model, hardware, and traffic shape; it prints the memory, prefill, decode, and final bills separately, not just the rounded GPU count.

Turning the Knobs

We demonstrated the mechanics through the working example. Now we change one input at a time and watch which bill moves. Each case below is one constant changed at the top of the script.

Raise the rate. Keep the dense 8B and the 4K-in, 1K-out shape, but push demand from 2 to 20 requests/sec. The load-dependent terms scale almost linearly: prefill grows to about 3 H200s, decode to about 5, and the KV-heavy fit floor to about 6.

Change the shape. Back to 2 requests/sec on the dense 8B, but vary the request shape. With 16K-token prompts and 1K-token answers, prefill rises because the prompt is longer, and decode rises because the average context is now 16.5K tokens. The bills are about 2 H200s for fit, 1.6 for prefill, and 1.9 for decode, so the answer is roughly four cards.

With 8K-token prompts and 8K-token answers, prefill is still under one H200, but decode jumps to about 11. Every one of the 8,000 output tokens rereads a cache averaging 12K tokens. The resident KV cache also grows to roughly the same size, so the answer is about 12 cards, with decode and KV cache as the binding terms.

Swap the model. Return to the original 2 requests/sec, 4K-in, 1K-out workload, but trade the dense 8B for Qwen3-30B-A3B. The MoE moves two numbers in opposite directions.

For prefill, 4K tokens sit well below the attention crossover for this model, so $C_\text{in} \approx 2 \times P_a = 6\text{B}$:

Decode does not get that discount. Under load the batch touches nearly the full expert pool, so the weight-read term uses all 60 GB:

The MoE buys cheap prefill, not a cheap decode budget. That pays off most for multi-turn chat, where each turn re-sends the full transcript: $L_\text{in}$ climbs with the conversation while $L_\text{out}$ per turn stays short.

The cases line up like this; the bottleneck changes with traffic shape:

A Floor, Not a Target

You now have an answer to both questions: the weights and KV cache fit on $G_\text{fit}$ cards; throughput demands at least $G_\text{prefill}$ plus $G_\text{decode}$ cards. Treat that result as a floor, not a target.

Real deployments usually land 20 to 30 percent higher. MFU sags under load, prefill and decode compete for bandwidth on shared cards, and tensor parallelism eats into both. The napkin gets the order of magnitude and names the bottleneck; a benchmark against your actual traffic tightens the number before you commit hardware.

References

1. NVIDIA. H200 Tensor Core GPU. Datasheet, 2024. 141 GB HBM3e, 4.8 TB/s. The datasheet quotes 1,979 bf16 TFLOPS, but that number assumes 2:4 sparsity; dense weights run at half, so we use ~990 TFLOPS throughout.
2. Ainslie et al. GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints. arXiv:2305.13245, 2023. EMNLP 2023.
3. Gated DeltaNet (Yang et al., ICLR 2025), the hybrid attention in Qwen3.5, and multi-head latent attention (DeepSeek-V2), the compressed-cache design. Both shrink the KV cache below standard attention, never above it.
4. Kwon et al. Efficient Memory Management for Large Language Model Serving with PagedAttention. arXiv:2309.06180, 2023. SOSP 2023.
5. Williams, Waterman, and Patterson. Roofline: An Insightful Visual Performance Model for Floating-Point Programs and Multicore Architectures. Communications of the ACM, 2009.
6. Qwen Team. Qwen3 and Qwen3.5 model cards. HuggingFace, 2025–2026. Layer counts, KV-head, and head-dimension values for the 0.6B/8B/32B dense models and the Qwen3.5 hybrids.
7. Kaplan et al. Scaling Laws for Neural Language Models. arXiv:2001.08361, 2020. The $\approx 2N$ forward FLOPs per token (Eq. 2.2, Table 1), and the condition $n_\text{ctx} < 12\,d_\text{hidden}$ for the attention term to stay negligible.
8. Pope et al. Efficiently Scaling Transformer Inference. arXiv:2211.05102, MLSys 2023. Prefill compute-bound vs. decode memory-bound; the per-step cost $(\text{params} + \text{batch}\times\text{KV})/\text{bandwidth}$; weights amortize across the batch, KV caches do not.
9. Databricks. LLM Inference Performance Engineering: Best Practices. Blog, 2023. Defines Model Bandwidth Utilization (MBU) and the memory-bandwidth limit on decode.
10. Kipply Chen. Transformer Inference Arithmetic. 2022. Decode is memory-bandwidth-bound: the GPU re-reads all weights ($\approx 2P/\text{BW}$ per token) for each step, while prefill is compute-bound; the two meet at the flops-to-bandwidth ratio (batch $\approx$ 208 on an A100).