Lesson 20: Prefill and decode as separate pools: expert and data parallelism
Lesson 19 moved the cache between two kinds of machine and took both for granted. This lesson opens Part 5 by spreading one model over many GPUs, and then explains why the GPUs that read prompts and the GPUs that generate are grouped differently.
Source: the DeepInfer deployment document (doc/inference_deployment_qa.md), its introduction and the sections "Inference system overview", "Cluster & storage", "Autoscaling & fault tolerance" and "Kernel implementation"; the DeepInfer README; the DeepSeek-V4 report (arXiv 2606.19348), section 3.1; the DeepEP README; the DeepSeek-V4.1-Flash report (arXiv 2609.19969), sections 2.1 and 4.2.1, for sizes. Every number in the worked examples comes from lesson20_check.py.
1. What lesson 19 left open
The book ran its model on one GPU for one request: it read the prompt, then generated token by token. Lesson 19 ended with machines whose 8 GPUs exchange messages while they run one model, and it did not say why they must. A count of the weights of DeepSeek-V4.1-Flash gives the reason.
one expert 3 x 5,120 x 2,304 = 35,389,440 numbers three matrices, as in lessons 6 and 10
routed experts 40 layers x 384 x 35,389,440 = 543.58B numbers 98.5 percent of the 552B backbone
in MXFP4 x 0.53125 bytes (lesson 12) = 288.8 GB
rest of backbone 552B - 543.58B = 8.42B numbers = 8.4 GB one byte each, chosen
backbone = 297.2 GB
Engram tables 196B numbers in FP8 = 196 GB
Neither DeepInfer nor the reports print the size of the weights or the memory of an accelerator, so we choose two memory sizes, 80 GB and 180 GB. The backbone is 3.71 times the first and 1.65 times the second. One accelerator of either size cannot hold it, and we have not yet counted a single cache entry (figure 20.1). A service must also run thousands of requests at once.
The document also states the scale it works at, and we can only repeat its two claims: a single cluster serves 155 trillion tokens per day, and revenue is 38 US dollars per GPU-hour at the off-peak price.

2. A toy layer and four GPUs
Our toy is one layer and the eight positions "The next day is bright and sunny and", with the hidden states of lessons 1 and 3 as X (8, 4).
attention 4 heads of one number each, 4 matrices of (4, 4) 64 weights
router E (8, 4), rows 1 to 4 from lesson 10 32
routed experts 8 experts, each 3 matrices of (4, 4), 2 per position 8 x 48 = 384
the layer 480
All weights are chosen. Four GPUs, G0 to G3, share the layer. For three of the four splits below, the eight positions are four requests of two tokens each. For the last they are one sequence. The check script computes the layer on one GPU and again under every split, and the outputs agree. Figure 20.2 draws the four splits.

3. Attempt 1: a full copy on every GPU
Data parallelism (DP) is the simplest split. Every GPU holds a complete copy of the model and takes a share of the requests. In the toy each GPU holds all 480 weights and one request, and no number crosses from one GPU to another.
This serves four times the requests and does nothing for memory, since each GPU of the real system would still need 297.2 GB. Data parallelism multiplies a model that already fits. We keep it for the parts of the model that are small.
4. Attempt 2: every matrix in slices
Tensor parallelism (TP) cuts every weight matrix into as many slices as there are GPUs. In the toy, GPU g holds head g of the attention and hidden unit g of every expert.
on each GPU 16 (attention) + 32 (router) + 96 (experts) = 144 of 480 weights
positions all 8 on every GPU
Memory is now divided. The cost is that each GPU computes only a part of every output. The four partial results (8, 4) must be added, and every GPU needs the sum before the next sublayer can start. This exchange is called an all-reduce. It runs after attention and again after the experts, and each run moves at least 2 x (4 - 1) x 32 = 192 numbers, 384 per layer. Every position crosses the network twice in every layer, and no GPU can go on until all four have reported.
5. Expert parallelism: dispatch and combine
The count in section 1 points to a better cut. The routed experts are 98.5 percent of the backbone, and a token uses 6 of the 384 in a layer. Expert parallelism (EP) therefore gives each GPU a share of the experts. Everything else is small enough to copy, so attention and the router stay data-parallel. In the toy, G0 holds experts E1 and E2, G1 holds E3 and E4, and so on: 64 + 32 + 2 x 48 = 192 weights per GPU. One GPU of such a group is called a rank, and the GPU on which a position waits is its home.
Step 1: attention and routing at home. Each GPU runs attention on its own request and scores its two positions, (2, 4) x (4, 8) โ (2, 8), with the Sqrt(Softplus) affinity of lesson 10.
Step 2: dispatch. The hidden state of each position, (1, 4), is sent to the GPUs that hold its two experts. DeepEP defines dispatch as routing tokens to their experts across all ranks.
Step 3: the experts compute. Each expert runs once on all the positions it received. E1 has five of them, (5, 4) x (4, 4) โ (5, 4).
Step 4: combine. Each output returns to the home GPU of its position, where the two outputs are added with their gates. DeepEP defines combine as reducing expert outputs back to their original ranks.
position token home experts (gate) on GPUs remote pairs
p1 The G0 E6 (0.5514) E8 (0.4486) G2, G3 2
p2 next G0 E7 (0.5416) E2 (0.4584) G3, G0 1
p3 day G1 E1 (0.5983) E6 (0.4017) G0, G2 2
p4 is G1 E3 (0.5342) E2 (0.4658) G1, G0 1
p5 bright G2 E7 (0.5140) E1 (0.4860) G3, G0 2
p6 and G2 E1 (0.5299) E4 (0.4701) G0, G1 2
p7 sunny G3 E1 (0.5325) E4 (0.4675) G0, G1 2
p8 and G3 E1 (0.5228) E4 (0.4772) G0, G1 2
positions per expert, E1 to E8 5, 2, 1, 3, 0, 2, 2, 1
pairs computed on G0 to G3 7, 4, 2, 3
Take "bright" on G2. Its state after attention is [1.5, 3, 1.5, 0]. E7 on G3 returns [-0.3115, -0.0373, 0.0454, 0.2625] and E1 on G0 returns [0.8511, -4.6008, -2.7030, -4.5674]. With the gates 0.5140 and 0.4860 the combined output is [0.2535, -2.2554, -1.2905, -2.0851].
Of the 16 position-expert pairs, 2 are local and 14 are remote. A remote pair sends 4 numbers out and 4 back, so 112 numbers cross in this layer, against 384 for tensor parallelism (figure 20.3). The traffic now depends on the routing. It is also uneven: G0 computes 7 pairs and G2 computes 2, because E1 is popular.
The V4 report counts the same trade for V4-Pro. A token-expert pair costs 6hd operations for the three matrices and 3h bytes on the network, with dispatch in FP8 and combine in BF16. That is 6,144 operations per byte sent, and the report concludes that within one MoE layer communication takes less time than computation.

6. Context parallelism
Context parallelism (CP) cuts a long sequence, where the other three cut weights or requests. Read the eight positions as one sequence with two positions per GPU. Attention at a position needs the keys and values of every earlier position, and most of those are on other GPUs.
earlier entries needed from other GPUs, G0 to G3 0, 2, 4, 6 12 entries x (4 + 4) numbers = 96
entries read by the two positions of each GPU 3, 7, 11, 15
One long prompt now uses four GPUs at once, which shortens the wait for its first token. The work is uneven, since the last piece reads five times as many entries as the first.
7. What DeepInfer does with each
| Split | A toy GPU holds | Crosses per layer in the toy | DeepInfer | Reason the document states |
|---|---|---|---|---|
| Data | all 480 weights | 0 | for attention, with as many replicas as the expert group has GPUs | none beyond following the V3 and R1 serving system; the replicas then need the routers of lesson 22 |
| Tensor | 144 | 384 | not used for now | high communication overhead; unnecessary when the model fits in memory and the latency target is met |
| Expert | 192, with the rest copied | 112 | for the MoE part, inside one machine or supernode | larger batches per expert and more efficient matrix products |
| Context | 480 | 96 | not used for now | up to 5 percent slower than DP end to end; V4.1 keeps the first token fast enough even on a complete cache miss; it tends to worsen the balance between experts and adds kernel, communication and CPU overhead |
A supernode is not defined in the document. From its context it is a group of 64 to 128 GPUs joined by NVLink, NVIDIA's direct links between GPUs. The README lists "balanced CP" among the engine's features and does not explain it.
8. Two batches, one cost model
Prefill is the forward pass over the prompt, and decode is the loop that generates. They bring very different batches to an MoE layer. In decode, every running sequence contributes one token per step. In prefill, one request contributes thousands of tokens at once. The following model is ours and is kept simple. A group has N GPUs, each GPU brings b tokens per step, and routing is uniform.
T = N x b tokens that reach one MoE layer in one step
n = 6 x T / 384 mean tokens per routed expert
touched = 384 x (1 - (1 - 6/384)^T) experts that receive at least one token
read = touched x 18.80 MB expert weights that must come out of memory
An expert that receives one token must read its full 18.80 MB of weights, and an expert that receives sixty reads the same bytes. For decode we take b = 60, which the document gives as its production operating point of about 60 concurrent sequences per rank.
decode, b = 60 T n touched read read per GPU read per token operations per byte
EP1 60 0.94 234.7 4.413 GB 4,413.1 MB 73.55 MB 5.8
EP8 480 7.5 383.8 7.216 GB 902.0 MB 15.03 MB 28.3
EP32 1,920 30 384 7.219 GB 225.6 MB 3.76 MB 112.9
EP64 3,840 60 384 7.219 GB 112.8 MB 1.88 MB 225.9
EP128 7,680 120 384 7.219 GB 56.4 MB 0.94 MB 451.8
prefill, b = 4,096 (chosen)
EP2 8,192 128 384 7.219 GB 3,609.7 MB 0.88 MB 481.9
EP8 32,768 512 384 7.219 GB 902.4 MB 0.22 MB 1,927.5
At EP8 a decode step already touches nearly every expert, so the group reads 7.2 GB of expert weights in this layer to compute 480 tokens. At EP64 the same bytes serve 3,840 tokens. The weights read per token fall from 15.03 MB to 1.88 MB, a factor of 8, and tokens per gigabyte read rise from 66.5 to 531.9 (figure 20.4). Over the 40 layers, one GPU reads 36.08 GB per step at EP8 and 4.51 GB at EP64, and in both cases it produces 60 tokens. A larger b would raise n as well, but b is each GPU's own attention work and each user's wait. A larger N raises n without adding to either.
Dispatch and combine are the price. With the V4 report's formats, a remote pair of V4.1-Flash moves 3 x 5,120 = 15,360 bytes. At EP64 that is 90,720 bytes per token and layer, 217.7 MB per GPU and step, against the 4.51 GB of weights.
Prefill does not need the width. With 4,096 prompt tokens per GPU, a number we chose, EP2 already gives each expert 128 tokens, more than decode reaches at EP128. Widening the group would only raise the share of remote pairs, from 0.5 at EP2 to 0.875 at EP8.

9. The document's own reasons
The document gives its reasons in a few sentences. We place them beside our model without claiming that the two are the same argument.
For prefill it says to use the smallest EP and DP size that accelerator memory (HBM) allows, from 2 to 8. Its reason is that prefill, unlike decode, already has a large enough batch per expert, and that smaller groups improve both kinds of load balance and reduce NVLink traffic. Our count in figure 20.1 fits the range. With 180 GB, EP2 holds the weights and leaves 27.2 GB. With 80 GB, EP4 needs 80.6 GB and does not fit, and EP8 needs 44.5 GB.
For decode it says to use the largest size the available GPUs allow, from 64 to 128, and calls this Wide-EP. Its reason is that Wide-EP substantially reduces the memory traffic of each step in the expert part and, with good balancing between experts, raises tokens per second. It shows one chart of EP8 against EP64 and prints no numbers.
A kernel remark sets a limit. Raising EP increases arithmetic intensity, the operations per byte read from memory. Once memory bandwidth no longer limits the step, the document says, a larger group only adds communication and synchronization, and gains may stop beyond EP32. Our model has no term for either, so it cannot locate that point.
10. Why two pools
One group cannot have both layouts, since a GPU holds 192 experts per layer at EP2 and 6 at EP64. Timing gives a second reason. Suppose one GPU decodes for several users, one step per time unit, and at time 4 a request arrives whose prefill takes 12 units. All units are chosen.
tokens reach the running users at longest wait new request: first token
one GPU, prefill runs whole 1 2 3 4 17 18 19 20 13 at 17
one GPU, prefill in chunks of 3 1 2 3 4 8 12 16 20 4 at 20
two pools, hand-off of 1 unit 1 2 3 4 5 6 7 8 ... 1 at 18
On a shared GPU the running users wait 13 units for their next token. Cutting the prefill into chunks shortens that wait to 4 and delays the new request by 3, so the two waits are traded against each other. With separate pools neither workload sees the other (figure 20.5). The document states this purpose: disaggregation decouples the resources of each stage, so that time to first token and time between tokens can be controlled independently.
The cost is the hand-off. What moves from the prefill pool to the decode pool is the cache of lessons 15 and 17, the blocks and the window state, which the document says share one transfer path. Lesson 16 stored those blocks, and lesson 19 showed two routes by which they reach a decode machine.

11. The vision encoder and Engram
V4.1-Flash reads images through a vision encoder of 32 layers with hidden size 1,024. It is small beside the backbone, and the document makes it a third pool: one GPU per instance is generally enough, and cheaper GPUs are recommended. The document writes E/P/D for the three stages, and PE and CE for the prefill and decode engines. Its CE is lesson 19's DE.
Engram (lesson 9) gets a parallel group of its own, called ENP and sharded along the vocabulary. Prefill uses ENP8, and decode uses an ENP equal to its EP size. On GPUs each rank keeps its shard in host memory and reads rows over RDMA at the start of a forward pass.
12. The price of a wide group
All GPUs of an instance form one communication domain. The document says that a single failed GPU makes the whole instance unusable. We choose a failure probability of 0.001 per GPU and day.
no failure in a day at least one mean days between failures sequences stopped at 60 per GPU
EP4 0.999^4 = 0.99601 0.00399 250 240
EP64 0.999^64 = 0.93797 0.06203 15.6 3,840
64 GPUs, 0.064 failures a day, restart of 10 minutes (chosen)
16 instances of EP4 4 GPUs stop 2.56 GPU-minutes a day 0.0028 percent of capacity
1 instance of EP64 64 GPUs stop 40.96 GPU-minutes a day 0.0444 percent of capacity
The same 64 GPUs fail equally often in both layouts. A failure stops 16 times as many GPUs in the wide one, and 3,840 running sequences where the narrow one would lose 240 (figure 20.6). In lost capacity both numbers are small.
The document handles this in three ways. On restart, the platform determines the EP size again from the GPUs that are healthy, and the instance runs with what it receives. Restarts take minutes, so it relies on spare quota and on rerouting requests, and scale-in is forbidden while quota utilization is above 0.8. Long requests are handled at planned shutdowns: a decode engine gets a grace period of one minute, after which its unfinished requests and their cache are forwarded in one batch to other engines. What happens to the requests of an instance that fails is not stated.

13. The three pools in one table
| Vision encoder | Prefill | Decode | |
|---|---|---|---|
| What runs | image patches to vectors | the non-cached part of a prompt: the encoder layers, and the last 128 tokens through the decoder layers (lesson 17) | one step per token, all 40 layers |
| Parallelism | one GPU per instance | EP2 to 8 with DP of the same size, the smallest that memory allows; ENP8 | EP64 to 128 with DP of the same size, the largest the GPUs allow; ENP of the same size |
| Hardware named | cheaper GPUs | ordinary 8-GPU machines | supernodes of 64 to 128 GPUs |
| Scales on | not stated | the number of non-cached tokens | the number of running requests |
| In GPU memory | the encoder | shared parts and 1/N of the experts: 152.8 GB at EP2, 44.5 GB at EP8 (our count); a paged KV cache | 12.9 GB of weights at EP64 (our count); the cache of about 60 sequences |
The document adds that the GPU ratio between prefill and decode is roughly 1:1. One prefill and one decode instance can be deployed, and since decode should run at EP64 or more, the smallest efficient unit is 64 prefill GPUs plus 64 decode GPUs. Storage is a fourth kind of node, the 3FS servers of lesson 16, at about 14.4 TB of logical capacity per GPU (figure 20.7).

14. Dispatch and combine in code
The listing is the expert part of section 5. route and swiglu are lesson 10's.
def moe_expert_parallel(Y, experts, home, owner, k=2):
idx, gate = route(Y, k) #A
out, crossed = torch.zeros_like(Y), 0
for e, weights in enumerate(experts): #B
tok, slot = (idx == e).nonzero(as_tuple=True) #C
if len(tok) == 0:
continue
y_e = swiglu(Y[tok], *weights) #D
out.index_add_(0, tok, gate[tok, slot][:, None] * y_e) #E
crossed += 2 * Y.shape[1] * int((home[tok] != owner[e]).sum()) #F
return out, crossed
#A Scores and gates are computed where each position lives: Y (8, 4) x Eแต (4, 8) โ (8, 8), top 2.
#B One pass per expert. In the real group these run at the same time on the GPUs given by owner.
#C Dispatch: the positions routed to expert e.
#D One batched product for everything the expert received.
#E Combine: each output is added at its position, weighted by its gate.
#F A pair whose expert is not on the home GPU moves d numbers out and d numbers back.
On the toy it returns the output of section 5 and 112 crossed numbers.
15. What the real system adds
| Toy | V4.1-Flash in DeepInfer | |
|---|---|---|
| Layers | 1 | 40 |
| Hidden size | 4 | 5,120 |
| Routed experts per layer, per token | 8, 2 | 384, 6, plus one shared expert |
| Weights of one expert | 48 | 35,389,440 (our count) |
| GPUs in a group | 4 | 2 to 8 for prefill, 64 to 128 for decode |
| Routed experts per GPU and layer | 2 | 192 to 48 for prefill, 6 to 3 for decode |
| Tokens per GPU and step | 2 | about 60 in decode; not stated for prefill |
| A remote pair moves | 4 numbers out, 4 back | 3h bytes in the V4 report's example |
Four neighbours of this lesson are left to later ones. Each GPU also holds a few redundant copies of popular experts, placed by the expert-parallel load balancer of lesson 21. The routers that assign requests and images to replicas are lesson 22. Asynchronous scheduling, which the document calls a zero-overhead batch scheduler, is lesson 23. Dispatch, the two products and combine run as one fused kernel, Mega MoE, which is lesson 27. V4 also removed V3's cap on how many nodes the experts of one token may lie on, which lesson 10 mentioned.
The README notes that the released engine does not include the disaggregated deployment, the KV cache system or the replica load balancers. Its benchmarks ran on 4 GPUs for prefill and 64 for decode, each inside one scale-up domain, and are shown as charts.
Not published: the size of the weights; the memory of the accelerators; the format of the weights outside the routed experts; the numbers behind the EP8 and EP64 chart; any latency target; the tokens per prefill rank; what limits a decode group to 128; what "balanced CP" is; the size of the vision encoder and what its pool scales on; and the fate of requests on a failed instance.
16. What is still open
The toy of section 5 ended with G0 computing 7 of 16 pairs while G2 computed 2, and with an expert that received nothing. Our cost model assumed uniform routing and so hid this. Combine cannot finish a layer before the busiest GPU has run its experts, so in a wide group one overloaded GPU holds back the other 63. The document attaches a matching condition to its claim for Wide-EP: the gain comes with good balancing between experts.
Lesson 21 takes this up: expert-parallel load balancing, which decides which experts each GPU holds and which of them get a second copy.