VizuaraLearn DeepSeek by Vizuara

Curriculum  /  Part 6: Kernels

Lesson 27: The fused MoE kernel: overlapping communication with computation

Part 6: KernelsFree lessonDownload lesson27_check.py

Part 5 took the time of one decode step as given. This lesson opens Part 6 by going inside the step, to the kernel that runs one MoE layer across a group of GPUs.

Source: the DeepSeek-V4 report (arXiv 2606.19348), section 3.1 and the caption of figure 5; the DeepGEMM README, pull request 316, tests/test_mega_moe.py, impls/sm100_fp8_fp4_mega_moe.cuh and scheduler/mega_moe.cuh; the DeepEP README; the DeepInfer deployment document (doc/inference_deployment_qa.md), section "Kernel implementation"; lessons 20 and 23. Every number in the worked examples comes from lesson27_check.py.


1. Three steps in a row

Lesson 20 spread the routed experts of one layer over a group of GPUs, which is expert parallelism. The layer then ran in three steps. Dispatch sends the hidden state of each token to the GPUs that hold its experts. Each expert then computes on the rows it received. Combine sends the outputs back to the home GPU of each token, where they are added with their gates.

The toy was the eight positions of "The next day is bright and sunny and", with 8 experts on 4 GPUs and two experts per position. Of its 16 position-expert pairs, 2 are local and 14 are remote. We now put a clock on it. Every duration is chosen.

read one remote pair (dispatch, FP8)         1 tick
write one output back (combine, BF16)        2 ticks      twice the bytes of the read
compute one pair through its expert          4 ticks
start one kernel                             1 tick

A remote pair costs 3 ticks on the link and 4 on the cores. Communication is therefore the smaller part, which is what the V4 report says of its own profile. Each GPU has an incoming link, its cores and an outgoing link, and each of the three does one thing at a time. The link stands for NVLink, NVIDIA's direct connection between GPUs.

       pairs   remote   dispatch   compute   combine   done at
G0     7       6        1 + 6      1 + 28    1 + 12    49
G1     4       3        1 + 3      1 + 16    1 + 6     28
G2     2       2        1 + 2      1 + 8     1 + 4     17
G3     3       3        1 + 3      1 + 12    1 + 6     24

The layer takes 49 ticks. On G0 the cores compute for 28 of them, and for 18 they wait on a link. During the 28 ticks of computing, both links carry nothing. Over the four GPUs the cores compute for 64 of 196 GPU-ticks, or 32.7 percent (figure 27.1). The early finish of G1, G2 and G3 was the subject of lesson 21. This lesson is about the 21 ticks on G0 that are not computation.

Figure 27.1 The unfused layer on lesson 20's toy: dispatch, compute and combine in a row on each GPU, with the idle time of links and cores hatched.
Figure 27.1 The unfused layer on lesson 20's toy: dispatch, compute and combine in a row on each GPU, with the idle time of links and cores hatched.

2. Attempt 1: overlap whole phases

The simplest idea is to start a phase before the previous one has ended. The phases do not allow it, because each needs the output of the one before. The products need the dispatched rows, and combine needs the products. The next layer's dispatch needs this layer's combined output, after attention and routing, so it cannot begin while this layer computes.

What can run under a message is work that needs no message. DeepEP's README gives this pattern: "launch communication first and wait at the point where the expert computation needs its results". In the toy the only such work is the two local pairs. G0 computes its local pair while its six remote rows arrive.

G0   link in    start [0, 1]     read 6 pairs [1, 7]
     cores      start [0, 1]     local pair [1, 5]      start [7, 8]     6 remote pairs [8, 32]
     link out   start [32, 33]   write [33, 45]

The layer takes 45 ticks, a gain of the 4 ticks of one pair. The rest of the communication stays exposed, and at EP64 only one pair in 64 is local. The V4 report cites a finer scheme, Comet, which overlaps dispatch with the first matrix product and the second product with combine. We have not read Comet and only repeat the report's caption.

3. Attempt 2: one expert at a time

On G0, the five rows of E1 need nothing from the rows of E2. The second idea therefore streams the layer by expert. Each expert starts as soon as its own rows have arrived, and its outputs travel while the next expert computes.

G0   link in    E1 [1, 6]      E2 [7, 8]
     cores      E1 [7, 27]     E2 [28, 36]
     link out   E1 [28, 38]    E2 [39, 41]

The layer takes 41 ticks (figure 27.2). From tick 28 to 36 the cores and the outgoing link work together, which the first two schedules never achieved.

The cost is in the kernels. G0 now starts 6 kernels where it started 3, three for every expert. A GPU of V4-Pro at EP8 holds 48 experts of a layer, which makes 144 kernels. The products also become small. In decode at EP8, an expert of V4.1-Flash receives 7.5 rows on average. A GEMM kernel hands blocks of rows to the GPU's streaming multiprocessors (SMs), the units that hold its tensor cores, and 7 rows cannot occupy them all. One published number points the same way. With one token per rank, that is per GPU of the group, the fused kernel on V4-Flash reads memory at 1,311 GB/s. With 512 tokens it reads at 3,192 GB/s. The interpretation is ours.

Figure 27.2 G0 under attempt 1 and attempt 2: what overlaps, and how many kernels are started.
Figure 27.2 G0 under attempt 1 and attempt 2: what overlaps, and how many kernels are started.

4. Waves

The V4 report keeps the streaming and changes its unit: "we split and schedule the experts into waves. Each wave consists of a small portion of experts." A wave is a group of experts on one GPU that is received, computed and sent together. A wave computes as soon as all its experts have their tokens, without waiting for other experts. The report describes the steady state: "computation of current wave, token transfer for the next wave, and result sending of completed experts all proceed concurrently". All of this happens inside one kernel. The report does not say how many experts form a wave, or how many waves a layer has.

A toy GPU holds two experts, so here a wave is one expert, and only the kernel starts differ from attempt 2.

G0   cores      start [0, 1]    E1 [6, 26]     E2 [26, 34]
     link in    E1 [1, 6]       E2 [6, 7]
     link out   E1 [26, 36]     E2 [36, 38]
schedule                     layer time   kernels on G0   against unfused
unfused                      49           3               1.00
attempt 1                    45           4               1.09
attempt 2                    41           6               1.20
waves                        38           1               1.29
all communication hidden     29           1               1.69

Two edges stay exposed: the first read of 5 ticks, and 4 ticks of writing after the last product. The order of the waves matters as well. With E2 first the layer takes 40 ticks, and with both experts in one wave it takes 47 (figure 27.3).

Figure 27.3 G0 under waves, and the layer time of the four schedules against the time with all communication hidden.
Figure 27.3 G0 under waves, and the layer time of the four schedules against the time with all communication hidden.

5. The pipeline by hand

A pipeline splits a job into stages. A stage is a step that works on one item while the other stages work on other items. Our stages are read, compute and write, and the items are waves.

Take one GPU with 4 experts and 2 remote pairs for each, a chosen example, in two waves of two experts. A wave reads for r = 4 ticks, computes for c = 16 and writes for s = 8. Unfused, the layer takes 56 ticks.

slot   ticks       link in        cores             link out
1      0 to 4      read wave 1    idle              idle
2      4 to 20     read wave 2    compute wave 1    idle
3      20 to 36    idle           compute wave 2    write wave 1
4      36 to 44    idle           idle              write wave 2

The first wave pays for every stage. Each later wave pays only for the slowest stage.

T = r + c + s + (W − 1) × max(r, c, s)          4 + 16 + 8 + 1 × 16 = 44

Communication is hidden when the slowest stage is the computation, that is when r ≤ c and s ≤ c. The layer then costs all of the computation plus the communication of a single wave. With four waves of one expert, T is 38, against 32 for the computation alone. Smaller waves shrink the exposed edges, which is one reason to keep a wave small.

Suppose the link is three times slower, so that r = 12 and s = 24. The slowest stage is now the write. T becomes 12 + 16 + 24 + 24 = 76, and each further wave adds 24 ticks where the cores need 16 (figure 27.4). The report states the same condition for a whole layer: C/B ≤ V_comp/V_comm, where C is peak compute, B is interconnect bandwidth and V stands for the two volumes.

A wave cannot shrink for free. Suppose each wave costs 4 ticks, which stand for learning that its rows have arrived and for a wave too small to occupy every SM. On a GPU with 48 experts the speedup is then 1.24 with 2 waves, 1.53 with 16 waves of 3 experts, and 1.45 with 48 waves (figure 27.5).

Figure 27.4 One GPU with four experts in two waves, slot by slot, and the same GPU on a link three times slower.
Figure 27.4 One GPU with four experts in two waves, slot by slot, and the same GPU on a link three times slower.
Figure 27.5 The speedup on a GPU with 48 experts against the number of waves, with and without a cost per wave.
Figure 27.5 The speedup on a GPU with 48 experts against the number of waves, with and without a cost per wave.

6. Why one kernel

Fusing brings three gains beyond the overlap. The first is the starts. The baseline that DeepGEMM's test times has five steps: dispatch, the first product, SwiGLU, the second product and combine. Mega MoE is one kernel. The second is the host, the CPU process that launches kernels. Between two kernels, control returns to it. Lesson 23 removed the CPU's waits between decode steps, and fusing removes them between the phases of a layer. The third is the order of the work, which the kernel now decides itself. In the toy that order was worth 2 ticks.

The README states what is fused: "Mega MoE fuses and overlaps EP dispatch, linear 1 and linear 2 (FP8xFP4 or FP8xFP8), SwiGLU, and EP combine into a single mega-kernel, overlapping NVLink communication and tensor core computation." It requires symmetric memory, a buffer with the same layout on every GPU of the group, so that each GPU can address the copies of the others directly. The deployment document calls Mega MoE the general solution within a scale-up domain. It does not define the term. We read it as the set of GPUs joined by such direct links.

The comments in the source show the roles inside the kernel (figure 27.6). A warp is a group of 32 threads. The dispatch warps carry the comment "Pull token data and SF from remote ranks into local L1 buffer". Two warps load tokens and weights, and one issues the tensor core instructions. Epilogue warps apply SwiGLU after the first product and, after the second, "write GEMM output to remote combine buffer via NVLink". At the end the kernel waits for the other ranks' writes and reduces the top-k results into the output. Dispatch is thus a pull and combine is a write. The report's wording is that "each GPU actively reads activations from remote GPUs".

The scheduler file settles what a wave is in the code. A task is a block of token rows, the blocks lie in the order of the experts, and one wave is one round of tasks over all pairs of SMs. A product on a block waits for "the entire token arrival" of that block, which it reads from a counter. We found no constant for the number of experts in a wave.

Figure 27.6 The roles inside Mega MoE as the source comments name them, and the path of one remote pair.
Figure 27.6 The roles inside Mega MoE as the source comments name them, and the path of one remote pair.

7. The bound

Let communication take a share f of the unfused layer.

unfused        T  = communication + compute         f = communication / T
all hidden     T′ = compute = (1 − f) × T           speedup = 1 / (1 − f)

The second line requires that the communication fits under the computation, so f ≤ 0.5 and the speedup is at most 2. We can solve for the f behind each published speedup. This is our arithmetic.

speedup   f = 1 − 1 / speedup   communication / compute
1.50      0.3333                0.50
1.73      0.4220                0.73
1.96      0.4898                0.96

The report prints no ratio. It says only that within one MoE layer "the total time of communication is less than that of the computation". That puts f below 0.5 and the bound below 2, and all three published speedups respect it. Each f is an upper limit on the share of communication, because part of the gain comes from the kernel starts of section 6. On the toy, G0 has f = 18 / 46 = 0.3913 and a bound of 1.64 (figure 27.7).

8. What was measured

The report gives 1.50 to 1.73 "for general inference workloads" and up to 1.96 for latency-sensitive ones such as RL rollouts, against "strong non-fused baselines". It names no setting. The settings are in pull request 316 of DeepGEMM: EP8, values averaged over the 8 ranks.

Tokens per rank V4-Flash, time Speedup V4-Pro, time Speedup
1 56.5 µs 1.96 108.1 µs 1.61
512 146.5 µs 1.73 369.6 µs 1.54
8,192 1,283.1 µs 1.56 2,818.5 µs 1.50
32,768 4,855.5 µs 1.62 10,655.2 µs 1.54

The range 1.50 to 1.73 covers the batches of 512 tokens and more on both models. The 1.96 is V4-Flash with one token per rank. That row moves 1 GB/s over the interconnect, so it has almost no communication to hide. Its gain must come from fixed costs, and that reading is ours. At 8,192 tokens V4-Pro computes at 2,304 TFLOPS, trillions of operations per second, and moves 393 GB/s. The quotient is 5,863 operations per byte, near the report's balance point of 2d = 6,144. The page does not name the GPU or define "legacy". The deployment document adds, without a number, that Mega MoE is significantly faster than unfused MoE "regardless of EP size or batch size".

Figure 27.7 The bound 1 / (1 − f) with the three published speedups marked, and the speedup of pull request 316 by batch size.
Figure 27.7 The bound 1 / (1 − f) with the three published speedups marked, and the speedup of pull request 316 by batch size.

9. The control flow in code

The listing is a sketch of the idea and is not DeepGEMM's code. gpu is the toy clock of the check script.

def fused_moe_layer(gpu, waves):
    gpu.pull(waves[0])                           #A
    for w, wave in enumerate(waves):
        x = gpu.wait_arrived(wave)               #B
        if w + 1 < len(waves):
            gpu.pull(waves[w + 1])               #C
        y = gpu.run_experts(wave, x)             #D
        gpu.write_back(wave, y)                  #E
    return gpu.wait_written()                    #F
#A The first read is exposed, since nothing can be computed yet.
#B The cores wait for this wave's rows only.
#C Receive wave w + 1: the read is started and not waited for.
#D Compute wave w: both products and SwiGLU for each of its experts.
#E Send: the write returns at once, and wave w − 1 may still be on the link.
#F The last write is exposed as well.

Run on lesson 20's matrices, the listing ends at tick 38 and returns lesson 20's output. For "bright" the combined expert output is [0.2535, −2.2554, −1.2905, −2.0851].

10. Beyond EP32

The deployment document says the gain of Mega MoE may stop growing beyond EP32 in decode: "Once memory bandwidth is no longer the bottleneck, a larger EP size introduces additional communication and synchronization overhead."

Lesson 20's cost model explains the shape. A decode step reads all expert weights of a layer once, so the weights read per token fall as 1 / N. We turn the bytes into time with a chosen memory rate of 4,000 GB/s and add a chosen cost of 1 µs per rank.

EP     weights read per token   memory time   per-rank cost   layer
8      15.03 MB                 225.5 µs      8 µs            233.5 µs
16     7.52 MB                  112.8 µs      16 µs           128.8 µs
32     3.76 MB                  56.4 µs       32 µs           88.4 µs
64     1.88 MB                  28.2 µs       64 µs           92.2 µs
128    0.94 MB                  14.1 µs       128 µs          142.1 µs

Each doubling halves what is left to save, while a cost per rank doubles. From EP32 to EP64 the memory term saves 28.2 µs and the other term adds 32 (figure 27.8). With 0.5 µs per rank the turn moves to EP64, so the model shows the shape and not the place. The document gives no number. The source does show work that grows with the group: the kernel notifies remote ranks and waits for all ranks before it ends.

The document names a second cost: "Different jobs sharing NVLink within a supernode can interfere with one another and slow down." In the terms of section 5, a shared link raises r and s. Once they pass c, every wave waits. The document's remedy is to give the whole supernode to a single EP group.

Figure 27.8 The memory term of lesson 20's cost model against a chosen cost per rank, for decode at 60 sequences per rank.
Figure 27.8 The memory term of lesson 20's cost model against a chosen cost per rank, for decode at 60 sequences per rank.

11. Four notes for hardware

The report closes section 3.1 with observations for hardware vendors.

Observation Proposal Reason given
Full overlap depends on the ratio of compute to bandwidth. For V4-Pro a pair needs 6hd operations and 3h bytes, so C/B ≤ 2d = 6,144 FLOPs per byte Design for this balance point, and do not scale bandwidth unconditionally Each GB/s hides 6.1 TFLOP/s of compute. Beyond that, more bandwidth has diminishing returns
Fusion drives compute, memory and network to high load at once Power headroom for fully concurrent workloads Power throttling becomes a key limiter
Dispatch is pull-based Lower-latency signaling across GPUs Fine-grained push has high notification latency. Faster signaling would make push viable
SwiGLU is costly after the product A low-cost element-wise activation without exponential or division The GEMM pipeline should not stall on the activation

12. What the real kernel adds

Toy Mega MoE as published
GPUs in a group 4 8 in the benchmark; 64 to 128 for decode in DeepInfer
Experts per GPU and layer 2 48 for V4-Pro at EP8; 6 for V4.1-Flash at EP64
Rows per expert 0 to 5 1,024 at 8,192 tokens per rank; 60 in decode at EP64
A wave one expert "a small portion of experts"
Kernels per layer 3, then 1 5 steps in the test's baseline, then 1
A remote pair on the link 1 + 2 ticks 3h bytes, FP8 out and BF16 back
Speedup 1.29 1.50 to 1.96 at EP8

The Mega MoE files we found are for SM100, the architecture a comment on the pull request calls Blackwell. The current DeepEP README has one buffer interface, with no separate low-latency kernels and no receive hook.

Not published: the number of experts in a wave or of waves in a layer; the ratio of communication to computation in the profile; the values in the report's figure 5; the benchmark's GPU and baseline; any number for Ascend or beyond EP8; the origin of the figure 32. We read the source files through summary questions and not line by line.

13. What is still open

Mega MoE begins after two decisions have been made. The caller copies topk_idx into the buffer before each call, so the six experts of every token are already chosen. The README remarks that this step may be fused "into previous kernels". Lesson 1's indexer makes a selection of the same kind over cache entries.

The baseline also contained a kernel that DeepGEMM does not own. Its SwiGLU step comes from a library written in TileLang. Lesson 28 closes Part 6 with both: TileLang, and the kernels that select the top k.

Keep going

This is one of 30 lessons

Members get every lesson, every lecture, the assignments and direct contact with me for a year. Founding members pay ₹4,999.