Learn how DeepSeek builds its models, one idea at a time.
DeepSeek keeps publishing how its models and its inference stack work. I read each paper and each repository, rebuild the idea on a small example with real numbers, and teach it. Join to follow the whole year with me.
Founding members pay ₹4,999 for the year. Payment is not open yet, so joining the list costs nothing.
A team that keeps showing its work
Many labs release a model and say little about how it works. DeepSeek publishes detailed reports and open-source code for its attention, its training and its inference engine. That makes it possible to learn the whole stack from the people who built it.
Everything I make this year, in one place
This is where I will post all of it. Members see every lecture, every note and every assignment, and can reach me directly.
Every lecture
Some lectures are live and some are recorded. Members get all of them. About one in ten also goes on YouTube.
My notes
Thirty lessons are already written, with 219 figures. I add new ones as DeepSeek publishes new work.
Numbers you can check
Each lesson works its idea on a small example with real matrices. A short script reproduces every number in it.
Assignments
Problems that make you rebuild the idea yourself, shared only inside the community.
Direct contact with me
You can write to me one to one about the material, your questions and your own work.
Learning in the open
When I read a new paper or repository, I share my screen and work through it with the group.
From the first token to the super-agent
We start with how a language model works and what DeepSeek built first. Then we go through everything it has published since V3. The first lesson of every part is free to read.
Start hereFoundations9 topics
Language models, the KV cache, and DeepSeek up to V3 and R1. These topics are also in the Build DeepSeek from Scratch series and the Manning book. The written lessons for this part are being prepared.
- F1How a language model writes textTokens, embeddings, attention and the loop that produces one token at a time.Coming
- F2The KV cacheWhy generation repeats the same work at every step, and the cache that removes it.Coming
- F3DeepSeek before V3What DeepSeek released first: DeepSeek-Coder, the first chat models and V2.Coming
- F4Multi-Head Latent AttentionThe idea from V2: keep one small latent vector per token and rebuild keys and values from it.Coming
- F5Rotary positions and decoupled RoPEHow position enters attention, and why latent attention keeps it separate.Coming
- F6DeepSeek's Mixture-of-ExpertsShared and routed experts, and load balancing without an auxiliary loss.Coming
- F7DeepSeek-V3: multi-token prediction and FP8 trainingThe training choices behind the model that made DeepSeek widely known.Coming
- F8DeepSeek-R1 and GRPOHow reinforcement learning taught V3 to reason step by step.Coming
- F9DistillationMoving the ability of a large model into smaller ones.Coming
Part 1Attention after MLA6 lessons
From V3.2 to V4.1: read fewer tokens, compress the rest, and share the cache across layers.
- 01DeepSeek Sparse Attention (DSA)A small indexer scores every past token and attention reads only the top k. The one change in V3.2.Free to read
- 02Training the indexer: KL loss, dense warm-up, sparse stageHow the indexer learns where attention looks: a KL loss, a dense warm-up, then a sparse stage.Members
- 03Compressing KV entries into blocks: Compressed Sparse Attention (DeepSeek-V4)V4 packs every few tokens into one cache entry and indexes the blocks, on the way to a 1M-token context.Members
- 04Heavily Compressed Attention, the sliding window, the attention sink and partial RoPE (DeepSeek-V4)Blocks of 128 tokens read densely, a window of raw recent tokens, the attention sink and partial RoPE.Members
- 05CSA2: sharing KV entries and indices across layers, and the hierarchical indexer (DeepSeek-V4.1-Flash)V4.1 lets layers share cache entries and indices, which brings the cache to 890 bytes per token.Members
- 06The Causal Encoder-Decoder: why prefill runs half the layers (DeepSeek-V4.1-Flash)Why reading a prompt runs only half the layers in V4.1-Flash.Members
Part 2The backbone4 lessons
What changed inside the block: a wider residual stream, a lookup memory, and new expert routing.
- 07Hyper-connections and why they blow upA wider residual stream, and how its mixing matrices can amplify a signal thousands of times.Free to read
- 08mHC: doubly stochastic matrices and Sinkhorn-KnoppDoubly stochastic matrices and the Sinkhorn-Knopp iteration that keep the wider stream stable.Members
- 09Engram: N-gram lookup memoryA lookup memory addressed by the last few token ids, placed beside the experts.Members
- 10MoE updates in DeepSeek-V4: Sqrt(Softplus) affinity, hash routing, anticipatory routing and SwiGLU clampingA new routing score, hash routing in early layers, anticipatory routing and SwiGLU clamping.Members
Part 3Training and post-training4 lessons
A new optimizer, four-bit numbers, specialists merged by distillation, and controlled thinking.
- 11Muon and Newton-Schulz iterations in DeepSeek-V4The optimizer V4 trains with: equalise the directions of a gradient using only matrix products.Free to read
- 12FP8, UE8M0 and FP4: number formats and quantization-aware trainingNumber formats built bit by bit, block scales, and training with the quantizer in the loop.Members
- 13Specialists, GRPO and on-policy distillationMany specialist models merged into one student that learns on its own text.Members
- 14Reasoning effort, interleaved thinking, the tool-call format and the generative reward modelControlling how long a model thinks, keeping thoughts across tool calls, and a model that judges itself.Members
Part 4The KV cache5 lessons
How the cache is laid out in memory, kept on SSD for days, rebuilt after a hit and moved over the network.
- 15The block KV cache and the state cacheHow a serving engine lays out compressed entries and the sliding window for many requests at once.Free to read
- 16The KV cache on SSD: 3FS, prefix matching and time to livePrefix matching with a hash chain, erasure coding, and a cache that lives for days on 3FS.Members
- 17SWA bounded replay: rebuilding the sliding window after a cache hitRebuilding the sliding window after a cache hit by replaying only the last 128 tokens.Members
- 18The FP4 KV cache and the bytes-per-token ledgerFrom 3,997,696 bytes of cache per token down to 890, one step at a time.Members
- 19DualPath: loading the cache through two network pathsLoading a cached prefix through the idle network cards of the decode machines.Members
Part 5The inference engine7 lessons
DeepInfer: pools of GPUs, load balancing, scheduling, speculative decoding and structured outputs.
- 20Prefill and decode as separate pools: expert and data parallelismFour ways to spread a model over GPUs, and why reading and generating are grouped differently.Free to read
- 21Expert-parallel load balancing (EPLB)Copies of hot experts and a greedy placement that evens the load across GPUs.Members
- 22The prefill and decode load balancersLength tiers, engines that pull work, and balancing sequences and cache length together.Members
- 23Asynchronous scheduling, and batch-invariant and deterministic kernelsOverlapping CPU and GPU work, and why the same prompt can give different answers in different batches.Members
- 24DSpark: drafting five tokens and deciding how many to verifyDrafting five tokens in one pass and deciding how many of them to verify.Members
- 25DFlash and EAGLE-3 compared with DSparkThree answers to one question: how does a cheap module draft tokens that agree with each other?Members
- 26Structured outputs and the tokenizer front endMasks that force valid JSON, safe handling of special tokens, and streaming.Members
Part 6Kernels2 lessons
The GPU code underneath: a fused expert kernel, a kernel language and top-k selection.
- 27The fused MoE kernel: overlapping communication with computationHiding communication behind computation by running experts in waves inside one kernel.Free to read
- 28TileLang and the top-k selection kernelsA language for writing kernels, and picking 512 of a million scores without sorting them.Members
Part 7The harness2 lessons
The program around the model: deepseek-harness and the super-agent prototype, read from their repositories.
- 29deepseek-harness, an agent harness where everything is a pluginAn agent harness where every part, including the loop, is a plugin. Read from the repository.Free to read
- 30The super-agent: Router, Workers and WatchdogA resident Router, isolated Workers and a Watchdog, and what the prototype's code really does.Members
One payment for the whole year
This is a year of my work, so it is paid. The price is kept low enough that anyone serious about learning this can join.
- Every lecture, live and recorded
- All notes, including the 332-page PDF
- Assignments
- Direct contact with me
- The same access as founding members
Payment is not open yet. People on the founding list hear first, and the founding price goes to the first 100 who pay. Members will open the private lessons and videos from their Vizuara account.
Be among the first 100
Leave your name and email. I will write to you when the community opens, with the start date and the payment link. Joining the list does not commit you to anything.
I will use your email only to write to you about this community. I will not share it.
You are on the list.
Thank you. I will write to you before anyone else when the community opens. Until then, the free lessons in the curriculum are yours to read.
Dr. Raj Dandekar
I teach at Vizuara AI Labs. I am a co-author of Build a DeepSeek Model (From Scratch), published by Manning, and I taught the Build DeepSeek from Scratch series on YouTube.
That book and series stop at DeepSeek-V3 and R1. DeepSeek has since replaced most of what they describe. This community is how I keep up with it, and you can keep up with me.
Is this an official DeepSeek course?
No. This is an independent community run by Vizuara. It is not affiliated with or endorsed by DeepSeek. Everything is taught from DeepSeek's public papers, technical reports and open-source code.
What do I need to know before joining?
Python, the basics of deep learning, and some exposure to the transformer. The Foundations part covers language models, the KV cache and DeepSeek's earlier models before the newer material begins.
Do I need a GPU?
No. The worked examples are small enough to run on a laptop. The real model sizes are given beside them so you can see the same idea at full scale.
How are the lectures delivered?
Some are live and some are recorded. All of them are shared with members, along with the notes and the assignments.
How long does it run?
One year. DeepSeek keeps publishing, so the material keeps growing through the year.
When can I pay, and what does the founding price mean?
Payment is not open yet. Join the founding list and I will write to you first with the start date and the payment link. The first 100 members pay ₹4,999 for the year. After that the price is ₹9,999 for the year.