A self-paced course
11 lessons and 3 spaced reviews on the Kimi K3 architecture, built from the technical report rather than from summaries of it.
It assumes you know what tokens and context windows are and would not confidently draw a transformer block. Everything past that is built, not assumed. There is no build step and nothing to install: every page is plain HTML that prints cleanly.
You opened it straight from the filesystem in Safari, which refuses to let a
file:// page load the stylesheet from the directory above it. Serve the course
instead, from the folder holding this page:
python3 -m http.server 8000 then http://localhost:8000/
The course follows the report's own organising idea: K3 scales information flow along three directions, so each component's purpose is predictable before you learn how it works. Every lesson is tagged with the axis it belongs to.
Reviews carry no new material. They are deliberately harder than the lessons, because retrieving something you half-remember is what makes it stick.
A 2.8-trillion-parameter model that runs 104 billion parameters at a time. One diagram explains where every innovation in the report belongs.
Three quarters of K3's attention layers never look at a past token. They keep a fixed-size notebook instead, and continuously fade, erase, and overwrite it.
A recurrence is sequential and a GPU is parallel. Resolving that tension is what separates a clever idea from a shipped model, and it turns on one number.
Most of K3's changes chase efficiency. This one chases quality instead, and it comes from noticing that depth has the same flaw sequence had.
No new material. This page is built to be harder than the lessons were, because retrieving something you half-remember is what makes it permanent.
896 experts per layer, 16 awake at a time. The expert pool is not a component of K3 so much as it is K3, and one compression trick is what makes it affordable.
Sparsity of 56 breaks the vanilla design in two ways. This lesson is the first fix: capping an activation function nobody had questioned in years, because K3 trains in 4-bit.
The second stability fix, and the more satisfying one. It turns a fiddly tuned heuristic into a question with a direct answer, and that answer is the solution to a known problem.
K3 throws away the standard recipe for building a vision encoder. The reason has nothing to do with how well that recipe performs, and everything to do with what it does to training stability.
The last piece of the model sits outside the network entirely: how its weights get updated, and the empirical study that produced the claimed 2.5×.
Second review, and the architecture is complete behind you. This one asks you to reconstruct numbers, not just mechanisms.
Every architectural choice in the preceding lessons shows up on an invoice. This is where the hybrid design gets awkward, and where it pays.
Pretraining produced something that can predict tokens. Nine specialist models and a consolidation step produced something that can work for hours. Then the team published three ways it falls short.
Three tests, one for each part of the mission. Reconstruct the architecture, explain it to someone else, and read a passage you have never seen.
Built to print and to be returned to, rather than read once.
In order, one sitting each. Lessons are short by design, because working memory is small. Stopping after one is the intended pace, not a failure to finish.
Do the quizzes before reading the answer. Committing to a wrong answer and being corrected beats recognising the right one. The review drills go further: they hide the answer until you have said yours out loud.
Open the equations when the intuition has landed, not before. Every mechanism leads with a
diagram, and the argument stands without opening
▸ show me the actual equation.
Follow the primary source. Each lesson ends with the exact section of the report to read next, and every number in the course cites where it came from. The course is built to be checked against the paper, not taken on trust.