Kimi K3 · Review C · capstone

Capstone

Three tests, one for each part of the mission. Reconstruct the architecture, explain it to someone else, and read a passage you have never seen.

The mission had three legs: read frontier papers unaided, explain K3 to other people, judge it for deployment. This page tests each one directly. The last exercise is the one that matters most, because it uses material the course never covered.

Test one: reconstruct it

Get a blank sheet of paper. Do this before reading anything below, and expect it to be harder than it looks.

Draw Kimi K3 from memory. The repeating block, the layer counts, what pairs with what, the input pathway, and the numbers on each component.

Check yourself against the one-page map further down this lesson. The things most people miss: the extra trailing Gated MLA that makes 93 rather than 92; that AttnRes uses a different partition (8 blocks of 12) than the attention blocks (23 blocks of 4); that every attention layer gets its own LatentMoE, so there are 92 MoE layers and 1 dense; and that the router's bias affects selection but never the mixture weights.

For each of these, state the problem it solves and what it costs: KDA, NoPE, AttnRes, LatentMoE, SiTU-GLU, Quantile Balancing, Per-Head Muon.

KDA — problem: KV cache grows without bound, so 1M context is unaffordable. Cost: retrieval becomes approximate; exact recall is lost.
NoPE — problem: extending a RoPE model's context needs frequency rescaling and usually costs quality. Cost: all positional information now rests on the KDA layers alone.
AttnRes — problem: the residual stream sums all earlier layers, destroying origin. Cost: ~4% training, ~2% inference, for modest gains; needs earlier outputs kept reachable.
LatentMoE — problem: parameters and dispatch traffic both scale with expert width. Cost: routed experts never see full width, plus two extra projections.
SiTU-GLU — problem: unbounded activations wreck 4-bit precision. Cost: slight loss of expressiveness at large magnitudes.
Quantile Balancing — problem: fixed-step bias updates misbehave at ~10³ experts. Cost: needs a Top-(k+1) pass and a histogram all-reduce.
Per-Head Muon — problem: one orthogonalisation across 96 heads lets loud heads set the scale. Cost: none, it is also cheaper.

Three numbers from memory: K3's parameter derivation, one max-resolution image in tokens, and one 50-turn coding session's cost with and without cache.

Parameters: 3 × 3,584 × 3,072 ≈ 33.0M per routed expert, × 896 ≈ 29.6B per layer, × 92 layers ≈ 2.72T, about 98% of the 2.78T total.
Image: 3584² at patch 14 gives 256² = 65,536 patches, pixel-shuffled 2×2 to 128² = 16,384 tokens, or 1.6% of a 1M context.
Session: 400K prefix + 4K new per turn. Miss $1.21/turn, hit $0.13/turn. Over 50 turns, $60.60 versus $6.60.

Test two: explain it

The same model, three audiences. Being able to switch register is what distinguishes understanding from recitation.

Explain to a smart non-specialist, in about thirty seconds, why K3 can read a million tokens when most models cannot.

One good version: "Most models remember by keeping every word they have read and re-checking all of them for each new word. That gets slower and heavier the longer you go. K3 uses a different kind of memory for three quarters of its layers: a fixed-size notebook it continuously updates, fading old entries and overwriting them. The notebook never grows, so reading the millionth word costs the same as the tenth. It keeps the old expensive method in the remaining quarter of layers, for when it needs to recall something exactly." No jargon, correct mechanism, and it names the tradeoff.

Explain to an ML engineer, in about a minute, what is architecturally novel in K3 rather than merely scaled up.

Something like: "Three things. First, hybrid attention at a 3:1 ratio, KDA to Gated MLA, with all-NoPE — position comes from KDA's recurrence, so context extension needs no retuning. K3's KDA contribution over Kimi Linear is bounding the log-decay at −5, which keeps the chunkwise reciprocal inside BF16 and let them delete the slow diagonal-tile path. Second, Attention Residuals: learned pseudo-queries let each layer attend over earlier block outputs instead of reading one summed residual. Third, Stable LatentMoE: 896 experts at half the model width, with Quantile Balancing setting router biases from score quantiles rather than nudging them, derived from balanced assignment via LP duality. The 2.5× scaling claim bundles all of this with data changes and is not decomposed."

A colleague asks whether to switch a production agentic coding workload from Claude Fable 5 to K3. Give them a real answer.

A defensible answer names the tradeoff and the conditions. "K3 is about 4 points behind Fable 5 on Kimi's own coding benchmark at 38% of the cost, and it leads on SWE-Marathon, ProgramBench and WebDev Arena. If cost matters more than the last few points of capability, it is worth a trial. Three caveats before you commit: your harness must pass reasoning blocks back between turns or quality gets unstable — that is the team's own first published limitation. It is prone to acting without asking, so constrain it in AGENTS.md. And the team flags a user-experience gap that benchmarks do not capture, so trial it on your real work rather than trusting the numbers. Self-hosting needs 64+ accelerators, so use the API unless you have a supernode." Recommending a trial rather than a switch is the right shape of answer here.

Test three: read something new

This is the real exam. The passage below is from §5.2.1, on MoonEP, K3's expert-parallelism scheme. The course never covered it. Read it cold, then answer.

    "In conventional EP schemes, token loads are imbalanced across ranks.
     The resulting computational imbalance degrades training throughput,
     and the dynamically varying shapes of routed-expert activations cause
     substantial memory fragmentation. We therefore propose MoonEP, an EP
     scheme that achieves perfect load balance with dynamic redundant
     experts. [...]

     MoonEP requires every rank to receive exactly S × K tokens, where S
     is the sequence length and K is the number of experts selected per
     token, so that all ranks perform identical amounts of computation.
     The key question is how many redundant experts suffice to guarantee
     such a balance. Let E be the number of experts and R the EP size. We
     prove that a balanced plan always exists with at most E/R redundant
     experts per rank and that this bound is essentially tight. Reserving
     E/R redundant-expert slots per rank therefore guarantees that planning
     always admits a feasible solution, so training is never interrupted.
     In contrast, prior work such as ECHO and UltraEP presets the number of
     redundant experts or imposes a per-rank token cap. Training is then
     forced to stop whenever no feasible plan exists within the cap, and the
     cap itself requires manual tuning..."
Kimi K3 technical report, §5.2.1, abridged. Material this course did not teach.

What two distinct problems does imbalanced expert parallelism cause here, and why is the second one non-obvious?

(1) Throughput loss — ranks do unequal work, so the fastest wait for the slowest. That one is intuitive, and you met it in lesson 7b with Quantile Balancing. (2) Memory fragmentation — because the shapes of routed-expert activations vary from step to step, the allocator cannot reuse buffers cleanly. This is the non-obvious one: it is not about how much work each rank does but about the work having unpredictable dimensions. It connects to the report's design goal of static computation shapes and zero-copy communication (§5.2.1): predictable shapes are a goal in their own right.

Compute the redundant-expert bound for K3's own configuration, assuming an EP size of 64. What does the answer mean practically?

E = 896 experts, R = 64, so E/R = 14 redundant-expert slots per rank. Practically: each rank reserves 14 spare slots, and that reservation is provably enough for a balanced plan to always exist. So planning can never fail, and training can never be interrupted for want of a feasible assignment. Note how this connects to the 64-accelerator recommendation from lesson 11: E/R falls as you add devices, so the reservation gets cheaper per rank at larger EP size.

What kind of claim is "essentially tight", and what is the authors' rhetorical move against ECHO and UltraEP?

"Essentially tight" is a claim about a bound: they prove E/R suffices, and also that you cannot do much better, so the bound is not merely an upper limit they happened to reach. The hedge "essentially" signals the lower bound matches up to some slack rather than exactly. The proof is deferred to appendix E.

The rhetorical move is to reframe a quantitative difference as a qualitative one. Prior work presets a number or caps tokens per rank, which means a plan can fail and training must stop, plus the cap needs hand-tuning. MoonEP's contribution is framed as a guarantee rather than a better heuristic: no tuning, no failure mode. This is a common and legitimate pattern in systems papers. Watch for it, and watch for whether the guarantee holds under assumptions the paper states quietly.

Which mechanism from this course does MoonEP most resemble in spirit, and why?

Quantile Balancing. Both replace a tuned heuristic with something that computes a guaranteed-correct answer. QB stops nudging biases and instead reads the bias that produces the target load off a quantile; MoonEP stops presetting redundancy and instead proves a reservation size that always admits a feasible plan. Both eliminate a hyperparameter and a failure mode at once. Arguably g_min belongs in the same family: bound the quantity so the bad case becomes impossible by construction, rather than detecting and handling it. Recognising that family across three different layers of the stack is the transferable insight.

The whole thing, one page

    ┌──────────────────────────────────────────────────────────────────┐
    │  KIMI K3   2.78T total · 104.2B active · 93 layers · 1M context  │
    │            native multimodal · open weights · Jul 2026           │
    └──────────────────────────────────────────────────────────────────┘

    THE BLOCK  × 23, plus one trailing Gated MLA = 93 layers

      3 × [ KDA        + Stable LatentMoE ]     linear, O(n), 69 total
      1 × [ Gated MLA  + Stable LatentMoE ]     global, NoPE, 24 total

    ── SEQUENCE ──────────── how tokens mix ─────────────────────────
      KDA          fade (channel-wise α) → erase (delta rule) → write
                   chunkwise: recurrent between, parallel within
                   g_min = −5 → 1/Γ < e⁸⁰ < BF16 → all Tensor Core
      Gated MLA    one latent per token, exact global recall
      NoPE         position from recurrence → no retuning 8K→1M

    ── DEPTH ─────────────── how layers mix ─────────────────────────
      Block AttnRes   8 blocks × 12 (final partial), learned pseudo-
                      query per layer, RMSNorm on keys,
                      O(Ld) → O(Nd), ~4% train / 2% infer
                      blocks 1/4/final feed the spec-decoding draft

    ── WIDTH ─────────────── how channels mix ───────────────────────
      Stable LatentMoE   896 routed @ ℓ=3,584 (0.5d), 16 active
                         sparsity 56, 2 shared @ full width
                         ≈2.72T params = 98% of the model
        RMSNorm pre-W↑     stabilises the routed branch
        SiTU-GLU           β₁=4, β₂=25 → bounded at 100, for MXFP4
        Quantile Balancing bias from score quantile, 1 step, LP dual

    ── INPUT ────────────────────────────────────────────────────────
      MoonViT-V2   401M · 27 layers · patch 14 · from scratch (NTP)
                   pixel-shuffle 2×2 → 3584² = 16,384 tok = 1.6% of 1M
                   hidden in pipeline bubbles

    ── OPTIMISATION ─────────────────────────────────────────────────
      Per-Head Muon   96 heads orthogonalised separately, and cheaper
      cosine > WSD    each schedule tuned independently
      claimed 2.5×    vs K2, undecomposed, unreplicated

    ── SERVING ──────────────────────────────────────────────────────
      $0.30 hit / $3.00 miss / $15.00 out per MTok · >90% hit coding
      MXFP4 experts + MXFP8 acts, QAT through SFT and RL
      64+ accelerators · decoupled prefix-cache granularities
      cache-affinity routing · budget-based admission control

    ── BEHAVIOUR ────────────────────────────────────────────────────
      SFT → RL (3 domains × 3 efforts = 9 experts) → MOPD → one model
      Limitations: thinking-history sensitivity · over-proactiveness
                   · UX gap vs Fable 5 and GPT-5.6 Sol
Everything the course covered. Print this. It is the compressed form of thirteen lessons.

Where to go now

The course is finished, so the useful thing is what replaces it.