Review A · Sequence and depth

Rereading feels productive and mostly is not. What builds durable memory is pulling something out of your head when it is not immediately available, and being slightly wrong on the way. So this page asks before it tells, and the questions are deliberately mixed across lessons rather than grouped by topic.

How to use this

Answer out loud or on paper before clicking to reveal. The commit step is the part that works. Then score yourself honestly, including "partly" when you had the shape but not the detail. The tally at the bottom reads your scores and names what to revisit.

Free recall

Nine questions, interleaved on purpose, with a tenth on paper-reading further down. If one feels like it came out of nowhere, that is the interleaving doing its job.

Name KDA's three operations per token, in order, and say what each removes or adds.

Fade, erase, write. Fade multiplies every channel of the memory by its own retention factor α, so stale channels decay fast and preserved ones decay slowly. Erase looks up what the current key already retrieves and subtracts it, clearing the slot so the write replaces rather than accumulates. Write files the new value under the new key with strength β. Then the read, Sᵀq, is a single matrix–vector product against the memory rather than a scan over history.

AttnRes borrows an idea from somewhere else in the architecture. From where, and what is the analogy?

From attention itself, applied to depth instead of sequence. RNNs crushed everything read so far into one hidden state and could not reach a specific earlier token; attention fixed that with learned, data-dependent weights over all positions. Standard residual connections have the identical flaw over depth, summing all earlier layers into one state with no way to weight layer 12 differently from layer 69. AttnRes gives each layer a learned pseudo-query so it can select. The report calls the residual stream a bottleneck reminiscent of RNNs over time.

K3 is 2.78T total and 104.2B activated. Which number governs which cost, and what does the ratio mean?

Total governs hosting, activated governs per-token compute. All 2.78T parameters must live in memory somewhere, which is why the launch post recommends 64 or more accelerators. Only 104.2B take part in computing any given token, so arithmetic cost resembles a 104B dense model. The ratio, roughly 27:1, says K3 is deliberately cheap to run and expensive to host. On any MoE config table this ratio is the single most informative thing.

Why is g_min = −5 in the report at all? What would break without it?

The chunkwise form rescales keys by 1/Γ, the reciprocal of accumulated decay. Γ is a product of numbers below 1, so its reciprocal grows fast and can overflow BF16. Bounding log-decay in (−5, 0) caps the accumulated decay over a 16-token tile at 80, so the reciprocal cannot exceed e⁸⁰ ≈ 5.5×10³⁴, inside BF16's 3.4×10³⁸ ceiling. Without the bound, diagonal tiles need a slow explicit position-pair path, which is where Kimi Linear was stuck. With it, every tile is a dense Tensor Core matmul. The cost: a channel can never fully clear in one step.

Why does K3 keep any global attention layers at all, when KDA is so much cheaper?

Because KDA's fixed-size memory is a lossy summary. It has faded and overwritten a given region many times, so it cannot promise to retrieve token 12,345 exactly. Full attention can, because that token's key and value still sit untouched in the cache. Tasks needing exact long-range retrieval depend on that: finding one line in a large file, or quoting a number from page 200. So K3 pays for perfect recall in one layer of four and takes the cheap linear path in the other three, with the final backbone layer always Gated MLA.

K3's global layers have no positional encoding. Why does the model still know word order?

Because KDA is a recurrence, and a recurrence that decays its memory is inherently recency-aware: something read recently has had less decay applied than something read long ago. Position is structural in the 69 KDA layers, so the 24 MLA layers can afford to be order-blind. The dividend shows up in context extension: with no positional parameters anywhere, going from 8K to 1M requires no frequency rescaling or interpolation. Nothing exists to retune.

What does Block AttnRes give up relative to the full form, and what does it get back?

It gives up fine-grained selectivity: within a 12-layer block, outputs are plainly summed rather than individually attended over, so a layer cannot distinguish its immediate neighbours. It gets back memory and interconnect, with overhead falling from O(Ld) to O(Nd). The report says N ≈ 8 recovers most of the benefit. Blocking also bounds inference-time state, letting inter-block and intra-block paths merge with online softmax. Note the binding constraint was never arithmetic: at L < 100 the O(L²d) compute was already affordable.

K3 grew on nearly every axis from K2. Name something that did not grow, and why that is interesting.

Hidden dimension held at 7,168, and vocabulary at 160K. Every bit of width growth went into the expert pool instead: routed experts 384 → 896, active per token 8 → 16, shared experts 1 → 2. That is a deliberate bet that capacity is cheaper to add sparsely than densely. Reading a config table for what stayed the same often tells you more about the design philosophy than reading what grew.

Sketch the K3 block from memory. Layers per block, the pattern, what pairs with what, and the totals.

Each block is 3 KDA layers then 1 Gated MLA layer, a 3:1 ratio, and every attention layer is followed by its own Stable LatentMoE feed-forward network. Twenty-three blocks give 92 layers, plus one trailing Gated MLA makes 93 total: 69 KDA + 24 MLA. Attention Residuals cut across this with a different partition, 8 blocks of 12 layers, the final one partial at 9. At the input, MoonViT-V2 feeds an MLP projector into the same token stream.

Explain it to someone

Mission leg two: being able to say this out loud to another person. Rehearse the answer aloud before revealing.

A colleague says "2.78 trillion parameters but only 104 billion active? That's a contradiction." In 60 seconds, explain why it is not.

The two numbers measure different things. Total parameters is everything the model stores, which sets how much memory you need to host it. Activated parameters is how much takes part in computing any one token, which sets the arithmetic per token. K3 is a mixture-of-experts: each layer holds 896 expert networks but routes each token to only 16, so a token touches about a fifty-sixth of the stored capacity. The upshot: K3 costs about what a 104B model costs to run, while carrying the knowledge of something roughly 27 times that size, and the price is needing enough hardware to hold all 2.78T. No contradiction, just two costs that a dense model conflates and an MoE separates.

Discriminating between mechanisms

These are the questions that separate real understanding from vocabulary. Each one names two things that sound similar and asks you to tell them apart.

MLA shrinks the KV cache and KDA replaces it. Which claim is true of exactly one of them?

Its state stops growing entirely as the context gets longer
It mixes information between tokens rather than between layers
It appears in K3's backbone as one of the attention layers
It applies an explicit positional encoding to its queries

Only KDA's state is truly fixed-size. MLA compresses each token to one latent vector, shrinking the constant, but the cache is still proportional to sequence length. That difference is exactly why MLA alone could not reach a million tokens.

Both KDA and AttnRes replace an accumulated state with weighted retrieval. What differs?

The axis they operate on: one across tokens, one across layers
Only one of them uses learned weights to choose its sources
Only one of them was introduced to reduce computational cost
Only one of them appeared for the first time in Kimi K3

Same idea, different axis. KDA replaces a growing history with a managed fixed memory across sequence; AttnRes replaces a summed residual with selective retrieval across depth. Recognising one idea reused on a new axis is how a lot of architecture research actually works.

Which pairing of a K3 choice with its true motivation is correct?

Bounded decay was adopted so every tile could reach the Tensor Cores
Block AttnRes was adopted to make depth retrieval more selective
Latent attention was adopted to stop the cache growing with length
NoPE was adopted to make position more precise at long range

Motivation is where superficial familiarity breaks down. Bounded decay is a numerics choice with a kernel payoff; blocking trades selectivity for memory; MLA changes the constant not the growth; NoPE eliminates retuning by having no positional parameters to retune.

Read the paper yourself

One excerpt, straight from §2.2 with no commentary. You now have everything needed to read it.

In that sentence, what is the author conceding, and what are they arguing? Why mention pipeline parallelism?

They concede nothing about compute. Quite the opposite: they preemptively dismiss the obvious objection that L² arithmetic sounds expensive, by noting depth is under 100. Then they argue the real cost is memory, O(Ld) for keeping every layer output alive. Pipeline parallelism appears because it makes that cost worse than it first looks: when consecutive layers sit on different devices, keeping outputs reachable means shipping them across device boundaries, not just holding them in local memory. This sentence is the entire justification for Block AttnRes existing, compressed into one line. Spotting "here is the objection, here is why it is not the real problem, here is the real problem" is a pattern you will see constantly in architecture papers.

The map so far

What comes next

Lesson 6 opens the width axis and Stable LatentMoE, where the 2.8T figure actually comes from. The central puzzle: how do you afford 896 experts per layer when a conventional MoE would drown in the cost of shipping full-width token representations to sixteen of them at once?

If the tally above flagged anything, tell me which and I will come at it from a different angle before we move on. That is a better use of the next few minutes than pressing forward.