Kimi K3 · Lesson 2 of 14 · Sequence axis
Three quarters of K3's attention layers never look at a past token. They keep a fixed-size notebook instead, and continuously fade, erase, and overwrite it.
Last lesson you drew the block: three KDA layers to one Gated MLA layer. This lesson is about what a KDA layer actually does. By the end you should be able to explain, without equations, why a fixed-size memory makes a million-token context affordable, and what it gives up in exchange.
The mechanism only makes sense against the thing it replaces, so we start there.
Here is the part of a transformer that matters for this lesson. When a model reads a token, it produces three vectors from it. The query says what this token wants to know. Its key is a label advertising what it offers in return. The value is the content it hands over if asked.
To compute attention for a new token, the model takes that token's query and compares it against the key of every token before it. Strong matches get a high weight, weak matches get near zero, and the output is the weighted blend of all those values.
The cache is the whole problem. It has to hold a key and a value for every token so far, so it grows without limit as the context fills. And each new token must compare itself against every entry, so the work per token grows too.
FULL ATTENTION — keep everything, compare against everything
token: 1 2 3 4 5 ... n
│ │ │ │ │ │
cache: [k₁] [k₂] [k₃] [k₄] [k₅] … [kₙ] grows with n
[v₁] [v₂] [v₃] [v₄] [v₅] … [vₙ]
new token n+1 arrives with query q
│
├──▶ compare q against ALL n keys
│
└──▶ blend ALL n values by match strength
cost per token: proportional to n ← gets slower as you go
memory: proportional to n ← never stops growing
At a 4,000-token context nobody cares. At a million tokens, both terms are ruinous. Drag this to feel the shape:
The numbers above are relative units showing growth shape, not K3's exact byte counts. The shape is the point: full attention's state climbs a thousandfold between 1K and 1M, while a KDA layer's stays flat.
Now the alternative. Instead of keeping every past key and value, what if a layer kept one fixed-size scratchpad and wrote each new token into it as it arrived?
That scratchpad is a matrix, and it is the central object in KDA. Call it S. Its job is to store associations: given a key, hand back the value that key was filed under. It is the same read-a-key-get-a-value contract as the KV cache, except the storage never grows, because new writes go on top of old ones.
The obvious way to write into such a matrix is to just add: take the new key and value, form their outer product, and add it to what is already there. That is classical linear attention, and it has a well-known failure. Nothing is ever removed, so the matrix accumulates every association ever written and the useful signal drowns in old writes. Read from it after a hundred thousand tokens and you get mud.
KDA fixes this with two ideas layered on top of the plain write. Both are about removing information rather than adding it.
This is the core of the lesson. Every KDA layer performs three operations per token, in order.
Step ② is the one worth pausing on, because it is the "delta" in Kimi Delta Attention and it is the least intuitive. The name comes from the delta rule in classical associative-memory research: rather than blindly adding, you first ask what the memory currently returns for this key, then write the difference. If the memory already holds a stale association for a similar key, that stale association gets cleared instead of blended with the new one.
Step ① is what KDA adds over its predecessor, Gated DeltaNet. The decay is not one scalar for the whole memory but a separate factor for every channel, computed from the current token.1 The layer can hold one part of the memory steady across thousands of tokens while letting another part turn over rapidly.
From §2.1.1 of the report, this is the whole recurrence in one line:1
Reading it right to left, in the order things happen:
S_{t−1}: the memory matrix arriving from the previous token, shape (d_k × d_v).Diag(α_t), fade. A diagonal matrix of per-channel retention factors, each in (0,1). Multiplying by a diagonal matrix scales each channel independently, which is exactly what "channel-wise" means.(I − β_t k_t k_tᵀ), erase. The term k_t k_tᵀ projects onto the direction of the current key. Subtracting it from the identity removes whatever the memory currently stores along that direction, scaled by β.+ β_t k_t v_tᵀ, write. An outer product files v under k, with strength β.õ_t = S_tᵀ q_t, read. One matrix–vector product. Note it uses S at time t, not t−1, so a token can read what it just wrote.Where α and β come from, also §2.1.1:
Two details that matter later. ShortConv is a small convolution across neighbouring positions, so each query, key, and value sees a little local context before entering the recurrence. L2Norm on q and k bounds their magnitude, which keeps the erase term well behaved. The decay logits z_t come from a low-rank projection plus a per-head bias, and lesson 3 is about how z_t becomes α_t, a seemingly small choice that turns out to unlock the entire Tensor Core fast path.
The gain is the one the simulator showed. State per layer is constant, so a KDA layer costs the same at token 900,000 as at token 9. This is the mechanism behind K3's million-token context. It is also why K3 needs no positional encoding in its global layers, which is a strange enough claim that it gets its own treatment in lesson 3.
The cost is real and you should be able to state it. A fixed-size memory is a lossy summary. Full attention can retrieve token 12,345 exactly, because token 12,345's key and value are still sitting in the cache untouched. KDA cannot promise that. It has faded and overwritten that region of memory many times since, and what survives is whatever the gates chose to preserve.
This is the reason the block is 3:1 and not 4:0. Some tasks genuinely need exact retrieval from an arbitrary distance: finding one line in a large file, or quoting a number from page 200. The Gated MLA layers keep that capability, and the report places one at the very end of the backbone so the model's final act is always full global attention. K3 pays for perfect recall in a quarter of its layers and takes the cheap linear path in the rest.
| Layer type | State grows? | Exact recall? | Layers in K3 |
|---|---|---|---|
| KDA linear |
no, fixed | approximate | 69 (74%) |
| Gated MLA global |
yes, with n (compressed) |
exact | 24 (26%) |
MLA has a trick of its own for that growing cache: it stores one small latent vector per token instead of full keys and values. Lesson 3 takes it apart. Note what it does and does not achieve, because the distinction matters for the 1M-context claim: it shrinks the cache substantially, but the cache still grows with every token.
From memory. These are deliberately the questions that separate understanding the mechanism from recognising the words.
Why does a plain linear-attention layer degrade over a long context?
Classical linear attention only ever adds outer products to its state. With no removal mechanism, signal-to-noise falls as the sequence runs on. KDA's fade and erase steps exist precisely to remove.
What does the "delta" in Kimi Delta Attention refer to?
The delta rule comes from classical associative memory: read what the memory currently returns for this key, then write the difference. This makes a write behave as replacement rather than accumulation.
A task needs one exact figure quoted from deep in a long document. Which K3 layers carry that?
Exact long-range retrieval is precisely what KDA gives up and what the periodic global layers preserve. This complementarity is the whole argument for a hybrid, and it is why the final backbone layer is Gated MLA.
Lesson 3 finishes the sequence axis with the two choices that make KDA practical at scale rather than merely elegant. First, how a memory that is defined one token at a time can be computed in parallel on a GPU. Second, the single constant g_min = −5, which looks like an arbitrary implementation detail and is in fact the reason K3's kernels run entirely on Tensor Cores. That lesson also covers Gated MLA properly, and the NoPE decision.