Kimi K3 · Lesson 3 of 14 · Sequence axis
A recurrence is sequential and a GPU is parallel. Resolving that tension is what separates a clever idea from a shipped model, and it turns on one number.
Lesson 2 left KDA with a problem. The memory update is defined one token at a time: fade, erase, write, then hand the state to the next token. That is a loop with a thousand-step dependency chain, and a GPU hates nothing more. This lesson is about the three engineering decisions that make KDA fast, and then about the strangest choice in the whole architecture.
The resolution is to stop treating the sequence as a chain of single tokens and treat it as a chain of chunks. Within a chunk, every position is computed in parallel with dense matrix multiplication. Between chunks, the state passes sequentially as before.
NAIVE — one step per token, nothing overlaps
t₁ ─▶ t₂ ─▶ t₃ ─▶ t₄ ─▶ t₅ ─▶ … ─▶ t₁₀₀₀₀₀₀
└──── 1,000,000 sequential steps, GPU mostly idle ────┘
CHUNKWISE — sequential between chunks, parallel within
┌─ chunk 1 ─┐ ┌─ chunk 2 ─┐ ┌─ chunk 3 ─┐
│ t₁ t₂ … │──▶│ t₁ t₂ … │──▶│ t₁ t₂ … │──▶ …
│ all in │ S │ all in │ S │ all in │ S
│ parallel │ │ parallel │ │ parallel │
└───────────┘ └───────────┘ └───────────┘
Each chunk's output has two parts:
O = (Γ ⊙ Q) S ← INTER-chunk: what arrived from the past,
└─────────┘ one matmul against the incoming state
+
A Ṽ ← INTRA-chunk: interactions among tokens
└───┘ inside this chunk, masked causal
There is a wrinkle, and it is the whole reason this lesson exists. To compute the intra-chunk term in parallel, the math needs each key rescaled by the reciprocal of its accumulated decay. Written as 1/Γ. Since Γ is a product of numbers below 1, it shrinks fast, and its reciprocal grows fast.
So the reciprocal rescale is a numerical hazard sitting in the middle of the fast path. Kimi Linear handled it by computing decay in log space and splitting each chunk into 16-token tiles. That helped, but only partly: the off-diagonal tiles became clean matmuls while the diagonal tiles still needed explicit position-by-position computation. The report names this as the main intra-chunk bottleneck.
1
K3's fix is to change how the decay logits become the actual decay. Kimi Linear used a mapping whose output was unbounded below, meaning a channel could in principle decay arbitrarily hard, sending 1/Γ arbitrarily high. K3 replaces it with a scaled sigmoid that cannot go below a fixed floor:
g_min = −5. Every channel's log-decay is squeezed into the range (−5, 0), so no channel may ever forget faster than a factor of e−5 per token. Over a 16-token tile, the accumulated log-decay therefore cannot exceed 80, and the reciprocal cannot exceed e80.
Now the payoff. e80 is about 5.5 × 1034, which fits inside BF16's ceiling of 3.4 × 1038. Overflow becomes impossible by construction, so the diagonal tiles no longer need their special slow path. Every tile can be a dense Tensor Core matmul.
Drag both sliders. The margin is tighter than you might expect: leave the tile at 16 and move the decay floor to 6, or leave the floor at 5 and move the tile to 24. Either single step over the line is enough to break it.
Two things are worth noticing from playing with it. First, g_min and the tile size are co-designed, and the margin is thin. Overflow begins at a cumulative log-decay of about 88.7, and K3 sits at 5 × 16 = 80. One step in either direction, a floor of 6 or a tile of 24, puts it over. Neither constant was free to choose. Second, the choice is a genuine tradeoff rather than a free win. A floor on decay means a channel cannot forget instantly even when the model would like it to. K3 bought a fast kernel by giving up the ability to fully clear a channel in one step.
The mapping that changed, from §2.1.1 eq. 5:1
A_h is a learnable per-head log-scale initialised to 0, and g_min = −5 is fixed, never learned. Softplus is unbounded above, so negating it gives an unbounded-below log-decay. Sigmoid is bounded in (0,1), so scaling it by g_min bounds the log-decay in (−5, 0).
The arithmetic the bound guarantees:
The report is explicit about the consequence: This finite range allows both diagonal and off-diagonal tiles to use dense Tensor Core matrix multiplications, eliminating the position-pair diagonal path.
It also notes the parameterisation is closely related to the lower-bounded recurrence gates in prior work.
Smaller, but it belongs to the same story of K3 spending a little capacity to buy quality. After the recurrent read, KDA applies a gate that decides which channels of the result actually pass through. Kimi Linear computed that gate with a low-rank projection, a cheap approximation. K3 makes it full-rank and input-dependent, so every token gets a genuinely independent say over all channels.1 Gated MLA received the same upgrade, which is where its name comes from.
Now the layer that handles what KDA cannot. One in four attention layers in K3 is a Gated MLA layer doing full global attention, with exact retrieval at any distance.
Full attention's problem is the KV cache from lesson 2: a key and a value stored per token, per head, forever. MLA's trick is to not store them.
STANDARD ATTENTION CACHE MLA CACHE
per token, per head: per token, ONE vector:
[ k ] [ v ] [ c ] ← low-dim latent
[ k ] [ v ] cₜ = W_c · xₜ
[ k ] [ v ] × 96 heads
[ k ] [ v ] at attention time:
… … k, v ◀── up-project from c
for whichever head
needs them
cache size ∝ n × heads × 2 cache size ∝ n × 1
Note what MLA does and does not fix. It shrinks the constant, so the cache is much smaller per token. It does not change the growth: the cache is still proportional to sequence length. That is exactly why K3 needs KDA in the other three quarters of its layers, and why MLA alone was not enough to reach a million tokens.
Here is the part of K3 most worth being able to explain to someone else, because it sounds wrong.
Attention has no inherent sense of order. Feed it a shuffled sentence and it computes the same thing, because it compares every query to every key without regard to where they sit. Transformers therefore inject position explicitly. The dominant method is RoPE, which rotates queries and keys by an angle proportional to position, so their match strength encodes how far apart they are.
K3 applies NoPE to every one of its MLA layers. No rotation, no learned position embeddings, nothing. The global attention layers are genuinely order-blind.
The argument for why this works is the cleanest illustration of what a hybrid architecture buys. KDA is a recurrence. It processes tokens in order and decays its memory as it goes, so position is baked into its structure: something read recently has had less decay applied than something read long ago. Position information does not need to be injected into the attention scores, because three out of every four layers already carry it inherently.1
DIVISION OF LABOUR ON POSITION
KDA layers (69) MLA layers (24)
────────────── ──────────────
position: implicit, from position: none (NoPE)
decay and recurrence order
recency-aware by construction order-blind by construction
approximate retrieval exact retrieval,
unrestricted global reach
───────────────────────────────────────────────────────────
between them, position is handled once and structurally,
rather than encoded in every layer
CONSEQUENCE FOR CONTEXT EXTENSION
RoPE model: 8K ──▶ 1M needs frequency rescaling (YaRN etc.),
retuning, and usually costs quality
K3 (NoPE): 8K ──▶ 1M no positional parameters exist,
so there is nothing to retune
position-sensitive and recency-aware sequence mixingwhile MLA supplies
unrestricted global content interaction.Raschka notes K3 is the first frontier-scale model he knows of to go all-NoPE.3
The practical dividend appears in how K3 reaches a million tokens. Training runs a four-stage curriculum, 8K to 64K during pre-training, then 256K to 1M during cooldown. At no point does anyone rescale a frequency base or apply an interpolation scheme, because there are no positional parameters to touch. The report puts it directly: the model extrapolates directly to 1M-token contexts without any positional-encoding modification.
1
All-NoPE is a genuinely unusual bet, and K3 is the first model at this scale to make it. The report presents it as working, but there is no independent ablation isolating what it costs, and the KDA layers now carry sole responsibility for all positional information. If you are evaluating K3 for a task that depends on precise positional reasoning over very long inputs, this is the design decision to probe first rather than trust.
What did bounding the log-decay at −5 actually buy K3?
The reciprocal rescale 1/Γ can no longer overflow BF16, so the diagonal tiles lose their slow special-case path. A constant chosen for numerical safety deleted an entire branch from the kernel.
MLA compresses each token to one latent vector. What does that not fix?
MLA shrinks the constant, not the growth rate. A cache proportional to n is still proportional to n. This is why MLA alone could not deliver a million-token context and KDA had to carry three quarters of the layers.
K3's global layers use no positional encoding. Why does the model still know order?
Position is structural rather than encoded. A recurrence that decays its memory is inherently recency-aware, so the 69 KDA layers supply order and the 24 MLA layers can afford to be blind to it. This also means nothing needs retuning when context grows from 8K to 1M.
✅ SEQUENCE AXIS — complete
KDA fade / erase / write, channel-wise decay L2
chunkwise recurrent outside, parallel inside L3
g_min = −5 bounded decay → all-Tensor-Core kernels L3
Gated MLA latent KV cache, exact global retrieval L3
NoPE position from recurrence, not encoding L3
▢ DEPTH AXIS — next
AttnRes each layer retrieves from all earlier ones L4
▢ WIDTH AXIS
LatentMoE, SiTU-GLU, Quantile Balancing L6, L7
Lesson 4 turns to the depth axis and Attention Residuals, which Raschka singles out as K3's one architectural change aimed at quality rather than efficiency. The idea reuses a move you have now seen twice: take a bottleneck where information is squashed into one accumulated state, and replace it with selective, weighted retrieval. Lesson 2 did that for sequence. Lesson 4 does it for depth.
Then lesson 5 is a review that interleaves everything from lessons 1 to 4, because retrieval practice a few days after learning is what turns this into knowledge you keep.