Point at the architecture

Last lesson you drew the block: three KDA layers to one Gated MLA layer. This lesson is about what a KDA layer actually does. By the end you should be able to explain, without equations, why a fixed-size memory makes a million-token context affordable, and what it gives up in exchange.

The mechanism only makes sense against the thing it replaces, so we start there.

What standard attention actually stores

Here is the part of a transformer that matters for this lesson. When a model reads a token, it produces three vectors from it. The query says what this token wants to know. Its key is a label advertising what it offers in return. The value is the content it hands over if asked.

To compute attention for a new token, the model takes that token's query and compares it against the key of every token before it. Strong matches get a high weight, weak matches get near zero, and the output is the weighted blend of all those values.

The cache is the whole problem. It has to hold a key and a value for every token so far, so it grows without limit as the context fills. And each new token must compare itself against every entry, so the work per token grows too.

    FULL ATTENTION — keep everything, compare against everything

    token:   1     2     3     4     5   ...   n
             │     │     │     │     │         │
    cache:  [k₁]  [k₂]  [k₃]  [k₄]  [k₅]  …  [kₙ]      grows with n
            [v₁]  [v₂]  [v₃]  [v₄]  [v₅]  …  [vₙ]

    new token n+1 arrives with query q
                          │
                          ├──▶ compare q against ALL n keys
                          │
                          └──▶ blend ALL n values by match strength

    cost per token:  proportional to n        ← gets slower as you go
    memory:          proportional to n        ← never stops growing

At a 4,000-token context nobody cares. At a million tokens, both terms are ruinous. Drag this to feel the shape:

The numbers above are relative units showing growth shape, not K3's exact byte counts. The shape is the point: full attention's state climbs a thousandfold between 1K and 1M, while a KDA layer's stays flat.

The other way to remember

Now the alternative. Instead of keeping every past key and value, what if a layer kept one fixed-size scratchpad and wrote each new token into it as it arrived?

That scratchpad is a matrix, and it is the central object in KDA. Call it S. Its job is to store associations: given a key, hand back the value that key was filed under. It is the same read-a-key-get-a-value contract as the KV cache, except the storage never grows, because new writes go on top of old ones.

The obvious way to write into such a matrix is to just add: take the new key and value, form their outer product, and add it to what is already there. That is classical linear attention, and it has a well-known failure. Nothing is ever removed, so the matrix accumulates every association ever written and the useful signal drowns in old writes. Read from it after a hundred thousand tokens and you get mud.

KDA fixes this with two ideas layered on top of the plain write. Both are about removing information rather than adding it.

Fade, erase, write

This is the core of the lesson. Every KDA layer performs three operations per token, in order.

Step ② is the one worth pausing on, because it is the "delta" in Kimi Delta Attention and it is the least intuitive. The name comes from the delta rule in classical associative-memory research: rather than blindly adding, you first ask what the memory currently returns for this key, then write the difference. If the memory already holds a stale association for a similar key, that stale association gets cleared instead of blended with the new one.

Step ① is what KDA adds over its predecessor, Gated DeltaNet. The decay is not one scalar for the whole memory but a separate factor for every channel, computed from the current token.¹ The layer can hold one part of the memory steady across thousands of tokens while letting another part turn over rapidly.

What this buys and what it costs

The gain is the one the simulator showed. State per layer is constant, so a KDA layer costs the same at token 900,000 as at token 9. This is the mechanism behind K3's million-token context. It is also why K3 needs no positional encoding in its global layers, which is a strange enough claim that it gets its own treatment in lesson 3.

The cost is real and you should be able to state it. A fixed-size memory is a lossy summary. Full attention can retrieve token 12,345 exactly, because token 12,345's key and value are still sitting in the cache untouched. KDA cannot promise that. It has faded and overwritten that region of memory many times since, and what survives is whatever the gates chose to preserve.

Locating this on the map

MLA has a trick of its own for that growing cache: it stores one small latent vector per token instead of full keys and values. Lesson 3 takes it apart. Note what it does and does not achieve, because the distinction matters for the 1M-context claim: it shrinks the cache substantially, but the cache still grows with every token.

Lesson 2 left KDA with a problem. The memory update is defined one token at a time: fade, erase, write, then hand the state to the next token. That is a loop with a thousand-step dependency chain, and a GPU hates nothing more. This lesson is about the three engineering decisions that make KDA fast, and then about the strangest choice in the whole architecture.

Chunks: recurrent outside, parallel inside

The resolution is to stop treating the sequence as a chain of single tokens and treat it as a chain of chunks. Within a chunk, every position is computed in parallel with dense matrix multiplication. Between chunks, the state passes sequentially as before.

    NAIVE — one step per token, nothing overlaps

    t₁ ─▶ t₂ ─▶ t₃ ─▶ t₄ ─▶ t₅ ─▶ … ─▶ t₁₀₀₀₀₀₀
    └──── 1,000,000 sequential steps, GPU mostly idle ────┘


    CHUNKWISE — sequential between chunks, parallel within

    ┌─ chunk 1 ─┐   ┌─ chunk 2 ─┐   ┌─ chunk 3 ─┐
    │ t₁ t₂ …   │──▶│ t₁ t₂ …   │──▶│ t₁ t₂ …   │──▶ …
    │ all in    │ S │ all in    │ S │ all in    │ S
    │ parallel  │   │ parallel  │   │ parallel  │
    └───────────┘   └───────────┘   └───────────┘

    Each chunk's output has two parts:

      O = (Γ ⊙ Q) S        ← INTER-chunk: what arrived from the past,
          └─────────┘        one matmul against the incoming state
                    +
          A Ṽ              ← INTRA-chunk: interactions among tokens
          └───┘              inside this chunk, masked causal

There is a wrinkle, and it is the whole reason this lesson exists. To compute the intra-chunk term in parallel, the math needs each key rescaled by the reciprocal of its accumulated decay. Written as 1/Γ. Since Γ is a product of numbers below 1, it shrinks fast, and its reciprocal grows fast.

So the reciprocal rescale is a numerical hazard sitting in the middle of the fast path. Kimi Linear handled it by computing decay in log space and splitting each chunk into 16-token tiles. That helped, but only partly: the off-diagonal tiles became clean matmuls while the diagonal tiles still needed explicit position-by-position computation. The report names this as the main intra-chunk bottleneck.¹

One constant, one deleted bottleneck

K3's fix is to change how the decay logits become the actual decay. Kimi Linear used a mapping whose output was unbounded below, meaning a channel could in principle decay arbitrarily hard, sending 1/Γ arbitrarily high. K3 replaces it with a scaled sigmoid that cannot go below a fixed floor:

Now the payoff. e⁸⁰ is about 5.5 × 10³⁴, which fits inside BF16's ceiling of 3.4 × 10³⁸. Overflow becomes impossible by construction, so the diagonal tiles no longer need their special slow path. Every tile can be a dense Tensor Core matmul.

Drag both sliders. The margin is tighter than you might expect: leave the tile at 16 and move the decay floor to 6, or leave the floor at 5 and move the tile to 24. Either single step over the line is enough to break it.

Two things are worth noticing from playing with it. First, g_min and the tile size are co-designed, and the margin is thin. Overflow begins at a cumulative log-decay of about 88.7, and K3 sits at 5 × 16 = 80. One step in either direction, a floor of 6 or a tile of 24, puts it over. Neither constant was free to choose. Second, the choice is a genuine tradeoff rather than a free win. A floor on decay means a channel cannot forget instantly even when the model would like it to. K3 bought a fast kernel by giving up the ability to fully clear a channel in one step.

The third change: a full-rank output gate

Smaller, but it belongs to the same story of K3 spending a little capacity to buy quality. After the recurrent read, KDA applies a gate that decides which channels of the result actually pass through. Kimi Linear computed that gate with a low-rank projection, a cheap approximation. K3 makes it full-rank and input-dependent, so every token gets a genuinely independent say over all channels.¹ Gated MLA received the same upgrade, which is where its name comes from.

Gated MLA, the other quarter

Now the layer that handles what KDA cannot. One in four attention layers in K3 is a Gated MLA layer doing full global attention, with exact retrieval at any distance.

Full attention's problem is the KV cache from lesson 2: a key and a value stored per token, per head, forever. MLA's trick is to not store them.

    STANDARD ATTENTION CACHE          MLA CACHE

    per token, per head:              per token, ONE vector:
      [ k ]  [ v ]                      [ c ]   ← low-dim latent
      [ k ]  [ v ]                              cₜ = W_c · xₜ
      [ k ]  [ v ]   × 96 heads
      [ k ]  [ v ]                    at attention time:
        …      …                        k, v  ◀── up-project from c
                                                 for whichever head
                                                 needs them

    cache size ∝ n × heads × 2        cache size ∝ n × 1

Note what MLA does and does not fix. It shrinks the constant, so the cache is much smaller per token. It does not change the growth: the cache is still proportional to sequence length. That is exactly why K3 needs KDA in the other three quarters of its layers, and why MLA alone was not enough to reach a million tokens.

The NoPE bet

Here is the part of K3 most worth being able to explain to someone else, because it sounds wrong.

Attention has no inherent sense of order. Feed it a shuffled sentence and it computes the same thing, because it compares every query to every key without regard to where they sit. Transformers therefore inject position explicitly. The dominant method is RoPE, which rotates queries and keys by an angle proportional to position, so their match strength encodes how far apart they are.

K3 applies NoPE to every one of its MLA layers. No rotation, no learned position embeddings, nothing. The global attention layers are genuinely order-blind.

The argument for why this works is the cleanest illustration of what a hybrid architecture buys. KDA is a recurrence. It processes tokens in order and decays its memory as it goes, so position is baked into its structure: something read recently has had less decay applied than something read long ago. Position information does not need to be injected into the attention scores, because three out of every four layers already carry it inherently.¹

    DIVISION OF LABOUR ON POSITION

    KDA layers (69)                    MLA layers (24)
    ──────────────                     ──────────────
    position: implicit, from           position: none (NoPE)
      decay and recurrence order
    recency-aware by construction      order-blind by construction
    approximate retrieval              exact retrieval,
                                         unrestricted global reach
    ───────────────────────────────────────────────────────────
    between them, position is handled once and structurally,
    rather than encoded in every layer


    CONSEQUENCE FOR CONTEXT EXTENSION

      RoPE model:   8K ──▶ 1M  needs frequency rescaling (YaRN etc.),
                              retuning, and usually costs quality

      K3 (NoPE):    8K ──▶ 1M  no positional parameters exist,
                              so there is nothing to retune

The practical dividend appears in how K3 reaches a million tokens. Training runs a four-stage curriculum, 8K to 64K during pre-training, then 256K to 1M during cooldown. At no point does anyone rescale a frequency base or apply an interpolation scheme, because there are no positional parameters to touch. The report puts it directly: the model extrapolates directly to 1M-token contexts without any positional-encoding modification.¹

You have now seen K3's answer on the sequence axis. This lesson is the depth axis, and it rests on an observation the report makes in its first sentence on the subject: the residual stream is a bottleneck of exactly the kind attention was invented to fix.

The flaw in the residual stream

A transformer passes information up through depth by accumulation. Layer 1 writes into a running state, layer 2 adds to it, layer 3 adds to that. By layer 70, that state holds contributions from 69 earlier layers, all summed together.

Summing works, but it destroys information about origin. Layer 70 receives one vector. It cannot ask what layer 12 contributed, or weight the embedding differently from layer 69. Everything arrives pre-mixed, in fixed proportions the layer has no say over.

The report's framing is the useful one, and it is worth quoting because it tells you exactly where the idea came from: standard residuals compress all prior information into a single state h_l over depth — a bottleneck reminiscent of RNNs over time.¹

That comparison is the whole lesson. Recurrent networks had this exact problem along the sequence axis: everything read so far crushed into one hidden state, no way to reach back to a specific earlier token. Attention solved it by letting each position query all previous positions with learned, data-dependent weights. AttnRes applies that same move to depth.

    THE SAME BOTTLENECK, TWICE

    over SEQUENCE                        over DEPTH
    ─────────────────                    ──────────────

    RNN                                  Standard residual
      one hidden state carries              one residual stream carries
      everything read so far                everything computed so far
      ↓                                     ↓
      cannot reach token 12,345             cannot reach layer 12
      specifically                          specifically

    ── solved by attention ──            ── solved by AttnRes ──

    Transformer                          Attention Residuals
      each position queries all             each layer queries all
      previous positions, with              previous layers, with
      learned weights                       learned weights

    Lessons 2–3                          this lesson

How a layer queries depth

The mechanism is small. Each layer gets a learned pseudo-query, a single vector belonging to the layer rather than to any token. Every earlier layer's output serves as both key and value, with the token embedding included as the earliest source. The layer's query is compared against all of them, softmax turns the comparisons into weights, and the layer reads a weighted blend instead of a plain sum.

One detail in the report matters more than its size suggests. The keys pass through RMSNorm before the comparison, which the authors say prevents layers with large-magnitude outputs from dominating the weights.¹ Without it, a layer that happens to produce big numbers would win the attention competition on volume rather than relevance.

Why K3 does not ship the full version

Full AttnRes has a cost you can see without any math. If layer 70 attends over 69 earlier outputs, all 69 must still be in memory. Multiply by the size of each and by pipeline stages that need them shipped across devices, and the bill lands on memory and interconnect rather than arithmetic.

So K3 ships Block AttnRes. The 93 layers are partitioned into blocks of 12. Within a block, outputs are summed into one running representation, exactly as a standard residual would. Across blocks, full attention is applied over just the block-level summaries.

Switch between the three modes below and drag the layer slider. Watch the source count in each.

The shape to take away: a layer in the middle of the network reads from a handful of sources rather than one, and rather than seventy. Overhead falls from O(Ld) to O(Nd), and the report reports that N ≈ 8 recovers most of the benefit across model scales.¹

What it costs, honestly

Raschka's read of the report puts numbers on the overhead: roughly 4% in training cost and 2% in inference cost, for gains he describes as consistent but modest in validation loss and downstream performance.² A few percent for a small, reliable gain is an unglamorous trade, which is worth stating plainly.

That is a genuinely unglamorous return, and saying so is the honest read. It also explains why the block form exists at all: at full-AttnRes memory cost the trade would likely not be worth making, and the block approximation is what brings the price down to where a few percent buys a real if small gain.

A contrast worth carrying

K3 is not the only 2026 frontier model to decide the residual stream needed work, and the comparison sharpens what AttnRes actually is. DeepSeek V4 attacked the same bottleneck by making the residual path wider, using manifold-constrained hyper-connections. K3 instead kept the width and made the path selective across layers.²

    TWO ANSWERS TO ONE BOTTLENECK

    DeepSeek V4 — widen the pipe        Kimi K3 — choose what to read
    ┌───────────────────────────┐       ┌───────────────────────────┐
    │  more channels carrying   │       │  same channels, but each  │
    │  more information         │       │  layer weights its own    │
    │  forward                  │       │  sources across depth     │
    └───────────────────────────┘       └───────────────────────────┘
    cost: width, so parameters          cost: keeping earlier outputs
          and memory everywhere               reachable (bounded by N)

The third and last axis. Sequence was about mixing across tokens, depth about mixing across layers. Width is about mixing across channels, the dimensions inside a single token's representation, and it is where almost all of K3's 2.78 trillion parameters live.

By the end of this lesson you should be able to derive that 2.78T figure yourself from the config table, which is a better test of understanding than remembering it.

How routing works

A conventional feed-forward layer sends every token through the same network. An MoE layer holds many networks and sends each token to a few. Two kinds of expert sit side by side in K3.

The problem with a big expert pool

More experts sounds straightforwardly good: more room for specialisation, and since only 16 activate, the compute per token stays flat. So why does nobody just use 10,000?

Because compute is not the only cost. In a conventional MoE, each selected expert receives the token's full-width representation. All 7,168 numbers, copied to all 16 experts, which typically live on 16 different accelerators. The report names both consequences of naive expansion: communication and expert-weight traffic grow with the routing multiplicity.¹

The trick: separate model width from expert width

LatentMoE's move is to decouple two things that are normally the same number. The model works in 7,168 channels. The routed experts do not have to.

Before dispatch, the token is compressed by a down-projection to 3,584 channels, exactly half. The 16 selected experts do their work entirely in that smaller space. Their weighted output is normalised and expanded back to full width by an up-projection. Shared experts, meanwhile, keep the full-width path, because they run for every token and are few.¹

Now the payoff, which you can compute. Move the width slider and watch the total.

At K3's own setting the routed pool works out to roughly 2.72T. Push expert width to full d and that pool becomes 5.45T on its own, taking the whole model past 5.4T for the same number of experts and the same compute per token. Halving expert width halves almost the entire parameter count, because the routed pool is essentially the whole model.

Deriving the headline number

This is worth doing once by hand, because it converts a memorised fact into something you can reconstruct, and because the method transfers to any MoE config table you meet.

    ONE ROUTED EXPERT
      a GLU holds three matrices: gate, up, down
      3 × latent × hidden  =  3 × 3,584 × 3,072  =  33.0M params

    ONE LAYER'S ROUTED POOL
      33.0M × 896 experts  =  29.6B

    ALL MoE LAYERS
      29.6B × 92 layers    =  2.72T      ← 98% of the model

    PLUS
      W↓ and W↑ projections    2 × 7,168 × 3,584 × 92  =  4.7B
      shared experts           3 × 7,168 × 3,072 × 2 × 92  =  12.2B
      embedding + output       160K × 7,168 × 2  =  2.3B
      MoonViT-V2              0.4B
      attention (KDA + MLA)   the remainder

                             ─────────────────────────
                             report's total:  2.78T  ✓

The same arithmetic explains the activated figure. Sixteen experts at 33.0M each across 92 layers is 48.6B, plus 4.7B of always-on projections and 12.2B of shared experts, giving 65.5B from the feed-forward path. Attention and embeddings supply the remaining ~39B of the 104.2B total.

What extreme sparsity breaks

Sparsity of 56 is aggressive, and the report is candid that the vanilla design does not survive it. Two failure modes are named, and the word Stable in "Stable LatentMoE" refers to the fixes.¹

    FAILURE 1 — exploding activations

      The routed path chains four matrix multiplies back to back:

        W↓  ──▶  gate  ──▶  up  ──▶  W↑
                 └── the expert's GLU ──┘

      An ill-conditioned chain at 2.8T scale produces
      "exploding internal activations in the routed branch."

      Fixes:  RMSNorm before W↑        (§2.3.1, this lesson)
              SiTU-GLU bounded activation (§2.3.2, lesson 7)


    FAILURE 2 — load imbalance

      Balancing ~10³ experts per layer "exceeds the regime in which
      existing auxiliary-loss-free bias updates remain well behaved."

      Some experts overheat, others never train at all.

      Fix:    Quantile Balancing      (§2.3.3, lesson 7b)

The fix in this lesson is the smallest change in K3 and a good illustration of how much a normalisation placement can matter. In the original LatentMoE, the up-projection is applied directly to the aggregated routed output, whose scale varies depending on which experts were chosen and how strongly. K3 inserts an RMSNorm between aggregation and up-projection, so the routed branch arrives at a predictable scale before joining the full-width shared branch.

The report claims this earns its place twice over: Beyond stabilizing training, the additional RMSNorm consistently improves validation loss and downstream benchmarks.¹

Lesson 6 ended with the report's admission that extreme sparsity amplifies two failure modes. First, activations explode in the routed branch. Second, load balancing stops working at nearly a thousand experts per layer. The word Stable in Stable LatentMoE names the fixes for both. This lesson takes the first; lesson 7b takes the second, which is the better story.

The activation function family

Every modern transformer's feed-forward network uses a gated linear unit. Two projections of the input, one modulating the other by multiplication.

The problem the report identifies is simple once stated: both of SwiGLU's factors are unbounded. When two large numbers happen to land in the same coordinate, their product is much larger, and the report warns this can produce activation outliers and increase overflow risk in low-precision arithmetic.¹

Why K3 cares more than most: it applies quantization-aware training from the SFT stage onward, with MXFP4 weights and MXFP8 activations (SFT, supervised fine-tuning, is the first post-training stage; lesson 12 covers it).¹ A 4-bit float represents very few distinct magnitudes. One wild activation forces the scaling factor for an entire block to accommodate it, and everything else in that block loses precision. Outliers are far more damaging at 4 bits than at 16.

The soft cap

SiTU-GLU's answer is a soft cap: β·tanh(x/β), applied to both branches. The tanh function is almost exactly the identity near zero and flattens to ±1 far out, so scaling it by β gives a function that passes through the origin like x and saturates at ±β.

Switch between the two views. The near-origin view is the one that should surprise you.

Near the origin the two curves are indistinguishable, a deviation of 0.02% at x = 0.1. At x = 100 SwiGLU has reached 10,000 while SiTU-GLU has flattened at just under 100. Same function where it matters for learning, hard ceiling where it matters for precision.

Lesson 7 fixed exploding activations with a bounded activation function. This is the other failure mode extreme sparsity creates: keeping nearly a thousand experts per layer evenly loaded. It is the last piece of Stable LatentMoE, and the width axis is complete once it lands.

Why imbalance is expensive

The router learns which experts suit which tokens, and nothing stops it developing favourites. Left alone it will route heavily to a popular few while others receive almost nothing. Both ends of that distribution are bad.

    WHY IMBALANCE IS EXPENSIVE

    overloaded expert            starved expert
    ─────────────────            ──────────────
    its device becomes the       receives too few tokens
    slowest in the batch;        to learn anything useful;
    every other device waits     its parameters are dead weight

    "Imbalanced routing slows expert-parallel training and may
     leave some experts poorly trained."          — §2.3.3

The standard fix is to add a per-expert bias to the router's scores, used only for choosing experts and deliberately excluded from the mixture weights. That exclusion is the clever part of the existing method: the bias steers dispatch without touching the router's gradients, so balancing does not fight learning.

The question is how to set that bias. The method K3 inherited nudges it by a fixed step: if an expert is above target, push its bias down a little; if below, push it up. It works, and the report names its weakness precisely: the step size trades off slow adaptation against load oscillation. Too small and imbalance persists for thousands of steps; too large and loads swing back and forth. At 896 experts, the report says this exceeds the regime in which existing auxiliary-loss-free bias updates remain well behaved.¹

Try all three settings below. Start with no balancing to see the problem, then step the fixed-step method a few times, then switch to Quantile Balancing and press the button once.

The difference is not incremental. Fixed-step nudging is still badly imbalanced after six steps; Quantile Balancing drops imbalance by an order of magnitude in one, and revives every dead expert.

Why one step is enough

The insight is a reframing. Instead of asking "which direction should this bias move," QB asks "what bias would have produced exactly the target load?" That question has a direct answer, because it is a question about a quantile.

    THE REFRAMING

    fixed-step               quantile
    ──────────               ────────
    "expert 7 is too busy    "expert 7 should serve 64 tokens.
     — nudge it down a bit    Among all tokens' margins for expert 7,
     and see what happens"    the 64th largest is the threshold that
                              admits exactly 64. Set the bias there."

    iterative, tuned          direct, no step size to tune


    HOW THE MARGIN IS OBTAINED

    for each token, route with Top-(k+1) instead of Top-k:

        rank:  1    2    3   …   k   │  k+1
               ●    ●    ●       ●   │   ○
               └── actually used ──┘ │   └── the CUTOFF: the score an
                                     │       expert must beat to get in

    margin for expert j on token i  =  its score  −  that token's cutoff

    bias for expert j  =  −(the (1 − k/n) quantile of its margins)

What makes this more than a trick is where it comes from. Appendix C derives QB from the optimal balanced assignment problem: maximise total router score subject to every token getting exactly k experts and every expert getting exactly mk/n tokens. That is an integer program, but its linear relaxation is exact for this structure, and in the dual, the expert-side multipliers are the biases.¹ So QB approximates the solution to a problem whose optimum is known, which is a stronger footing than a heuristic that merely works in practice.

Everything so far has been the backbone. This lesson is what happens before it: how images and video enter the same token stream as text, and one decision that contradicts several years of accepted practice.

What "native" means here

Most models that handle images were language models first. A vision encoder gets attached afterwards, and a separate alignment stage teaches the two halves to talk. K3 does not work that way.

    THE CONVENTIONAL RECIPE               KIMI K3

    1. train a language model             1. train everything at once
       on text                               text + images interleaved,
                                              one next-token objective
    2. take a vision encoder
       pretrained contrastively           ┌────────────────────────┐
       (SigLIP, CLIP)                     │  MoonViT-V2            │
                                          │  trained from scratch  │
    3. bolt them together with a          │  with next-token       │
       projector                          │  prediction            │
                                          └───────────┬────────────┘
    4. run an alignment stage to                      │
       teach them to cooperate            ┌───────────▼────────────┐
                                          │   MLP projector        │
                                          └───────────┬────────────┘
                                                      │
                                          ┌───────────▼────────────┐
                                          │  shared backbone       │
                                          │  93 layers             │
                                          └────────────────────────┘

                                          no post-hoc alignment stage

The payoff the report emphasises is behavioural, not benchmark. Because rendered output and the code that produced it live in the same token stream, K3 can write code, look at a screenshot of the result, and revise, with no hand-off to another model. That loop is what makes the launch post's chart-building and slide-making demos possible.²

The decision that goes against the grain

Here is the part worth being able to explain to someone else.

Standard practice is to initialise a vision encoder from a model pretrained with a contrastive objective, typically SigLIP or CLIP. Kimi K2.5 did exactly this. The reasoning is intuitive: a contrastively pretrained encoder already knows a great deal about images, so the multimodal model starts ahead.

K3 trains MoonViT-V2 from scratch with next-token prediction instead. The report is direct about why, and the justification is about optimisation rather than accuracy: We depart from this practice primarily for training stability.¹

Two arguments are stacked here, and they are worth separating because only one of them is about stability.

The first is the stability finding above. Joint optimisation of a pretrained encoder and a language model fights itself: the encoder arrives with representations shaped by a different objective, and the gradient norms show the friction.

The second is about what the encoder learns. A contrastive objective rewards global semantics, which the report notes comes at a cost: it favors global semantics over fine-grained textual and structural cues. Next-token prediction shapes the encoder's representations by the objective the whole model is actually judged on. For K3's real workloads, reading text in a screenshot, judging whether a rendered chart looks right, fine-grained structure is the thing that matters.

How images become tokens

The mechanics matter for a practical reason: images are expensive in context, and K3's design decisions are visible in the arithmetic.

    ONE MAX-RESOLUTION IMAGE THROUGH MoonViT-V2

    3584 × 3584 pixels
          │
          │  patch size 14
          ▼
    256 × 256 = 65,536 patches            6.6% of a 1M context
          │
          │  pixel-shuffle, 2 × 2 downsample
          ▼
    128 × 128 = 16,384 tokens             1.6% of a 1M context
          │
          │  MLP projector
          ▼
    into the shared backbone


    Roughly 61 max-resolution images fit in a 1M-token context.
    Without the pixel-shuffle step, only 15 would.

Three more design choices, each small and each purposeful:

• RMSNorm, no bias terms anywhere. Removing biases from every linear and attention projection further stabilizes the from-scratch optimization. The same stability theme again.
• Images and video share all parameters. Attention is factorised into intra-frame spatial passes and inter-frame temporal passes, with temporal pooling compressing along time. One set of weights handles both.
• Coordinates supervised in two formats. Training provides both absolute pixel coordinates and normalised [0,1] coordinates, giving precise and resolution-robust localization. That is what lets the model point at things in a screenshot regardless of its size.

Making the encoder nearly free

One infrastructure detail is worth a mention, because it is the kind of engineering that decides whether native multimodality is affordable.

A vision encoder is a serial cost sitting in front of the backbone, and a large image or long video makes it worse by loading some devices far more than others. K3 attacks this by hiding the work in gaps that already exist.

    PIPELINE PARALLELISM HAS IDLE GAPS

    stage 1  ████░░░░░░░░████        ░ = bubble, device idle
    stage 2  ░░████░░░░████░░           waiting for other stages
    stage 3  ░░░░████████░░░░

    K3 schedules ViT work into those bubbles:

    stage 1  ████▓▓▓▓▓▓▓▓████        ▓ = vision encoder work
    stage 2  ░░████▓▓▓▓████░░
    stage 3  ░░░░████████░░░░

    Result: "most of the ViT computation is hidden within pipeline
             bubbles, largely eliminating the effective overhead of
             the vision encoder."                        — §5.2.3

A callback worth noticing

One neat connection to lesson 4. K3 uses speculative decoding to speed up generation, and its draft model needs to see the target model's internal features. Which features? The outputs of AttnRes blocks 1, 4, and the final one, concatenated and projected, giving low-, mid-, and high-level representations respectively.¹

Block AttnRes was introduced to cut memory overhead on the depth axis. A side effect is that it leaves clean, well-defined representations at block boundaries, and those turn out to be exactly what a draft model wants. The fusion matrix is even initialised as [0 0 I], so at the start it uses only the high-level feature and learns to incorporate the others gradually.

Architecture is only half of what determines whether a 2.8T model trains well. This lesson covers the optimiser and the empirical study that set K3's hyperparameters. Both are short sections in the report, and both contain a lesson that transfers well beyond K3.

What Muon does

Start with the optimiser everyone knows. Adam-family optimisers treat every weight as an independent scalar: track a running average of each weight's gradient, scale by a running average of its magnitude, step. Nothing in that procedure knows that the weights are arranged in a matrix.

Muon takes the arrangement seriously. Before applying an update to a matrix parameter, it orthogonalises the momentum matrix.

    WHY ORTHOGONALISE AN UPDATE

    raw momentum matrix              after orthogonalisation
    ───────────────────              ───────────────────────

      one direction dominates          all directions comparable
      ████████████████████  ◀── the    ██████
      ██                    update     ██████
      █                     is mostly  ██████
      █                     this row   ██████

      the step moves the weights       the step uses the full
      almost entirely one way          capacity of the matrix

The per-head change

Now K3's refinement, and it is worth reasoning through because the flaw it fixes is a general one.

Attention does not use one big matrix. It uses many heads, each reading and writing its own slice of the representation. K3 has 96 of them. But the query, key, and value projections are stored as single large matrices, with the heads sitting side by side inside them.

So when Muon orthogonalises the full Q, K, or V projection, it treats all 96 heads as one coupled block. The report spells out the consequence: heads with larger gradient or momentum scales dominate the shared update direction, while smaller-scale heads receive insufficiently normalized updates.¹

    FULL-MATRIX MUON                    PER-HEAD MUON

    ┌───────────────────────────┐       ┌─────┬─────┬─────┬─────┐
    │  h1 │ h2 │ h3 │ ... │ h96 │       │ h1  │ h2  │ h3  │ h96 │
    └───────────────────────────┘       └─────┴─────┴─────┴─────┘
                 │                         │     │     │     │
                 ▼                         ▼     ▼     ▼     ▼
        one orthogonalisation           orthogonalise each
        over everything                 head separately

    a loud head sets the scale          every head gets its own
    for all 96; quiet heads get         properly normalised update
    updates that are barely
    normalised at all

    ── and it is cheaper: Newton-Schulz on tall per-head blocks
       costs less than on the full projection matrix

The report also notes it slightly reduces optimizer overhead, since Newton–Schulz on tall thin per-head blocks is cheaper than on the full matrix. A quality improvement that also runs faster is rare enough to be worth noticing.

Scaling laws: where 2.5× comes from

Now the number quoted in every summary of K3, and what it actually means.

A scaling law is an empirical curve: train a family of small models at varying sizes and compute budgets, measure loss, and fit a relationship you can extrapolate. Its practical purpose is to choose hyperparameters for a run you can only afford to do once.

What the study retuned, because the architecture changes moved the optimum: batch size, learning rate, tokens-per-parameter ratio, and model shape.¹ That last one is why K3 has 93 layers and a 7,168 hidden dimension rather than some other combination. The shape was fitted, not guessed.

The finding that overturns received wisdom

The most interesting paragraph in §3.2 has nothing to do with 2.5×, and it is a lesson in experimental method rather than architecture.

Two learning-rate schedules compete in current practice. Cosine decay smoothly lowers the rate along a cosine curve. Warmup-Stable-Decay holds the rate constant for most of training, then drops it at the end, which is convenient because you can stop at any point. Recent work has reported WSD matching or beating cosine.

K3's study finds the opposite, and the explanation is the valuable part.

    THE METHODOLOGICAL TRAP

    the usual comparison                     what K3 did
    ────────────────────                     ───────────

    pick one set of hyperparameters          search for the optimal
              │                              hyperparameters for
              ▼                              EACH schedule separately
    run cosine ──▶ loss A                              │
    run WSD    ──▶ loss B                              ▼
              │                              cosine at its own optimum
              ▼                                    ──▶ loss A'
    declare a winner                         WSD at its own optimum
                                                   ──▶ loss B'
    but the two schedules have                        │
    "markedly different optimal                       ▼
    hyperparameters" — so this                 a fair comparison:
    just measures which schedule               cosine consistently wins
    the shared settings suited

This is a trap worth recognising in any paper you read. Whenever two methods are compared under one shared configuration, ask whether that configuration was tuned for one of them. It very often was.

K3's final recipe, for reference: cosine decay, 1% linear warmup, weight decay 0.1 throughout, Per-Head Muon plus K2's weight-clipping mechanism, Quantile Balancing for the experts. Context starts at 8K and reaches 64K in pre-training, then 256K to 1M during cooldown.¹

The architecture is behind you. This lesson is about running the thing: what it costs, why the price list has a tenfold gap in it, and what the hybrid attention design does to a serving stack. This is the material that decides whether K3 is right for a given job.

The price list, and the number that matters

    KIMI K3 API PRICING, per million tokens

    input, cache hit     $0.30   ◀── ten times cheaper
    input, cache miss    $3.00
    output              $15.00

The report gives the scenario that explains the pricing, and it is worth doing the arithmetic. In a typical coding session at long context, a typical coding input carries a prefix of 400K tokens but requires a prefill increment of only 4K tokens.¹

    ONE TURN OF A CODING SESSION
    400,000 tokens of prefix  +  4,000 new tokens

    cache miss    404,000 × $3.00/M           =  $1.21
    cache hit     400,000 × $0.30/M
                  + 4,000 × $3.00/M           =  $0.13     9.2× cheaper

    ACROSS A 50-TURN SESSION

    all misses                                =  $60.60
    with prefix caching                       =  $ 6.60
                                                 ───────
                                       saved     $54.00

    The launch post reports a cache hit rate above 90% in coding
    workloads, giving a blended input price near $0.57/MTok.

What the hybrid does to a cache

Here is where lessons 2 and 3 come due. K3 has two kinds of attention layer, and they need fundamentally different caches.

    TWO CACHES, ONE REQUEST

    Gated MLA layers (24)              KDA layers (69)
    ──────────────────────             ───────────────
    grows with sequence length         fixed size
    paged per token                    one state per request
    natural fine granularity           snapshots affordable only
                                        at sparse boundaries
    ───────────────────────────────────────────────────────────
    a cached prefix is reusable ONLY if BOTH can be
    restored at the SAME boundary

The naive resolution is to force one shared block size on both. That fails badly, and the failure is instructive. Because a KDA state is one large object per sequence rather than per-token entries, checkpointing it is expensive, so it can only be saved at sparse boundaries. Forcing MLA to match pushes the shared block size to 1024 to 6144 tokens, and the report's verdict on that is blunt: At such a coarse granularity caching is nearly useless: requests shorter than one block can never be reused, and chunked prefill exports no cacheable prefix until it crosses a full block boundary.¹

K3's fix is to stop insisting the two granularities match.

That last detail is the elegant part. Checkpoints are expensive, so K3 keeps the ones most likely to be reused: the boundaries between conversation turns, which is exactly where a follow-up request will resume.

Why 64 accelerators

Recall the derivation from lesson 6: nearly all of K3's 2.78T parameters sit in the routed expert pool. Those weights have to live somewhere, and each token's 16 chosen experts must be reachable quickly.

The requirement follows directly from the architecture. Total parameters set the memory floor, and dispatch traffic sets the interconnect floor. LatentMoE halves the second, and MXFP4 quantization shrinks the first, but 2.78T is 2.78T.

    QUANTIZATION-AWARE TRAINING

    MXFP4 weights  +  MXFP8 activations,  from the SFT stage onward

    trained AT the precision it is served at, not compressed after
              │
              └── which is why SiTU-GLU exists (lesson 7):
                  bounded activations are what make 4 bits survivable

This is the cleanest example in the whole report of architecture and deployment being designed together. SiTU-GLU's bound of 100 looks like an odd detail in §2.3.2. It is there because §4.1.4 decided to train in 4-bit.

Scheduling a fleet

Two production problems, both created by the million-token context, both with solutions worth knowing.

Cache affinity. If a hit is orders of magnitude cheaper than a miss, requests must be routed to the machine holding their cache, since moving a cache across clusters is slower than recomputing. But binding a session to one cluster means a cluster failure kills every session on it. K3 uses consistent hashing to pin each session to two clusters: a primary that serves it, and a secondary that takes over on failure and re-prefills from scratch. Because secondary assignments are spread uniformly, that recovery work lands across many clusters rather than one.¹

Traffic variance. Production mixes requests under 2K tokens with requests up to 1M, so per-request cost spans roughly three orders of magnitude. The failure mode is specific: a burst of long-context requests saturates compute, and short requests behind them wait, degrading time-to-first-token for everyone. Averages are useless here, so K3 gives each request class its own resource budget, letting long-context bursts consume at most their own share.¹

Where it actually lands

Cost only means something next to capability, so before the price comparison, the standing. The report's own one-line summary of §6.1.4 is worth taking at face value: K3 closely trails the strongest proprietary models, Claude Fable 5 and GPT-5.6 Sol, while consistently outperforming Claude Opus 4.8, GPT-5.5, and GLM-5.2 across the benchmark suite.¹

The shape to remember: K3 is at or near the top on agentic and document work, and behind on research-level reasoning. On HLE-Full it trails both leaders at 56.0% with tools and 43.5% without, and the authors name the weakness themselves rather than leaving you to find it: research-level reasoning remains a key direction for improvement.¹ That sentence is worth more than any single score, because it tells you which workloads to trial and which to leave alone.

Is it worth the money

Now the price. The report includes a cost-efficiency section, which is unusual and to its credit. The consistent finding: K3 trails the two strongest proprietary models on quality and beats them substantially on price.

The efficiency picture, revisited

You saw this simulator in lesson 2, when the question was theoretical. Now it is a serving question: this is the shape of what you pay for at long context.

Three quarters of K3's attention layers have flat per-token cost. That is why a million-token context is offered at all, and why the coding-session arithmetic above works out the way it does.

Last piece of new material. The architecture explains what K3 can compute; this explains what it actually does, and the three caveats that matter if you deploy it.

Three stages

    ① SFT                      cold start
      synthesised agent trajectories from earlier Kimi models,
      multi-stage verified, human-in-the-loop annotated,
      serialised in an XTML-based chat template

                    ▼

    ② RL                       specialise
      3 domains  ×  3 effort levels  =  9 expert models

      general tasks      ┐
      general agents     ├──  each trained at low / high / max
      coding agents      ┘

                    ▼

    ③ MOPD                     consolidate
      Multi-Teacher On-Policy Distillation:
      all nine experts distilled into ONE model that keeps
      each specialisation at each effort level

The choice worth noticing is that effort level is a trained axis, not a knob bolted on afterwards. Nine experts means each domain got its own model at each of low, high, and max. When you set effort on the API, you are selecting behaviour that was optimised at that budget.

How nine models become one

MOPD is conceptually simple once you see the reward. For each token the student generates, compare the probability the appropriate teacher would have assigned it against the student's own. The log-ratio is the reward, clipped to keep extremes from destabilising training.¹

    MULTI-TEACHER ON-POLICY DISTILLATION

    sample a domain d and an effort level e
              │
              ▼
    pick the matching teacher from the nine
              │
              ▼
    student generates a token  yₜ
              │
              ▼
    reward = clip( log [ π_teacher(yₜ) / π_student(yₜ) ] , ±R_max )
                        └── dense: one signal per token, not per task ──┘

    ON-POLICY means the student learns from its OWN outputs,
    so it improves where it actually goes wrong

One negative result reported in passing, and negative results are worth collecting: they also tried more fine-grained top-k distillation objectives and observed no clear advantage in either convergence speed or final performance. The simple log-ratio was enough.

What the RL environments teach

The environments are where K3's agentic behaviour is actually built. The report describes training a general loop of reasoning, acting, observing, verifying, and adapting, often over hundreds or thousands of tool calls and millions of accumulated context tokens.¹

Two infrastructure details make that loop trainable, and both are worth knowing because they explain what K3 is good at.

• Resumable microVM sandboxes. A trajectory spanning thousands of tool calls cannot restart from scratch when something interrupts it. The RL system keeps persistent rollout and environment state, with partial rollouts and external KV-cache retention.
• Verifiable environments. Tasks are constructed so success can be checked automatically, including knowledge-graph-guided task synthesis and a unified white-box RL environment. Without automatic verification, RL at this volume is not possible.

Three published limitations

The launch post includes a limitations section written against the team's own model. This is the most useful part of the release for anyone deciding whether to deploy, and the first one is a genuine operational trap.

    ① SENSITIVITY TO THINKING HISTORY                    ← the trap

      K3 was trained in preserved-thinking-history mode.
      If you do NOT pass its reasoning back on the next turn,
      or switch sessions mid-stream, quality
      "may become highly unstable."

      ▸ Practical: persist and replay the reasoning blocks.
        Dropping them to save tokens is the expensive kind
        of saving.


    ② EXCESSIVE PROACTIVENESS

      May "make unexpected decisions on the user's behalf."

      ▸ Practical: constrain scope explicitly in the system
        prompt or AGENTS.md. Assume it will act unless told
        where the boundaries are.


    ③ USER-EXPERIENCE GAP

      A "noticeable gap in user experience" versus Claude
      Fable 5 and GPT-5.6 Sol, beyond what benchmarks capture.

      ▸ Practical: benchmark scores will overstate how it
        feels to use. Trial it on your own work.

Judging K3

Your mission included being able to make a grounded call on where K3 fits. Here is the summary the evidence supports.