Kimi K3 · Lesson 1 of 14

The shape of K3

A 2.8-trillion-parameter model that runs 104 billion parameters at a time. One diagram explains where every innovation in the report belongs.

By the end of this lesson you should be able to draw the repeating block of Kimi K3 on a whiteboard and say what each piece is for. That drawing is the map for everything that follows. Every named innovation in the technical report sits somewhere on it.

The claim K3 is making

Open-weight models had stalled around the 1-trillion-parameter mark while proprietary labs kept going. The Kimi team's argument is that the open ecosystem was scaling the wrong axis: lots of work on reasoning and reinforcement learning, applied to pretrained foundations that all sat at roughly the same size.1 K3 pushes both at once: a 2.8T foundation and a million-token context with heavy RL on top.

The headline number: 2.78 trillion total parameters, 104.2 billion activated per token. Understanding that gap is the first real concept, and everything else depends on it.

Mixture-of-Experts (MoE)
An architecture where each layer holds many parallel feed-forward networks (experts), but each token is sent to only a few of them. A small router network scores the experts for that token and picks the top handful.
Total parameters
Every weight the model owns. This sets how much memory you need to hold it. For K3: 2.78T.
Activated parameters
The weights that actually take part in computing one token. This sets how much arithmetic each token costs. For K3: 104.2B.
Attention head
One independent read-and-write channel inside an attention layer. A layer runs many in parallel, each with its own queries, keys, and values, and combines the results. K3 has 96 per layer.

So K3 costs about as much compute per token as a 104B dense model, while carrying the stored knowledge of something 27 times larger. The tradeoff is memory and networking: you have to hold all 2.78T parameters somewhere, and route tokens to the right experts across many accelerators. This is why the launch post recommends supernode configurations with 64 or more accelerators.2

Why this matters for reading papers

Any MoE report gives you both numbers. The ratio between them is the single most useful thing on a config table: it tells you whether the model is expensive to run (activated) or expensive to host (total). K3 is deliberately cheap to run and expensive to host.

Three axes of information flow

The report organises the whole architecture around one idea, stated in its own words: K3 is designed to scale information flow along three complementary dimensions: sequence length, network depth, and model width.1

This framing is worth adopting, because it makes each component's job predictable before you know how it works. A token in a transformer needs to gather information from three places: other tokens, earlier layers, and its own channels. K3 has one named innovation for each.

Three axes of information flow in Kimi K3 Three axes radiate from a single point, one for each direction a token can gather information from. Upward is width, meaning channel mixing, handled by Stable LatentMoE with 16 of 896 experts per token, in section 2.3 and lessons 6 and 7. Rightward is sequence, meaning token mixing, handled by Hybrid Attention with three KDA layers to one Gated MLA layer, in section 2.1 and lessons 2 and 3. Down and to the left is depth, meaning layer mixing, handled by Attention Residuals so each layer retrieves from all earlier ones, in section 2.2 and lesson 4. WIDTH (channel mixing) Stable LatentMoE 16 of 896 experts per token §2.3 → Lessons 6, 7 SEQUENCE (token mixing) Hybrid Attention 3 × KDA + 1 × Gated MLA §2.1 → Lessons 2, 3 DEPTH (layer mixing) Attention Residuals each layer retrieves from all earlier ones §2.2 → Lesson 4
The organising idea. Three directions a token can gather information from, one headline mechanism for each. Section numbers refer to the technical report. Each axis keeps its colour for the rest of the course: blue for sequence, brown for depth, purple for width.

Two more components sit outside this scheme. MoonViT-V2 is the vision encoder at the input, making K3 natively multimodal rather than a language model with images bolted on (Lesson 8). Per-Head Muon is the optimiser. It sits outside the network proper, but it is part of why training a 2.8T model at this shape works at all (Lesson 9).

The repeating block

Now the drawing that matters. K3's backbone is 93 layers, built from a block that repeats. Each block contains four attention layers in a fixed pattern, and every attention layer is followed by its own MoE feed-forward network.

The repeating Kimi K3 block: three KDA layers to one Gated MLA layer One block, repeated 23 times. A residual stream runs up the left side, carrying state in from the previous block and out to the next. Three times, it feeds a KDA linear-attention layer whose output passes to a Stable LatentMoE feed-forward network of 16 active experts out of 896, which returns to the stream. Once per block, it feeds a Gated MLA global-attention layer using no positional encoding, followed by its own Stable LatentMoE network. Attention Residuals let each attention layer read every earlier layer's output rather than only the current stream. ONE BLOCK · repeated × 23 from previous block to next block Attention Residuals read every earlier layer Gated MLA global, NoPE Stable LatentMoE 16 of 896 experts × 1 KDA linear, O(n) Stable LatentMoE 16 of 896 experts × 3
The K3 block. Three linear-attention layers to one global-attention layer, a 3:1 ratio, each paired with a sparse feed-forward network. Blue is the sequence axis, purple the width axis. The residual stream up the left is what Attention Residuals generalise in lesson 4: each attention layer reads a learned mixture of every earlier layer, not just the stream as it stands. A single extra Gated MLA layer caps the backbone, so the last thing the model does is always full global attention.

Zoom out one level and that block sits in the middle of a short pipeline. Images and text both become token sequences before the backbone sees them, which is what natively multimodal means in practice.

The full Kimi K3 stack, input to output Two input streams merge into one backbone. The vision path runs MoonViT-V2 into an MLP projector; the text path is a token sequence. Both meet at a merge point and flow as one sequence through 23 repeated blocks, then a final Gated MLA global-attention layer, then the output. The backbone is 93 layers total: 69 KDA and 24 Gated MLA. THE FULL STACK MoonViT-V2 MLP projector text tokens 23 × BLOCK Gated MLA output final layer is always global 93 layers total = 69 KDA + 24 Gated MLA
Input to output. The vision encoder and its projector turn an image into tokens that join the text stream, so one backbone handles both. Lesson 8 takes MoonViT-V2 apart.

Check the arithmetic, because it is a good habit with config tables: 23 blocks × (3 KDA + 1 MLA) = 92 layers, plus the one trailing Gated MLA = 93. That gives 69 KDA and 24 MLA, matching the report's table exactly.1

The bet in one line

Only a quarter of K3's attention layers do the expensive thing. The other three quarters use a mechanism whose cost grows linearly with sequence length instead of quadratically, which is what makes a million-token context affordable. Lesson 2 is entirely about how that works.

Reading the config table

The report's Table 1 compares K3 against its predecessor K2 directly. This is the densest single page of information in the paper, so it is worth learning to read.

Table 1 of the technical report,1 abridged. Values in purple are new mechanisms rather than bigger numbers.
PropertyKimi K2Kimi K3What it tells you
Total parameters1.04T2.78THosting cost ↑167%
Activated parameters32.6B104.2BCompute per token ↑220%
Layers6193Depth ↑52%
Hidden dimension7,1687,168Unchanged — growth went elsewhere
Routed experts384896Specialisation capacity ↑133%
Experts active / token816Combinations per token ↑100%
Shared experts12Always-on capacity
Latent MoE dimension—3,584New: experts work at half width
Attention heads6496Parallel read channels ↑50%
Attention mechanismMLAHybrid KDA–MLANew: the 3:1 pattern
Activation functionSwiGLUSiTU-GLUNew: bounded, for low precision
Training context length128K1M8×
Vocabulary size160K160KUnchanged
Vision encoder—401M, 27 layersNew: native multimodal

Two things stand out once you know what to look for.

First, the hidden dimension did not move. K2 and K3 both work in 7,168 channels. All the width growth went into the expert pool, not the residual stream. That is a deliberate choice about where capacity is cheap.

Second, most of the interesting entries are new mechanisms, not larger numbers. Hybrid attention, latent experts, a different activation function. The report attributes the combined effect of these to a 2.5× improvement in overall scaling efficiency over Kimi K21, meaning K3 reaches a given validation loss for roughly 40% of the training compute K2 needed.

Hold this one loosely

The 2.5× figure comes from the authors fitting scaling-law curves on their own held-out validation data. No independent replication exists. It is a reasonable claim from a credible team, and it is also exactly the sort of number a lab has every incentive to present favourably. Cite it as "the report claims," not "K3 is 2.5× more efficient."

Check yourself

Answer from memory before scrolling back up. Getting one wrong is more useful than getting it right, because it marks what to reread.

A lab announces a 900B-parameter MoE with 40B activated. What does the gap tell you?

Total parameters set hosting cost, activated parameters set per-token compute. This is why K3 wants 64+ accelerators despite being compute-cheap for its size.

In one K3 block, what is the ratio of KDA layers to Gated MLA layers?

3:1, repeated through the backbone, with one extra Gated MLA at the very end so the final layer always performs global attention.

K3 grew from K2 on almost every axis. Which listed property stayed identical?

Hidden dimension held at 7,168 and vocabulary at 160K. Width grew through the expert pool instead of the residual stream, a bet that capacity is cheaper to add sparsely than densely.

What comes next

Lesson 2 opens the box on Kimi Delta Attention, the mechanism in three quarters of K3's attention layers. The core idea is a shift in how a model remembers: instead of keeping every past token and re-reading them all, KDA maintains a fixed-size memory it continuously fades and overwrites. That change is what makes a million-token context something you can afford.