Lesson 6 · LatentMoE

The third and last axis. Sequence was about mixing across tokens, depth about mixing across layers. Width is about mixing across channels, the dimensions inside a single token's representation, and it is where almost all of K3's 2.78 trillion parameters live.

By the end of this lesson you should be able to derive that 2.78T figure yourself from the config table, which is a better test of understanding than remembering it.

How routing works

A conventional feed-forward layer sends every token through the same network. An MoE layer holds many networks and sends each token to a few. Two kinds of expert sit side by side in K3.

Shared and routed paths (§2.3, eq. 11), following the DeepSeekMoE organisation. The heavy bars are fork and join: both paths run for every token, and the layer output is the sum of the two. Shared experts handle transformations every token needs, so the routed experts do not each have to relearn them. The dashed border marks the one box where only a selected subset runs, and the routed path is where the parameter count lives.

Router: A small network scoring each expert for the current token. K3 computes Sigmoid(W_r x) and takes the top 16.
Top-k routing: Selecting the k highest-scoring experts. K3's k is 16, out of 896.
Sparsity ratio: Total routed experts divided by active ones. K3: 896 / 16 = 56. Each token touches roughly one fifty-sixth of the routed capacity.

The problem with a big expert pool

More experts sounds straightforwardly good: more room for specialisation, and since only 16 activate, the compute per token stays flat. So why does nobody just use 10,000?

Because compute is not the only cost. In a conventional MoE, each selected expert receives the token's full-width representation. All 7,168 numbers, copied to all 16 experts, which typically live on 16 different accelerators. The report names both consequences of naive expansion: communication and expert-weight traffic grow with the routing multiplicity.¹

WHAT SCALES WITH WHAT (E = expert count, k = experts per token)
Cost	Conventional MoE	K3 LatentMoE
compute per token	flat in E	flat in E
parameters	E × full width	E × latent width
dispatch traffic	k × 7,168 = 114,688 floats	k × 3,584 = 57,344 floats

Latent width halves everything in this table that scales with E.

The bottleneck is not arithmetic. It is parameters to store and bytes to move, both of which scale with expert width. That is the quantity LatentMoE attacks.

The trick: separate model width from expert width

LatentMoE's move is to decouple two things that are normally the same number. The model works in 7,168 channels. The routed experts do not have to.

Before dispatch, the token is compressed by a down-projection to 3,584 channels, exactly half. The 16 selected experts do their work entirely in that smaller space. Their weighted output is normalised and expanded back to full width by an up-projection. Shared experts, meanwhile, keep the full-width path, because they run for every token and are few.¹

Why this is the same idea as MLA

You have met this move already. MLA compresses each token's key–value pair into a low-dimensional latent to shrink the KV cache. LatentMoE compresses each token's representation into a low-dimensional latent to shrink the expert pool. Raschka points out the family resemblance directly, noting LatentMoE compresses large linear layers in the spirit of multi-head latent attention.² One architectural instinct, applied in two places.

Now the payoff, which you can compute. Move the width slider and watch the total.

At K3's own setting the routed pool works out to roughly 2.72T. Push expert width to full d and that pool becomes 5.45T on its own, taking the whole model past 5.4T for the same number of experts and the same compute per token. Halving expert width halves almost the entire parameter count, because the routed pool is essentially the whole model.

Deriving the headline number

This is worth doing once by hand, because it converts a memorised fact into something you can reconstruct, and because the method transfers to any MoE config table you meet.

    ONE ROUTED EXPERT
      a GLU holds three matrices: gate, up, down
      3 × latent × hidden  =  3 × 3,584 × 3,072  =  33.0M params

    ONE LAYER'S ROUTED POOL
      33.0M × 896 experts  =  29.6B

    ALL MoE LAYERS
      29.6B × 92 layers    =  2.72T      ← 98% of the model

    PLUS
      W↓ and W↑ projections    2 × 7,168 × 3,584 × 92  =  4.7B
      shared experts           3 × 7,168 × 3,072 × 2 × 92  =  12.2B
      embedding + output       160K × 7,168 × 2  =  2.3B
      MoonViT-V2              0.4B
      attention (KDA + MLA)   the remainder

                             ─────────────────────────
                             report's total:  2.78T  ✓

Where 2.78 trillion comes from. 92 MoE layers, not 93, because the report lists one dense layer. The routed expert pool accounts for about 98% of the parameter count on its own.

The same arithmetic explains the activated figure. Sixteen experts at 33.0M each across 92 layers is 48.6B, plus 4.7B of always-on projections and 12.2B of shared experts, giving 65.5B from the feed-forward path. Attention and embeddings supply the remaining ~39B of the 104.2B total.

show me the actual equation

The layer, from §2.3 eq. 11:¹

u = Σ p_i · E_i^routed( W↓ x ) ∈ ℝ^ℓ i ∈ T_k(x) N_s y = Σ E_j^shared(x) + W↑ RMSNorm(u) ∈ ℝ^d j=1

W↓ x compresses the token from d = 7,168 to ℓ = 3,584 before dispatch, so the smaller vector is what crosses the interconnect.
E_i^routed : ℝ^ℓ → ℝ^ℓ — routed experts map latent space to latent space. They never see full width.
p_i — the router weight for expert i, normalised over the selected set.
T_k(x) — the selected top-16.
E_j^shared : ℝ^d → ℝ^d — shared experts, full width, N_s = 2, always active.
W↑ RMSNorm(u) — normalise, then expand back to d. The RMSNorm placement is the subject of the next section.

Router scores and weights, from §2.3.3 eq. 13:

s_i = Sigmoid( W_r x_i ) T_i = argtopk( s_i + b ) ← bias b affects selection only p_{i,j} = s_{i,j} / Σ_{r∈T_i} s_{i,r} ← bias absent from the weights

That the bias b steers dispatch but never enters p is the design detail that makes load balancing possible without disturbing the router's gradients. Lesson 7b is about how b is set.

What extreme sparsity breaks

Sparsity of 56 is aggressive, and the report is candid that the vanilla design does not survive it. Two failure modes are named, and the word Stable in "Stable LatentMoE" refers to the fixes.¹

    FAILURE 1 — exploding activations

      The routed path chains four matrix multiplies back to back:

        W↓  ──▶  gate  ──▶  up  ──▶  W↑
                 └── the expert's GLU ──┘

      An ill-conditioned chain at 2.8T scale produces
      "exploding internal activations in the routed branch."

      Fixes:  RMSNorm before W↑        (§2.3.1, this lesson)
              SiTU-GLU bounded activation (§2.3.2, lesson 7)


    FAILURE 2 — load imbalance

      Balancing ~10³ experts per layer "exceeds the regime in which
      existing auxiliary-loss-free bias updates remain well behaved."

      Some experts overheat, others never train at all.

      Fix:    Quantile Balancing      (§2.3.3, lesson 7b)

Two problems, three fixes. This lesson covers the first fix; lessons 7 and 7b cover the other two, which are the more interesting engineering.

The fix in this lesson is the smallest change in K3 and a good illustration of how much a normalisation placement can matter. In the original LatentMoE, the up-projection is applied directly to the aggregated routed output, whose scale varies depending on which experts were chosen and how strongly. K3 inserts an RMSNorm between aggregation and up-projection, so the routed branch arrives at a predictable scale before joining the full-width shared branch.

The report claims this earns its place twice over: Beyond stabilizing training, the additional RMSNorm consistently improves validation loss and downstream benchmarks.¹

Check yourself

What does LatentMoE actually decouple that is normally one number?

The model's channel width from the width its routed experts use
The count of experts per layer from the count active per token
The compute cost per token from the total size of the expert pool
The shared expert pathway from the routed expert pathway

The model works in 7,168 channels; routed experts work in 3,584. Compress before dispatch, expand after aggregation. Halving expert width roughly halves the parameter count and the dispatch traffic, since both scale with expert width.

Why can't a lab simply raise the expert count to 10,000 and keep compute flat?

Parameters to store and bytes to move both scale with the pool
Arithmetic per token would rise in proportion to the expert count
The router itself becomes the dominant cost at that expert count
Each token would receive too little capacity to compute well

Compute stays flat, which is exactly why the other costs bind. Every expert's weights must be stored, and every active expert receives a copy of the token across the interconnect. Balancing thousands of experts is the third limit, and lesson 7b's subject.

Roughly what share of K3's 2.78T parameters sits in the routed expert pool?

Nearly all of it, about 2.72T across the 92 MoE layers
About half, with attention layers holding a comparable share
Around a third, with shared experts holding most of the rest
A small minority, with embeddings dominating the count

33.0M per expert × 896 experts × 92 layers ≈ 2.72T, roughly 98% of the model. When you read that K3 is a 2.8T model, you are essentially reading a fact about its expert pool.

Where we are

    ✅ SEQUENCE   KDA · chunkwise · g_min · Gated MLA · NoPE      L2–L3
    ✅ DEPTH      Block AttnRes · N = 8 · pseudo-queries          L4
    ◐  WIDTH      LatentMoE ✅ · SiTU-GLU ▢ · Quantile Bal. ▢     L6–L7
    ▢  INPUT      MoonViT-V2 native vision                       L8
    ▢  OPTIMISE   Per-Head Muon · scaling law                    L9

Course progress. The width axis is part-done: LatentMoE is in place, and the two stability fixes that complete it are next.

Lessons 7 and 7b finish the width axis with the two remaining stability fixes, and they are the most satisfying engineering in the architecture. SiTU-GLU replaces the activation function every modern transformer uses, because unbounded activations are dangerous when you train in 4-bit. Quantile Balancing solves the load-balancing problem by reframing it as a question about quantiles, which turns a fiddly tuned heuristic into something that just computes the right answer.