Review B · Width and optimisation

Same format as Review A: commit to an answer before revealing, score yourself honestly, and let the tally tell you where to go back. A few of these ask for arithmetic rather than recall, because being able to rebuild a figure is the difference between having read the paper and understanding it.

One note on the interleaving

Several drills below reach back into lessons 1–4 as well. Material you reviewed a week ago and have not touched since is exactly what benefits most from being asked again, and it is also what you are most likely to have quietly lost.

Free recall

What does LatentMoE decouple, and what two costs does that halve?

It decouples the model's channel width from the width its routed experts operate in. The backbone works in 7,168 channels; routed experts work in 3,584, exactly half. A down-projection compresses before dispatch, an up-projection expands after aggregation. This halves parameters (each expert is half the size) and dispatch traffic (each active expert receives half as many numbers across the interconnect). Both are quantities that scale with the expert pool, which is what makes 896 experts affordable. Shared experts keep the full-width path, since there are only two and they run for every token.

Why does Muon orthogonalise per head rather than over the whole Q, K, V matrix?

Because the 96 heads sit side by side inside one stored matrix, so a single orthogonalisation treats them as one coupled block. The report: heads with larger gradient or momentum scales dominate the shared update direction, while smaller-scale heads receive insufficiently normalized updates. Partitioning the momentum matrices along the head dimension gives each head its own properly scaled update. It also costs less, since Newton–Schulz on tall thin per-head blocks is cheaper than on the full projection.

Derive SiTU-GLU's output bound, and say why K3 cares about having one.

Each branch is capped by a scaled tanh, so the gate branch is bounded by β₁ = 4 and the up branch by β₂ = 25. Since |tanh| < 1 and 0 < σ < 1, the product cannot exceed β₁β₂ = 100. K3 cares because it applies quantization-aware training with MXFP4 weights and MXFP8 activations from the SFT stage onward. A 4-bit float represents very few magnitudes, so one outlier forces the whole block's scale to accommodate it and everything nearby loses precision. For contrast, SwiGLU at x = 100 reaches 10,000 and keeps climbing.

Rebuild K3's total parameter count from the config table. Show the steps.

One routed expert is a GLU, so three matrices: 3 × 3,584 × 3,072 ≈ 33.0M. One layer's pool: 33.0M × 896 ≈ 29.6B. Across the 92 MoE layers: 29.6B × 92 ≈ 2.72T, about 98% of the model. Add the W↓/W↑ projections (≈4.7B), shared experts (≈12.2B), embedding and output (≈2.3B), MoonViT-V2 (0.4B), and attention, and you reach the reported 2.78T. Note 92 MoE layers rather than 93, because the report lists one dense layer.

Why did K3 train its vision encoder from scratch, and what makes the finding notable?

For training stability. The SigLIP-initialised MoonViT-3D showed persistently higher gradient norms with frequent spikes; from-scratch MoonViT-V2 stayed flat. What makes it notable is that quality was equal: MoonViT-V2 matches the SigLIP-initialized baseline across vision evaluations, so the report concludes contrastive pretraining is unnecessary as an initialization for multimodal language models at scale. A negative result about accepted practice, which tells you what you can stop doing. There is also a second argument: contrastive objectives favour global semantics over the fine-grained textual and structural cues K3's real workloads depend on.

Explain Quantile Balancing's reframing, and why it converges so much faster.

Fixed-step balancing asks "which direction should this bias move, and by how much?", so it needs a tuned step size that trades off slow adaptation against load oscillation. QB asks "what bias would have produced the target load?" That is a question about a quantile, and it has a direct answer. Route with Top-(k+1) instead of Top-k; the (k+1)-th score is the cutoff an expert must beat. Each expert's margin is its score minus that cutoff, and its bias is minus the (1 − k/n) quantile of those margins. One step, no step size to tune, nothing to oscillate. Appendix C derives it from optimal balanced assignment via LP duality, where the expert-side dual multipliers are the biases.

What exactly does the 2.5× claim assert, and what are its three main caveats?

It asserts that K3 reaches a given validation loss for roughly 40% of the training compute K2 needed. Read Figure 7 horizontally, not vertically: it is a claim about compute efficiency, not about final quality or inference speed. Caveats: (1) it is the authors' own fit on their own held-out validation set against their own previous model, (2) nobody has replicated it, (3) it is undecomposed, bundling architecture with "refined data recipes", so no individual component can be credited. Safe phrasing: "the report claims a 2.5× scaling-efficiency gain, attributed to the architecture and data changes together."

A 3584×3584 image enters MoonViT-V2. How many tokens does it become, and why does that matter?

Patch size 14 gives (3584/14)² = 256² = 65,536 patches. Pixel-shuffle with 2×2 downsampling rearranges each 2×2 neighbourhood into one token with four times the channels, giving 128² = 16,384 tokens. It matters because that is the difference between a max-resolution image costing 1.6% of a 1M context instead of 6.6%, so roughly 61 such images fit rather than 15. Pixel-shuffle preserves the information by moving it into channels rather than discarding it.

Three K3 changes fix the same shape of mistake. Name them, and name the mistake.

The mistake is letting one statistic stand in for many things that differ. (1) Channel-wise decay in KDA: one scalar decay rate cannot express that different channels need different retention, so each channel gets its own α. (2) Attention Residuals: summing all earlier layers into one state loses the distinction between them, so each layer gets a learned query over depth. (3) Per-Head Muon: one orthogonalisation across 96 heads produces one shared update scale, so each head is orthogonalised separately. Recognising this pattern is more useful than any single mechanism, because it tells you where to look for improvements in an unfamiliar architecture.

Explain it to someone

An ML engineer asks how K3 can afford 896 experts per layer. Explain it in about a minute, including the trick and its cost.

Compute is not the obstacle, because only 16 of the 896 fire per token, so arithmetic per token stays flat no matter how large the pool. What actually scales with the pool is parameters to store and bytes to move: each active expert normally receives the token's full-width representation across the interconnect. LatentMoE breaks that by letting experts work at half the model width. A down-projection compresses the token before dispatch, experts operate in the smaller space, and an up-projection expands the result back. That halves both the per-expert parameter count and the dispatch traffic, which is what makes 896 affordable. The cost is that routed experts never see full width, and you pay for two extra projection matrices per layer. It is the same compress-to-a-latent instinct as MLA, applied to the expert pool instead of the KV cache.

Discriminating between mechanisms

MLA and LatentMoE both compress to a latent. What differs between them?

What is compressed: cached KV pairs versus token vectors
Only one compresses to half the model's channel width
Only one actually appears in the shipped architecture
Only one was adopted to improve quality, not cost

Same instinct, different bottleneck. MLA compresses each token's key–value pair to shrink the KV cache on the sequence axis. LatentMoE compresses each token's representation to shrink the expert pool and its dispatch traffic on the width axis. Noticing one architectural reflex applied in two places is the durable observation.

SiTU-GLU and Quantile Balancing both carry the "Stable" in Stable LatentMoE. Which is true?

One targets exploding activations, one uneven load
One works only in training, the other only at inference
One makes the other unnecessary at high sparsity
One cuts parameters, the other cuts compute per token

Two independent failure modes at extreme sparsity. SiTU-GLU bounds activations in the routed branch's four-matmul chain; Quantile Balancing keeps 896 experts evenly loaded. RMSNorm before the up-projection is the third fix, and the one from lesson 6.

Which K3 change is an optimiser change rather than an architectural one?

Per-Head Muon, which alters how updates are applied to weights
SiTU-GLU, which bounds the activation magnitudes inside experts
Quantile Balancing, which sets the routing biases from quantiles
Attention Residuals, which reach across depth to earlier layers

Per-Head Muon is the only one that lives outside the network. It changes nothing about what K3 computes, only how its weights move during training. Worth keeping the distinction sharp when reading a paper: architecture changes the function, optimiser changes the search for it.

Both KDA's bounded decay and SiTU-GLU's cap exist for numerical reasons. What differs?

One prevents reciprocal overflow, one product outliers
Only one uses a learned bound rather than a fixed one
Only one applies in training and is dropped at inference
Only one is motivated by low-precision arithmetic

Different hazards. Bounded decay (g_min = −5) keeps the chunkwise form's reciprocal rescale 1/Γ inside BF16, which unlocked all-Tensor-Core kernels. SiTU-GLU's cap keeps the product of two unbounded factors from producing outliers that wreck 4-bit quantization. Both are constants chosen to make a precision problem disappear by construction.

Read the paper yourself

Straight from §2.3, no commentary. Read it carefully, including for what is odd about it.

What is "nearly four consecutive matrix multiplications" counting? And what is wrong with "nearly 103 experts"?

The chain is W↓ → gate → up → W↑. Four matrices in sequence, and "nearly" because the gate and up branches run in parallel rather than strictly one after the other, so it is not quite a clean chain of four. The point is that composing this many projections without normalisation is ill-conditioned, which is why RMSNorm goes before W↑.

"Nearly 103 experts" is a rendering artifact. K3 has 896 experts, so 103 is wrong on its face. The original text is nearly 10³, meaning nearly a thousand. The superscript was flattened when the PDF was converted to text. This is worth internalising: extracted paper text loses superscripts, subscripts, and symbols constantly, and a number that makes no sense against the config table is usually a formatting casualty rather than an error by the authors. Always sanity-check figures against the table.

The whole architecture, one page

What comes next

The remaining two lessons leave the model and turn to using it. Lesson 11 is serving: what K3 costs per token, why the cache-hit price is ten times lower than cache-miss, what MXFP4 quantization means for deployment, and why the launch post asks for 64 accelerators. Lesson 12 is post-training and the three limitations the team published against themselves, which is where the "judge and deploy" half of your mission gets settled.