Reference · Kimi K3 glossary

Terms are added only once they have been taught and used correctly. This file grows with the course. It covers all 13 lessons.

Attention fundamentals

Query, key, value q, k, v: Three vectors a model derives from each token. The query states what the token wants to know, the key advertises what it offers, and the value is the content handed over when a query matches a key.
KV cache: The stored keys and values of all previous tokens, kept so they need not be recomputed. Grows with context length, which is the cost full attention pays.; Attention cache, context cache
Linear attention: Attention that replaces the growing KV cache with a fixed-size state updated once per token, making per-token cost constant. KDA is K3's variant.; Sub-quadratic attention, efficient attention
Recurrent state: The fixed-size memory a linear-attention layer carries from token to token. In KDA, a matrix of shape (key dim × value dim) per head.; Memory buffer. ("Hidden state" is fine for RNNs, which is the contrast KDA is drawn against.)
Delta rule: Writing to an associative memory by first subtracting what the memory currently returns for that key, so a write replaces rather than accumulates. The "delta" in Kimi Delta Attention.
Forget gate: A learned multiplier below 1 applied to a recurrent state to decay old information. KDA's is channel-wise: every channel gets its own rate, rather than one rate for the whole state.; Decay gate, retention gate
Chunkwise parallel form: Computing a recurrence sequentially across fixed-size chunks while computing all positions within a chunk in parallel. This is what makes a per-token recurrence trainable on a GPU.; Chunked scan, block-parallel recurrence
Lower-bounded decay g_min: Constraining log-decay to a fixed floor, in K3 to (−5, 0), so the reciprocal rescale in the chunkwise form cannot overflow BF16. This is what allows every tile to use a dense Tensor Core matmul.; Clamped decay, bounded gate
BF16: The 16-bit float format most large models train in. Wide range, low precision, ceiling around 3.4 × 10³⁸.
Tensor Core: Dedicated matrix-multiply hardware on modern GPUs. Dense matmuls run on it; branching special cases do not, which is why eliminating the diagonal-tile path mattered.
RoPE: Rotary Position Embedding. Injects position by rotating queries and keys by a position-dependent angle. Extending context usually requires rescaling its frequencies. K3 uses none.; Rotary embeddings, positional rotation

Scale and sparsity

Mixture-of-Experts MoE: An architecture where a layer holds many parallel feed-forward networks and each token is computed by only a few of them, chosen by a router.; Sparse model, expert model
Total parameters: Every weight the model owns. Determines memory needed to host it. K3: 2.78T.; Model size, parameter count
Activated parameters: The weights that participate in computing a single token. Determines arithmetic cost per token. K3: 104.2B.; Effective parameters, live parameters
Sparsity ratio: Routed experts divided by experts active per token. K3: 896/16 = 56. Higher means each token sees a smaller slice of the model's stored capacity.; Sparsity factor, expert ratio
Router: The small network that scores every expert for a token and selects the top-k. In K3 it computes Sigmoid(W_r x).; Gate, gating network, dispatcher
Shared expert: An expert every token passes through unconditionally, holding transformations common to all inputs. K3 has 2 per layer, at full width.; Always-on expert, dense expert
Routed expert: An expert a token reaches only if the router selects it. K3 has 896 per layer, operating at latent width.; Conditional expert, sparse expert

Architecture

Hidden dimension d: The width of the representation carried between layers. K3: 7,168, unchanged from K2.; Model dimension, embedding size, d_model
Residual stream: The running representation each layer reads from and writes back to as a token passes through depth.; Skip path, residual path
Hybrid attention: Interleaving two different attention mechanisms by layer. K3 uses 3 KDA layers to 1 Gated MLA layer per block.; Mixed attention, layered attention
Kimi Delta Attention KDA: K3's linear-attention layer: a fixed-size memory matrix updated per token with channel-wise decay and a delta-rule write. Cost grows linearly with sequence length. Used in 69 of 93 layers.; Delta attention, linear attention layer
Multi-head Latent Attention MLA: Global attention that caches a single low-dimensional latent vector per token instead of full per-head keys and values, reconstructing them on demand. Introduced in DeepSeek-V2.; Latent attention, compressed attention
Gated MLA: K3's MLA variant, adding an input-dependent full-rank output gate so each token can modulate which channels it reads from global attention.
Attention Residuals AttnRes: A mechanism letting each layer selectively retrieve representations from the embedding and all preceding blocks using learned pseudo-queries, rather than reading one uniformly accumulated residual.; Cross-layer attention, depth attention, hyper-connections
LatentMoE: An MoE design where routed experts operate in a compressed latent space of width ℓ while shared experts keep full width, making a large expert pool affordable. K3: ℓ = 3,584 = 0.5 d.; Compressed MoE, low-rank MoE
SiTU-GLU: K3's activation function. Applies a smooth β·tanh(x/β) cap to both branches of a gated linear unit, bounding output magnitude while tracking SwiGLU near the origin. K3: β₁ = 4, β₂ = 25.; Sigmoid tanh activation, capped SwiGLU
Quantile Balancing QB: K3's load-balancing method. Sets each expert's routing bias from the router-score quantile matching its target load, replacing fixed-step bias nudges.; Quantile routing, quantile load balancing
No Position Encoding NoPE: Applying no explicit positional signal to queries or keys. K3 uses NoPE in every MLA layer, relying on KDA's recurrence to carry position implicitly.; Position-free attention, implicit positions
Per-Head Muon: K3's optimiser for attention projections. Partitions Q, K, V momentum matrices by head and orthogonalises each head's block separately, equalising update scale across heads.
MoonViT-V2: K3's vision encoder. 401M parameters, 27 layers, trained from scratch with next-token prediction rather than initialised from a contrastive model.; Vision tower, image encoder

Systems and serving

Prefix cache: Stored intermediate state for a conversation's earlier tokens, letting a follow-up request skip recomputing them. A hit reuses it; a miss reprocesses the whole prefix.; Context cache, prompt cache
Prefill: Processing the input prompt before generation begins. Dominates cost at long context, and is what a prefix-cache hit avoids.
Expert parallelism: Splitting an MoE layer's experts across devices, routing each token to wherever its chosen experts live.
Supernode: A tightly coupled group of accelerators with high-bandwidth interconnect. K3's launch post recommends 64 or more.
MXFP4 / MXFP8: Micro-scaled 4-bit and 8-bit float formats. K3 trains with MXFP4 weights and MXFP8 activations from the SFT stage onward, which is why SiTU-GLU's bound exists.
Quantization-aware training QAT: Training at the precision the model will be served at, rather than compressing a higher-precision model afterwards.; In-training quantization
Reasoning effort: K3's inference-time setting controlling how much test-time computation it spends. K3 supports low, high, and max; the template schema reserves a fourth, medium, which K3 does not use. It uses max by default (low and high modes are promised in later updates), and all reported evaluations use max.; Thinking budget. ("Effort level" is fine as a plain descriptor, e.g. "three effort levels".)
Newton–Schulz iteration: A cheap iterative approximation to orthogonalisation using only matrix multiplications. What makes Muon practical.
Muon: An optimiser that orthogonalises the momentum matrix before each update to matrix-shaped parameters. K3 uses a per-head variant.
Pixel-shuffle: Rearranging a 2×2 neighbourhood of patches into one token with four times the channels, cutting visual token count fourfold without discarding information.
Contrastive pretraining: Training an image encoder by pulling matched image–text pairs together and pushing mismatched pairs apart. Favours global semantics. K3 declines to use it as initialisation.

Mathematical vocabulary

Operations and terms the report uses without explanation. You do not need to compute any of these, but reading §2 unaided means knowing what each one does.

softmax: Turns a list of arbitrary scores into positive weights that sum to 1, exaggerating the gaps so the largest score dominates. Every "attention weight" in this course is a softmax output.
RMSNorm: Rescales a vector so its typical element size is about 1, without shifting it. Used whenever one quantity must not dominate another purely by being numerically larger. K3 applies it to AttnRes keys, KDA's recurrent output, and the aggregated routed expert output.; Layer norm (a different operation: it also re-centres)
attention head: One independent read-and-write channel inside an attention layer. A layer runs many in parallel, each with its own queries, keys, and values, and concatenates the results. K3 has 96 per layer.; Attention channel, head unit
gradient: For each weight, the direction and amount that changing it would reduce the loss. Training repeatedly nudges weights along their gradients. A gradient norm is the overall size of that update signal, which is why a spiking gradient-norm plot signals unstable training.
momentum (matrix): A running average of recent gradients, used so updates follow a consistent direction rather than reacting to each batch. For a matrix-shaped parameter, the momentum has the same shape, which is what lets Muon operate on it as a matrix.
rank, low-rank, full-rank: How many genuinely independent directions a matrix can express. A low-rank projection is a cheap approximation built from far fewer directions than its size allows; full-rank means no such restriction. K3 upgraded KDA's and MLA's output gates from low-rank to full-rank.
outer product: Multiplying a column vector by a row vector to produce a matrix. It is how a single key–value pair is written into KDA's memory matrix: k v^T stores the association between them.
logit: A raw, unbounded score before it is squashed into a probability or a bounded range. KDA's decay logits z become retention factors α; the router's scores are logits before the sigmoid.
quantile: The value below which a given fraction of a set falls. The 0.9 quantile of a list is the point 90% of it sits under. Quantile Balancing sets each expert's bias from the quantile of router-score margins matching its target load.; Percentile (same idea on a 0–100 scale)
coefficient of variation CV: Standard deviation divided by the mean, so it measures spread relative to size and has no units. Used in the load-balancing widget: CV near 0 means every expert gets a similar number of tokens, CV above 1 means wild imbalance.
ill-conditioned: A computation where small changes in the input produce disproportionately large changes in the output, so errors and magnitudes amplify. Chaining four projections without normalisation between them is ill-conditioned, which is the first failure mode Stable LatentMoE fixes.
linear programming duality LP duality: Every constrained optimisation problem of this type has a partner ("dual") problem whose solution carries a price for each constraint. For balanced expert assignment, the prices attached to the per-expert load constraints are the router biases, which is what makes Quantile Balancing principled rather than a heuristic.
FLOPs: Floating-point operations: a count of arithmetic done, used as the unit of training compute. The scaling-law curves plot loss against FLOPs.
shard, all-reduce: To shard is to split one large tensor across many devices so each holds a slice. An all-reduce combines a value held on every device (usually by summing) and gives every device the total. Quantile Balancing needs one all-reduce to pool its histogram counts.
pipeline parallelism: Splitting a model's layers across devices so each holds a consecutive stage, with batches flowing through like an assembly line. It creates idle bubbles while stages wait, and it means passing data between non-adjacent layers costs cross-device communication. Both facts shape Block AttnRes and the vision encoder's scheduling.
Elo: A relative rating from head-to-head comparisons, borrowed from chess. Only the differences carry meaning: roughly 100 points implies the higher-rated side wins about 64% of matchups. Used by Arena leaderboards and GDPval-AA.

Post-training

Supervised fine-tuning SFT: Training on curated examples of desired behaviour, after pretraining and before RL. Establishes a starting policy good enough for reinforcement learning to improve on.
Multi-Teacher On-Policy Distillation MOPD: K3's consolidation step. Nine RL experts (3 domains × 3 effort levels) act as teachers; the student is rewarded per token by the clipped log-ratio of teacher to student probability. On-policy means the student learns from its own generations, so it improves where it actually errs.
Verifiable environment: An RL task whose success can be checked automatically, without a human judging it. A precondition for reinforcement learning at K3's volume.
Thinking history: The reasoning tokens a model produced on earlier turns. K3 was trained with these preserved and replayed, so discarding them between turns is a correctness problem rather than a saving.

Ambiguities resolved for this course

"Block": Two different groupings share this word in the report. In this course, block always means the attention block — the repeating unit of 3 KDA + 1 Gated MLA. The 12-layer partitions used by Attention Residuals are called AttnRes blocks. K3 has 23 blocks and 8 AttnRes blocks; they do not align.
"Attention": Used in three senses: over tokens (KDA, MLA), over layers (AttnRes), and over experts (routing, loosely). Always name the axis unless context makes it unambiguous.
"Gate": Reserved for multiplicative modulation of a signal — KDA's output gate, SiTU-GLU's gate branch. The MoE selection network is called the router, never the gate, even though much of the literature uses "gating network."
"Effort": Reasoning effort is K3's inference-time setting (low / medium / high / max) controlling how much test-time computation it spends. Distinct from training compute. K3 defaults to max.