Kimi K3 · Lesson 8 of 14 · Input pathway
K3 throws away the standard recipe for building a vision encoder. The reason has nothing to do with how well that recipe performs, and everything to do with what it does to training stability.
Everything so far has been the backbone. This lesson is what happens before it: how images and video enter the same token stream as text, and one decision that contradicts several years of accepted practice.
Most models that handle images were language models first. A vision encoder gets attached afterwards, and a separate alignment stage teaches the two halves to talk. K3 does not work that way.
THE CONVENTIONAL RECIPE KIMI K3
1. train a language model 1. train everything at once
on text text + images interleaved,
one next-token objective
2. take a vision encoder
pretrained contrastively ┌────────────────────────┐
(SigLIP, CLIP) │ MoonViT-V2 │
│ trained from scratch │
3. bolt them together with a │ with next-token │
projector │ prediction │
└───────────┬────────────┘
4. run an alignment stage to │
teach them to cooperate ┌───────────▼────────────┐
│ MLP projector │
└───────────┬────────────┘
│
┌───────────▼────────────┐
│ shared backbone │
│ 93 layers │
└────────────────────────┘
no post-hoc alignment stage
by a single shared backbone within one context, with no post-hoc modality-alignment stage.1
The payoff the report emphasises is behavioural, not benchmark. Because rendered output and the code that produced it live in the same token stream, K3 can write code, look at a screenshot of the result, and revise, with no hand-off to another model. That loop is what makes the launch post's chart-building and slide-making demos possible.2
Here is the part worth being able to explain to someone else.
Standard practice is to initialise a vision encoder from a model pretrained with a contrastive objective, typically SigLIP or CLIP. Kimi K2.5 did exactly this. The reasoning is intuitive: a contrastively pretrained encoder already knows a great deal about images, so the multimodal model starts ahead.
K3 trains MoonViT-V2 from scratch with next-token prediction instead. The report is direct about why, and the justification is about optimisation rather than accuracy: We depart from this practice primarily for training stability.
1
matches the SigLIP-initialized baseline across vision evaluations, indicating
contrastive pre-training is unnecessary as an initialization for multimodal language models at scale.
Two arguments are stacked here, and they are worth separating because only one of them is about stability.
The first is the stability finding above. Joint optimisation of a pretrained encoder and a language model fights itself: the encoder arrives with representations shaped by a different objective, and the gradient norms show the friction.
The second is about what the encoder learns. A contrastive objective rewards global semantics, which the report notes comes at a cost: it favors global semantics over fine-grained textual and structural cues.
Next-token prediction shapes the encoder's representations by the objective the whole model is actually judged on. For K3's real workloads, reading text in a screenshot, judging whether a rendered chart looks right, fine-grained structure is the thing that matters.
The finding is a negative result about accepted practice, and negative results are the most useful things in an architecture paper because they tell you what you can stop doing. If it holds up, an entire preparatory step in the standard multimodal recipe is optional at scale. Treat it as one team's ablation until someone replicates it, but note that they had no incentive to publish this particular conclusion.
The mechanics matter for a practical reason: images are expensive in context, and K3's design decisions are visible in the arithmetic.
ONE MAX-RESOLUTION IMAGE THROUGH MoonViT-V2
3584 × 3584 pixels
│
│ patch size 14
▼
256 × 256 = 65,536 patches 6.6% of a 1M context
│
│ pixel-shuffle, 2 × 2 downsample
▼
128 × 128 = 16,384 tokens 1.6% of a 1M context
│
│ MLP projector
▼
into the shared backbone
Roughly 61 max-resolution images fit in a 1M-token context.
Without the pixel-shuffle step, only 15 would.
affordable within the 1M-token context.1
Three more design choices, each small and each purposeful:
further stabilizes the from-scratch optimization. The same stability theme again.
precise and resolution-robust localization.That is what lets the model point at things in a screenshot regardless of its size.
The vision corpus deliberately scales programmatic data: code paired with the visual it renders, across SVG, 3D assets, webpages, games, and CAD schematics.1 This is training data purpose-built for the write-code-then-look-at-it loop. When you read that K3 tops WebDev Arena, this is a large part of why.
One infrastructure detail is worth a mention, because it is the kind of engineering that decides whether native multimodality is affordable.
A vision encoder is a serial cost sitting in front of the backbone, and a large image or long video makes it worse by loading some devices far more than others. K3 attacks this by hiding the work in gaps that already exist.
PIPELINE PARALLELISM HAS IDLE GAPS
stage 1 ████░░░░░░░░████ ░ = bubble, device idle
stage 2 ░░████░░░░████░░ waiting for other stages
stage 3 ░░░░████████░░░░
K3 schedules ViT work into those bubbles:
stage 1 ████▓▓▓▓▓▓▓▓████ ▓ = vision encoder work
stage 2 ░░████▓▓▓▓████░░
stage 3 ░░░░████████░░░░
Result: "most of the ViT computation is hidden within pipeline
bubbles, largely eliminating the effective overhead of
the vision encoder." — §5.2.3
One neat connection to lesson 4. K3 uses speculative decoding to speed up generation, and its draft model needs to see the target model's internal features. Which features? The outputs of AttnRes blocks 1, 4, and the final one, concatenated and projected, giving low-, mid-, and high-level representations respectively.1
Block AttnRes was introduced to cut memory overhead on the depth axis. A side effect is that it leaves clean, well-defined representations at block boundaries, and those turn out to be exactly what a draft model wants. The fusion matrix is even initialised as [0 0 I], so at the start it uses only the high-level feature and learns to incorporate the others gradually.
Why did K3 train its vision encoder from scratch rather than from SigLIP?
"We depart from this practice primarily for training stability." The SigLIP-initialised MoonViT-3D showed persistently higher gradient norms with frequent spikes. Quality was equal, which is what makes the conclusion strong: contrastive pretraining may simply be unnecessary at this scale.
What does the pixel-shuffle step accomplish before projection?
A 2×2 neighbourhood is rearranged into one token with four times the channels, so 65,536 patches become 16,384 tokens. That takes a max-resolution image from 6.6% of a 1M context down to 1.6%, which is what makes 3584×3584 inputs affordable.
What does contrastive pretraining favour that next-token prediction does not?
Contrastive training rewards matching an image to a caption, which is a global judgement. K3's real workloads depend on fine structure: reading text in a screenshot, judging whether a rendered chart is correct. Next-token prediction shapes representations by the objective the whole model is judged on.
✅ SEQUENCE KDA · chunkwise · g_min · Gated MLA · NoPE L2–L3
✅ DEPTH Block AttnRes · N = 8 · pseudo-queries L4
✅ WIDTH LatentMoE · SiTU-GLU · Quantile Balancing L6–L7
✅ INPUT MoonViT-V2 · from scratch · pixel-shuffle L8
▢ OPTIMISE Per-Head Muon · scaling law L9
▢ REVIEW B interleaves 6–9 L10
▢ SYSTEM serving · cache · quantization L11
▢ BEHAVIOUR post-training · effort levels · limitations L12
Lesson 9 is the last piece of the model itself: Per-Head Muon and the scaling-law study. The optimiser change is small and the reasoning behind it is a good lesson in why averaging over things that should not be averaged is a recurring mistake. The scaling-law section is where the claimed 2.5× improvement comes from, and it is also where the report quietly overturns a piece of conventional wisdom about learning-rate schedules.