Kimi K3 · Lesson 11 of 14 · Systems

Serving K3

Every architectural choice in the preceding lessons shows up on an invoice. This is where the hybrid design gets awkward, and where it pays.

The architecture is behind you. This lesson is about running the thing: what it costs, why the price list has a tenfold gap in it, and what the hybrid attention design does to a serving stack. This is the material that decides whether K3 is right for a given job.

The price list, and the number that matters

    KIMI K3 API PRICING, per million tokens

    input, cache hit     $0.30   ◀── ten times cheaper
    input, cache miss    $3.00
    output              $15.00
From the launch post.2 Most providers price cached input lower; the size of the gap here is the thing to notice.
Prefix cache
Stored intermediate state for a conversation's earlier tokens, so a follow-up request need not recompute them. A hit reuses it; a miss means processing the whole prefix again.
Prefill
Processing the input prompt before generating anything. For a long context this dominates cost, and it is what a prefix-cache hit avoids.

The report gives the scenario that explains the pricing, and it is worth doing the arithmetic. In a typical coding session at long context, a typical coding input carries a prefix of 400K tokens but requires a prefill increment of only 4K tokens.1

    ONE TURN OF A CODING SESSION
    400,000 tokens of prefix  +  4,000 new tokens

    cache miss    404,000 × $3.00/M           =  $1.21
    cache hit     400,000 × $0.30/M
                  + 4,000 × $3.00/M           =  $0.13     9.2× cheaper

    ACROSS A 50-TURN SESSION

    all misses                                =  $60.60
    with prefix caching                       =  $ 6.60
                                                 ───────
                                       saved     $54.00

    The launch post reports a cache hit rate above 90% in coding
    workloads, giving a blended input price near $0.57/MTok.
Why the cache is the whole cost story at long context. The 99% of the prompt that has not changed is what you would otherwise pay to reprocess on every single turn.
If you take one practical thing from this lesson

At 1M context, cost is dominated by whether your prefix is stable. Append to a conversation and you hit the cache. Edit anything early in the prompt, reorder a system message, or reshuffle context between turns, and you invalidate everything after it and pay full price. Prompt architecture is cost architecture.

What the hybrid does to a cache

Here is where lessons 2 and 3 come due. K3 has two kinds of attention layer, and they need fundamentally different caches.

    TWO CACHES, ONE REQUEST

    Gated MLA layers (24)              KDA layers (69)
    ──────────────────────             ───────────────
    grows with sequence length         fixed size
    paged per token                    one state per request
    natural fine granularity           snapshots affordable only
                                        at sparse boundaries
    ───────────────────────────────────────────────────────────
    a cached prefix is reusable ONLY if BOTH can be
    restored at the SAME boundary
The joint-restoration constraint. The report is direct that this complicates prefix caching: the two cache types differ fundamentally in size and lifetime, yet a cached prefix is reusable only when both can be restored together at the same boundary.1

The naive resolution is to force one shared block size on both. That fails badly, and the failure is instructive. Because a KDA state is one large object per sequence rather than per-token entries, checkpointing it is expensive, so it can only be saved at sparse boundaries. Forcing MLA to match pushes the shared block size to 1024 to 6144 tokens, and the report's verdict on that is blunt: At such a coarse granularity caching is nearly useless: requests shorter than one block can never be reused, and chunked prefill exports no cacheable prefix until it crosses a full block boundary.1

K3's fix is to stop insisting the two granularities match.

Decoupled prefix-cache granularities for MLA and KDA state One coarse 6,144-token physical block subdivides into twelve fine 512-token hash blocks. The first five hash blocks are cached, shown filled; the rest are empty. Below, KDA checkpoints sit at hash-block endpoints, most of them discarded and two retained at conversation-turn boundaries. A dashed line marks the hit boundary at token 2,560, which falls exactly on a hash-block edge. one physical block = 6,144 tokens = twelve 512-token hash blocks MLA KV KDA ckpt hit boundary B = 2,560 hash blocks of 512 tokens, finely reusable large dots: retained checkpoints, at turn boundaries cached not yet computed
Figure 12 of the report, simplified. Prefix hashing runs on fine 512-token blocks while the physical block stays coarse. KDA checkpoints can only be saved at MLA hash endpoints, since those are the only positions a lookup can reference, and the retained ones typically coincide with conversation-turn boundaries. On a hit at B the server restores the KDA checkpoint, copies the partial MLA block on write, and resumes prefill from token B with zero recompute of the span before it.

That last detail is the elegant part. Checkpoints are expensive, so K3 keeps the ones most likely to be reused: the boundaries between conversation turns, which is exactly where a follow-up request will resume.

A debugging trick worth stealing

Both cache types are packed into one paged pool with pages unified to the same byte size, so allocation, reference counting, and eviction have a single implementation. The report notes a side benefit found during development: because the two layouts are structurally different, any type-confused access yields garbage rather than plausible data — a zero-overhead sanity check on the pooled layout.1 Making your bugs loud instead of subtle is a good design instinct.

Why 64 accelerators

Recall the derivation from lesson 6: nearly all of K3's 2.78T parameters sit in the routed expert pool. Those weights have to live somewhere, and each token's 16 chosen experts must be reachable quickly.

Supernode
A tightly coupled group of accelerators with high-bandwidth interconnect between them, so cross-device traffic is cheap. The launch post recommends supernode configurations with 64 or more accelerators.2
Expert parallelism
Splitting an MoE layer's experts across devices, so each device stores some experts and tokens are routed to wherever theirs live.

The requirement follows directly from the architecture. Total parameters set the memory floor, and dispatch traffic sets the interconnect floor. LatentMoE halves the second, and MXFP4 quantization shrinks the first, but 2.78T is 2.78T.

    QUANTIZATION-AWARE TRAINING

    MXFP4 weights  +  MXFP8 activations,  from the SFT stage onward

    trained AT the precision it is served at, not compressed after
              │
              └── which is why SiTU-GLU exists (lesson 7):
                  bounded activations are what make 4 bits survivable
The connection back to lesson 7. Post-hoc quantization degrades a model trained in higher precision. Training in the deployment format means the weights adapt to it, and the activation cap is part of the same design decision.

This is the cleanest example in the whole report of architecture and deployment being designed together. SiTU-GLU's bound of 100 looks like an odd detail in §2.3.2. It is there because §4.1.4 decided to train in 4-bit.

Scheduling a fleet

Two production problems, both created by the million-token context, both with solutions worth knowing.

Cache affinity. If a hit is orders of magnitude cheaper than a miss, requests must be routed to the machine holding their cache, since moving a cache across clusters is slower than recomputing. But binding a session to one cluster means a cluster failure kills every session on it. K3 uses consistent hashing to pin each session to two clusters: a primary that serves it, and a secondary that takes over on failure and re-prefills from scratch. Because secondary assignments are spread uniformly, that recovery work lands across many clusters rather than one.1

Traffic variance. Production mixes requests under 2K tokens with requests up to 1M, so per-request cost spans roughly three orders of magnitude. The failure mode is specific: a burst of long-context requests saturates compute, and short requests behind them wait, degrading time-to-first-token for everyone. Averages are useless here, so K3 gives each request class its own resource budget, letting long-context bursts consume at most their own share.1

Is it worth the money

The report includes a cost-efficiency section, which is unusual and to its credit. The consistent finding: K3 trails the two strongest proprietary models on quality and beats them substantially on price.

From §6.4. Costs are per task, measured or cited as noted in the report.1
BenchmarkK3 resultCost comparison
Kimi Code Bench 2.04.0 points behind Fable 538% of Fable 5's cost
Kimi Code Bench 2.0high effort ≈ Opus 4.8 at maxroughly one third the cost
BrowseCompbest score, 91.2$2.03/task, half of GPT-5.6 Sol
GDPval-AA v2within 50 Elo of GPT-5.6 Sol13% lower; 2.6× cheaper than Fable 5
AA-Briefcasesecond, behind Fable 5roughly half Fable 5's cost
Read these with the caveats attached

Every Fable 5 number in the report includes potential fallbacks, and every GPT-5.6 Sol number includes potential cyberguards. On SWE-Marathon, Fable 5 hit fallbacks on 35% of the tasks. A fallback means the harness gave up and did something else, so those baseline scores are not clean. This cuts both ways: it may understate the competitors, and it also means K3's relative standing is measured against a noisy reference. Third-party standings, cited as of 23 July 2026, drift as leaderboards accumulate matches.

The efficiency picture, revisited

You saw this simulator in lesson 2, when the question was theoretical. Now it is a serving question: this is the shape of what you pay for at long context.

Three quarters of K3's attention layers have flat per-token cost. That is why a million-token context is offered at all, and why the coding-session arithmetic above works out the way it does.

Check yourself

Why is K3's cache-hit input price ten times lower than cache-miss?

A typical coding turn carries 400K tokens of prefix and 4K of new input. A hit avoids re-prefilling the 400K, making the turn about 9× cheaper. Over 50 turns that is $6.60 versus $60.60.

What makes prefix caching harder in K3 than in an MLA-only model?

MLA's cache grows per token and pages finely; KDA's is one fixed-size state per request, checkpointable only at sparse boundaries. A prefix is reusable only if both restore at the same point. Forcing one shared block size pushed it to 1024–6144 tokens and made caching "nearly useless", so K3 decoupled the granularities.

Which lesson-7 mechanism exists because of a deployment decision?

Quantization-aware training uses MXFP4 weights and MXFP8 activations from the SFT stage onward. At 4 bits one outlier forces the whole block's scale and destroys precision nearby, so bounding activations at 100 is what makes that format survivable. The activation function was chosen for the deployment format.

Where we are

    ✅ ARCHITECTURE   sequence · depth · width · input · optimise   L2–L9
    ✅ SYSTEMS        cost · prefix cache · quantization · fleet    L11
    ▢  BEHAVIOUR      post-training · effort levels · limitations   L12
    ▢  REVIEW C       capstone                                     L13
Course progress. The architecture and the systems that serve it are both covered. One lesson of behaviour remains before the capstone.

Lesson 12 is the last piece of new material: how K3 was taught to behave. Reinforcement learning across three domains at multiple reasoning-effort levels, then consolidation of many specialist policies into one model. It also covers the three limitations the team published against themselves, one of which will change how you use the model if you deploy it.