Reference · KDA cheat sheet

The recurrence

    S_t = (I − β_t k_t k_tᵀ) · Diag(α_t) · S_{t−1}  +  β_t k_t v_tᵀ
           └──────┬───────┘    └────┬────┘  └──┬──┘     └────┬────┘
              ② ERASE           ① FADE     old state      ③ WRITE

    õ_t = S_tᵀ q_t                    READ — one matvec, no history scan

§2.1.1, eq. 1 of the technical report. Order of operations: fade the old state channel-by-channel, erase what the current key already retrieves, then write the new association.

Every symbol

Symbol	Shape	What it is
`S_t`	d_k × d_v	The recurrent state. A memory matrix mapping keys to values. Fixed size regardless of sequence length.
`q_t, k_t`	d_k	Query and key. Both pass through ShortConv → Swish → L2Norm.
`v_t`	d_v	Value. ShortConv → Swish, no L2Norm.
`α_t`	d_k	Per-channel retention factor, in (e⁻⁵, 1). The fade. Each channel decays at its own rate.
`β_t`	scalar	Write strength, in (0,1). How firmly this token commits its value.
`z_t`	d_k	Decay logits, from a low-rank projection plus per-head bias. Becomes α via eq. 5.
`g_t`	d_k	Log-decay, in (−5, 0). `α_t = exp(g_t)`.
`A_h`	scalar	Learnable per-head log-scale. Initialised to 0.

Symbol

Shape

What it is

S_t

d_k × d_v

The recurrent state. A memory matrix mapping keys to values. Fixed size regardless of sequence length.

q_t, k_t

d_k

Query and key. Both pass through ShortConv → Swish → L2Norm.

v_t

d_v

Value. ShortConv → Swish, no L2Norm.

α_t

d_k

Per-channel retention factor, in (e⁻⁵, 1). The fade. Each channel decays at its own rate.

β_t

scalar

Write strength, in (0,1). How firmly this token commits its value.

z_t

d_k

Decay logits, from a low-rank projection plus per-head bias. Becomes α via eq. 5.

g_t

d_k

Log-decay, in (−5, 0). α_t = exp(g_t).

A_h

scalar

Learnable per-head log-scale. Initialised to 0.

How the inputs are produced

q_t, k_t = L2Norm( Swish( ShortConv( W_q/k · x_t ) ) )     ∈ ℝ^d_k
v_t      =         Swish( ShortConv( W_v   · x_t ) )       ∈ ℝ^d_v
β_t      =       Sigmoid( W_β · x_t )                      ∈ (0,1)
z_t      =       W_α↑ ( W_α↓ · x_t ) + b_α                 ∈ ℝ^d_k

§2.1.1, eq. 2. ShortConv gives each projection a small local receptive field before the recurrence. L2Norm on q and k bounds the erase term.

Lower-bounded decay — the g_min trick

	Kimi Linear	Kimi K3
Mapping	`g = −e^A · Softplus(z)`	`g = g_min · Sigmoid(e^A z)`
Range of g	(−∞, 0)	(−5, 0)
Min retention α	unbounded below	e⁻⁵ ≈ 6.7 × 10⁻³
Cumulative log-decay per 16-token tile	unbounded	(−80, 0)
Reciprocal rescale factor	can overflow BF16	< e⁸⁰, inside BF16 range
Diagonal tiles computed by	explicit position-pair path	dense Tensor Core matmul

Kimi Linear

Kimi K3

Mapping

g = −e^A · Softplus(z)

g = g_min · Sigmoid(e^A z)

Range of g

(−∞, 0)

(−5, 0)

Min retention α

unbounded below

e⁻⁵ ≈ 6.7 × 10⁻³

Cumulative log-decay per 16-token tile

unbounded

(−80, 0)

Reciprocal rescale factor

can overflow BF16

< e⁸⁰, inside BF16 range

Diagonal tiles computed by

explicit position-pair path

dense Tensor Core matmul

Why this is the whole point

The chunkwise form rescales keys by 1/Γ, a reciprocal of accumulated decay. Unbounded decay means that reciprocal can overflow in BF16, which forced Kimi Linear to handle diagonal tiles on a slow special-case path. Bounding the log-decay below at −5 caps the reciprocal at e⁸⁰, inside BF16 range, so every tile can use dense Tensor Core matrix multiplication. A constant chosen for numerics deleted an entire slow code path.

Chunkwise parallel form

γ_[t]^(i→j) = Π_{r=i..j} α_r[t]              channel-wise cumulative decay
Ṽ_[t]       = U_[t] − W_[t] S_[t]            pseudo-value, from the UT transform

A_[t] = Tril[ (Q_[t] ⊙ Γ_[t]) (K_[t] / Γ_[t])ᵀ ]

O_[t] = (Γ_[t] ⊙ Q_[t]) S_[t]  +  A_[t] Ṽ_[t]
        └──── inter-chunk ────┘    └── intra-chunk ──┘

§2.1.1, eqs. 3–4. Recurrent across chunks, parallel within each chunk. This is what makes a sequential recurrence trainable on a GPU. Tril keeps the diagonal because each output reads the state after the current token's update. The UT transform and full derivation live in the Kimi Linear paper.

Lineage — what changed at each step

Mechanism	Fade	Erase	Contribution
Classical linear attention	none	none	Fixed-size state. Degrades as writes accumulate.
DeltaNet	none	delta rule	Writes replace rather than accumulate.
Gated DeltaNet	scalar	delta rule	Adds a forget gate, one rate for the whole state.
KDA (Kimi Linear)	channel-wise	delta rule	Each channel decays at its own learned rate.
KDA (K3)	channel-wise, bounded	delta rule	Bounded decay → all-Tensor-Core kernels. Full-rank output gate.

Mechanism

Fade

Erase

Contribution

Classical linear attention

none

Fixed-size state. Degrades as writes accumulate.

DeltaNet

none

delta rule

Writes replace rather than accumulate.

Gated DeltaNet

scalar

delta rule

Adds a forget gate, one rate for the whole state.

KDA (Kimi Linear)

channel-wise

delta rule

Each channel decays at its own learned rate.

KDA (K3)

channel-wise, bounded

delta rule

Bounded decay → all-Tensor-Core kernels. Full-rank output gate.

	KDA	Gated MLA
State per layer	fixed (d_k × d_v per head)	grows with n (one latent vector per token)
Work per new token	constant	grows with n
Long-range retrieval	approximate, gate-dependent	exact
Position information	implicit, via decay and recurrence	none (NoPE)
Layers in K3	69 of 93	24 of 93

KDA

Gated MLA

State per layer

fixed (d_k × d_v per head)

grows with n (one latent vector per token)

Work per new token

constant

grows with n

Long-range retrieval

approximate, gate-dependent

exact

Position information

implicit, via decay and recurrence

none (NoPE)

Layers in K3

69 of 93

24 of 93

KDA cheat sheet

The recurrence

Every symbol

How the inputs are produced

The output gate

Lower-bounded decay — the g_min trick

Chunkwise parallel form

Lineage — what changed at each step

Cost profile vs. the alternative