Lesson 7a · SiTU-GLU

Lesson 6 ended with the report's admission that extreme sparsity amplifies two failure modes. First, activations explode in the routed branch. Second, load balancing stops working at nearly a thousand experts per layer. The word Stable in Stable LatentMoE names the fixes for both. This lesson takes the first; lesson 7b takes the second, which is the better story.

The activation function family

Every modern transformer's feed-forward network uses a gated linear unit. Two projections of the input, one modulating the other by multiplication.

THE FAMILY
Variant	Gate branch	Up branch	Bounded?
GLU	σ(x)	x	no
SwiGLU	x · σ(x)	x	no
SiTU-GLU	β₁tanh(x/β₁) · σ(x)	β₂tanh(x/β₂)	yes, ≤ β₁β₂

Three generations of gating (§2.3.2, Fig. 4). SwiGLU replaced GLU's plain sigmoid gate with Swish and became the default in essentially every large language model. K3 caps both of its branches, which is what bounds their product.

Swish: x · σ(x). Roughly linear for large positive inputs, smoothly suppressed for negative ones. The gate function in SwiGLU.
SwiGLU: The standard feed-forward design since about 2020, multiplying a Swish-gated branch by a linear branch. Empirically strong; the report notes a complete account of its empirical effectiveness remains open.

The problem the report identifies is simple once stated: both of SwiGLU's factors are unbounded. When two large numbers happen to land in the same coordinate, their product is much larger, and the report warns this can produce activation outliers and increase overflow risk in low-precision arithmetic.¹

Why K3 cares more than most: it applies quantization-aware training from the SFT stage onward, with MXFP4 weights and MXFP8 activations (SFT, supervised fine-tuning, is the first post-training stage; lesson 12 covers it).¹ A 4-bit float represents very few distinct magnitudes. One wild activation forces the scaling factor for an entire block to accommodate it, and everything else in that block loses precision. Outliers are far more damaging at 4 bits than at 16.

The design brief

Cap the large values without changing the behaviour near zero. That second half matters, because SwiGLU's response shape around the origin is presumably why it works, and nobody wants to discover otherwise at 2.8T scale.

The soft cap

SiTU-GLU's answer is a soft cap: β·tanh(x/β), applied to both branches. The tanh function is almost exactly the identity near zero and flattens to ±1 far out, so scaling it by β gives a function that passes through the origin like x and saturates at ±β.

Switch between the two views. The near-origin view is the one that should surprise you.

Near the origin the two curves are indistinguishable, a deviation of 0.02% at x = 0.1. At x = 100 SwiGLU has reached 10,000 while SiTU-GLU has flattened at just under 100. Same function where it matters for learning, hard ceiling where it matters for precision.

show me the actual equation

From §2.3.2 eq. 12, with the soft cap softcap(x, β) = β·tanh(x/β):¹

SiTU-GLU(x) = β₁ tanh( W_g x / β₁ ) ⊙ Sigmoid( W_g x ) └──── capped Swish gate ────────────────┘ ⊙ β₂ tanh( W_u x / β₂ ) └── capped up branch ──┘ K3: β₁ = 4 (gate) β₂ = 25 (up)

Note the gate branch keeps its sigmoid factor uncapped. Appendix B explains why: Because the sigmoid already drives the negative gate response toward zero, this change primarily controls large positive activations without removing the negative tail.

The two properties, from appendix B eqs. 18–19:

near origin: β tanh(z/β) = z + O(z³/β²) ← matches SwiGLU to first order limit: β₁,β₂ → ∞ recovers SwiGLU exactly bound: ‖SiTU-GLU(x)‖∞ ≤ β₁β₂ = 100

The bound follows immediately from |tanh| < 1 and 0 < σ < 1: the product of two factors capped at β₁ and β₂ cannot exceed β₁β₂.

One last detail worth noticing, because it explains why a smooth cap rather than a hard clamp: Unlike hard clamping of gate pre-activations, the smooth cap preserves nonzero gradients away from saturation boundaries, which we find to give better training behavior. A hard clamp has zero gradient past the threshold, so a unit that saturates stops learning. Tanh always has some gradient.

This is where architecture meets deployment

SiTU-GLU's bound of 100 looks like an arbitrary detail here. It is not: it exists because §4.1.4 decided to train and serve K3 in 4-bit. Bounded activations are what make MXFP4 survivable, so an activation-function choice in §2.3.2 is really a consequence of a deployment decision. Lesson 11 returns to this from the serving side.

Check yourself

Why does K3 cap its activation function when SwiGLU worked fine for years?

It trains in 4-bit, where one outlier costs precision nearby
Bounded activations store the same quality in fewer weights
The cap makes each expert more expressive at latent width
Unbounded activations block gradients from reaching early layers

MXFP4 weights with MXFP8 activations, applied from the SFT stage onward. A 4-bit float has very few representable magnitudes, so one wild value forces the whole block's scale to accommodate it. The report also cites exploding activations in the four-matmul routed chain.

SiTU-GLU caps SwiGLU. What does it deliberately preserve?

The response shape near the origin, matched to first order
Unbounded growth in the positive direction, for large inputs
Hard clamping behaviour, applied to the gate pre-activations
The parameter count of the gate and up projections combined

A scaled tanh is almost exactly the identity near zero, so SiTU-GLU deviates from SwiGLU by about 0.02% at x = 0.1 while being hard-bounded at β₁β₂ = 100. It also recovers SwiGLU exactly as the betas go to infinity, so it is a strict generalisation.

One fix down, one to go

    STABLE LATENTMOE — the "Stable" is three fixes

    LatentMoE          896 experts affordable at half width      L6
    RMSNorm before W↑  routed branch arrives at a stable scale    L6
    SiTU-GLU           activations bounded at 100, for 4-bit      L7  ← done
    Quantile Balancing 896 experts balanced in one step           L7b ← next

Two down, one to go. LatentMoE and the pre-W↑ RMSNorm came from lesson 6; SiTU-GLU is this lesson. Quantile Balancing is the last piece.

SiTU-GLU handles the first failure mode: exploding activations. Lesson 7b handles the second, load balancing across 896 experts, and it is the more satisfying of the two. Quantile Balancing reframes a fiddly tuned heuristic as a question with a direct answer, and the answer turns out to be the solution to a known optimisation problem.