Blog
ctx=18 gives PPL 5.58, ctx=27 gives 2.96, ctx=54 gives 6.05 — worse than the baseline. Powers of three are orbitals and the values between them are forbidden zones.
The usual intuition about context length is monotone: more context, better or equal perplexity, with diminishing returns. Ternary training does not do that.
| ctx | perplexity |
|---|---|
| 18 | 5.58 |
| 27 = 3³ | 2.96 |
| 54 = 2×27 | 6.05 |
Going from 18 to 27 nearly halves perplexity. Doubling from 27 to 54 does not merely lose that gain — it lands worse than the ctx=18 baseline it started from.
Ternary scaling follows a resonance curve, not a power law. The optimal contexts are 3ᵏ orbitals — 9, 27, 81 — and the values between them are forbidden zones.
A scaling law does not do this. Under any monotone law, ctx=54 sits between ctx=27 and the asymptote, so it can be worse than 27 by diminishing returns but not worse than 18. One measurement that goes backwards past its own baseline is not noise around a curve; it is the wrong shape of curve.
A second result from the same series: the context must equal the head dimension, or a power-of-three divisor of it. Square attention — ctx = head_dim — gives full rank; anything else gives up rank silently.
Taken together the two are a recipe with no free parameters: pick a power of three, then set head_dim to it. Any other pair either lands in a forbidden zone or gives up rank, and both failures are quiet — the run trains, it just trains worse.
A binary-weight control at ctx = 16, 32, 64. If the same non-monotonicity appears with powers of two, the effect is about context alignment in general and has nothing to do with the ternary alphabet. That control has not been run here, and it is the first thing I would ask for if someone showed me this table.
Every figure above is measured, and the limits are named with it.