Blog
Doubling the context made it worse: ternary scaling follows a resonance, not a power law
ctx=18 gives PPL 5.58, ctx=27 gives 2.96, ctx=54 gives 6.05 — worse than the baseline. Powers of three are orbitals and the values between them are forbidden zones.
- THE CONTEXTDoubling context lost the gain.
- THE RESONANCEA small-scale resonance claim.
- THE FORBIDDEN ZONEThe binary control is still missing.
The usual intuition about context length is monotone: more context, better or equal perplexity, with diminishing returns. Ternary training does not do that.
The measurement
| ctx | perplexity |
|---|---|
| 18 | 5.58 |
| 27 = 3³ | 2.96 |
| 54 = 2×27 | 6.05 |
Going from 18 to 27 nearly halves perplexity. Doubling from 27 to 54 does not merely lose that gain — it lands worse than the ctx=18 baseline it started from.
Ternary scaling follows a resonance curve, not a power law. The optimal contexts are 3ᵏ orbitals — 9, 27, 81 — and the values between them are forbidden zones.
Why this is a claim and not an anomaly
A scaling law does not do this. Under any monotone law, ctx=54 sits between ctx=27 and the asymptote, so it can be worse than 27 by diminishing returns but not worse than 18. One measurement that goes backwards past its own baseline is not noise around a curve; it is the wrong shape of curve.
The companion constraint
A second result from the same series: the context must equal the head dimension, or a power-of-three divisor of it. Square attention — ctx = head_dim — gives full rank; anything else gives up rank silently.
Taken together the two are a recipe with no free parameters: pick a power of three, then set head_dim to it. Any other pair either lands in a forbidden zone or gives up rank, and both failures are quiet — the run trains, it just trains worse.
What would falsify it
A binary-weight control at ctx = 16, 32, 64. If the same non-monotonicity appears with powers of two, the effect is about context alignment in general and has nothing to do with the ternary alphabet. That control has not been run here, and it is the first thing I would ask for if someone showed me this table.
What this does not settle
- The runs are from the HSLM series recovered 2026-04-20 by walking git history; the perplexities are theirs, not re-measured. Published 2026-08-11.
- These are HSLM runs at small scale. Nothing here shows the effect survives at sizes where a context of 81 or 243 is practical, and the orbital spacing means the next test point is three times away rather than adjacent.
- The mechanism is asserted from the ratio structure, not derived. Why powers of three specifically — rather than powers of any base matching the weight alphabet — is not established here, and a binary-weight control at ctx=16/32/64 would test it directly.
- Perplexity ranges are reported per configuration (ctx=27 gives 2.96–5.55 across runs), so the ctx=27 advantage is larger than run-to-run variance but the exact margin depends on which runs are compared.
Receipts
Work with me
Want your team to build and verify this themselves?
I run a self-paced FPGA course, a four-week cohort and two-day team workshops built around a problem your engineers actually have.