Blog

Twenty-three reference models, because implementing a competitor from memory always flatters you

2026-08-11 · 6 min read

Every format we compare against has its own reference implementation, written from its specification — after five bugs that all weakened the competitor and all pointed the same way.

Numeric formatsBenchmarkingMethodologyOpen source

If you compare your numeric format against others, you have to implement the others. That sounds like a chore and it is where the result is decided.

Five bugs, all in the same direction

I implemented five competitor formats from memory. Every one had a bug, and every bug weakened the competitor:

#	format	the error
1	MX shared scale	ceiling instead of floor(log2 max) − emax
2	E2M1	missing its subnormal — 7 magnitudes, not 8
3	NF4	a symmetric reconstruction instead of the real 16-value table
4	E4M3	reserved NaN encoding ignored — max 480 instead of 448
5	E5M2	reserved exponent ignored — max 114688 instead of 57344

Five for five, one direction. That is not luck and it is not dishonesty — it is structural.

Why the direction is not random

Every one of those five is a simplification. A subnormal, a reserved encoding, an asymmetric table — these are exactly the details a format adds to work better at the edges of its range.

Implementing from memory implements the IDEA of the format, and the idea is always simpler than the specification.

So implementing from memory hands your competitor a worse version of itself, every time, in the direction that favours you. You do not choose the bugs. You choose not to open the spec.

What we do instead

Twenty-three reference models, one per format family we compare against, each written from the format’s own specification or its published reference implementation:

IEEE and friends — ieee, bf16, fp8, extended, decimal
MX — mxfp, e8m0, gf_mx
posit and takum — posit, takum, takum_log, tekum
others — lns, nf4, int, legacy
ours — gf, gf16_plus, gfternary, tnf, tnf16, tnf_spec, bnf

The check you can run on someone else’s paper

When a paper reports beating E4M3 or NF4, the question is not what the numbers are. It is where the competitor’s implementation came from. Three answers:

from the specification — the numbers are worth discussing
from a published reference implementation — the same
from the paper describing the format — this is where my table starts

The third looks conscientious and is not. A paper describing a format gives you the idea, not the specification, and the gap between them is where all five of my bugs lived.

One concrete check costs nothing: E4M3 maxes at 448 and E5M2 at 57344. If a comparison implies 480 or 114688, the reserved encodings were skipped and the competitor was handed extra range it does not have.

What this does not settle

The 23 models were counted 2026-08-11 through the GitHub API. The five bugs are from the block-quantisation campaign recorded in FINDINGS.md, earlier.
Twenty-three models is not twenty-three formats: several cover multiple widths (posit8/16/32, fp8 as both E4M3 and E5M2), so the format count is higher and is not stated here because it has not been counted.
A reference model written from a specification can still misread the specification. These are checked against published test vectors where those exist, and where they do not the model is only as good as the reading.
Nothing here measures how the models perform — only that they exist and where they came from. A correct competitor implementation is a precondition for a fair comparison, not a result.

Receipts

Every figure above is measured, and the limits are named with it.

Open the interactive version Читать по-русски admin@t27.ai All posts