T27.AI

Blog

What five backends teach about single-source verification

2026-11-15 · 9 min read

[numbers are from docs/reports/t27b-bench-2026-10.md and the receipts it cites; not rerun for this publication] Five backends, one spec source: the whole-corpus differential run compares t27b against the reference test by test - 0 of 8,344 tests disagree - while Zig, clang and gcc pipelines disagree with themselves across optimization levels. The article also states what the zero does not prove: no independent front end exists, the fuzzer still finds 7 disagreements per 1,000 programs, and only 12% of the code t27b runs is generated.

Title card, no illustration yet, for: What five backends teach about single-source verification
View the title card at full size
#verification#compilers#DifferentialTesting#trust#backends

Our earlier article, [One spec, five targets: the t27 backend shootout](/blog/one-spec-five-targets-the-t27-backend-shootout/), ranked five backends on speed and size. This article is deliberately not that. Speed and size answer "which build should you ship." A different question matters more when the thing you build is a compiler: how do you know its output is correct? This piece is about verification - what the same five-backend setup can prove about trust, and what it cannot.

The one-disagreement budget

The t27 repository keeps one correctness budget, stated as a number: the number of tests where two backends disagree. For the newest backend, t27b, the budget is spent at exactly zero.

The whole-corpus differential run compares t27b against the reference pipeline (t27c's generated Zig, built in Debug mode and run natively). Over 1,315 spec files that both sides ran, the verdicts were compared test by test, by name: 0 of 8,344 tests disagree. The same zero holds when the reference itself is rebuilt for aarch64 and executed under emulation.

That single number is the article's thesis. A compiler project that cannot state its disagreement count as one figure does not yet have a verification story; it has a test suite. A test suite tells you the code runs. A differential budget tells you the code agrees with something else that was built a different way.

The budget only means something because the other backends spend it. In the same run:

./scripts/tri test
./target/release/t27c suite --repo-root .

Zig's own ReleaseFast build disagrees with its Debug build on 7 files (11 tests). The generated C, compiled by clang at -O0 versus -O2, differs on 39 files (107 tests); gcc at the two optimization levels differs on 4 files. Those disagreements exist because the generated C contains undefined behavior - the verdict depends on the optimization level - and because ReleaseFast optimizes differently from Debug.

So the picture is not "t27b is perfect." It is: one backend agrees with its reference on every single test, while the mainstream pipelines cannot even agree with themselves across optimization levels. Exact agreement with a reference is worth more than approximate agreement with a family of your own builds.

Per-test reference diffing

Agreement at the file level ("the pipeline passed") is weak. A file can pass while one test inside it silently returns the wrong verdict. So the corpus runner compares at the test level, and it does so over the entire spec corpus, not a sample: 1,819 .t27 files under specs/, the same set t27b corpus specs walks, with nothing left out for being slow or broken.

The invocation:

./target/release/t27b test specs/a/b_c.t27 --overflow trap

shows the unit of comparison: one spec, its tests run in the JIT with overflow checks that trap. At corpus scale the same command becomes a walk over every file, and --reference adds the other side:

./target/release/t27b corpus specs --blockers --reference ./target/release/t27c

With --reference, every file also goes through the reference path (t27c test-report, the Zig backend), and where both sides ran a file's tests, the runner compares verdicts test by test and counts files where a test differs, or where only one side has it. The files the reference itself fails are left out of the denominator, so the comparison is honest about what was actually compared: at this tree, 1,311 files pass with at least one test in the reference, and t27b passes 1,296 of them (98.9%).

The smallest disagreement the differential ever catches is instructive, and it is not in t27b. specs/a/b_c.t27 makes the generated C emit bool dummy(void) { true; } with no return: it passes under gcc -O0 and clang -O2, and aborts under gcc -O2 and clang -O0. Four build configurations, one function, four verdicts. That is the failure mode per-test diffing exists to catch, and the reason the budget is counted per test, not per file.

What it does NOT prove

This is the section that matters most, and the one a marketing page would omit.

The 0-of-8,344 result is not independent verification. The repository ran a Diverse Double-Compiling check - build the self-compile three ways, compare stage-2 outputs - and the signed receipt records the verdict honestly: DDC_BACKEND_AGREEMENT, not DDC_INDEPENDENT, with filter C4 FALSE. All three routes share one hand-written front end. The reference that t27b agrees with on 8,344 tests is produced by the same front end that t27b mounts. Agreement here proves the backends implement the same front end the same way. It cannot catch a bug that lives in the shared front end.

The other checks have the same shape:

No check in the current setup compares t27b against an independently written t27 front end. That is the honest boundary: the verification is unusually *visible* - signed receipts for every commit, a per-test differential over the whole corpus - but almost all of it is internal consistency, not independence.

The shape of the trade

Verification at this level is not free, and the shootout article's speed numbers are the price tag. The reference pipeline compiles a spec in 3.05 seconds (Zig Debug, with LLVM and a link step); t27b does the same work, tests included, in 4.8 ms per spec. Its whole binary is 3.08 MB, with no assembler and no linker.

The trade cuts the other way too. The negative result is stated as plainly as the positive one: over the tier-1 benchmarks, t27b executes 3.4x the instructions of gcc -O2 and produces code 1.8x the size. Its generated code sits at gcc -O0 quality, behind every optimizing pipeline in the comparison. Coverage also trails the reference it mirrors: 15 files the reference passes do not pass in t27b.

Most projects would bury one of these halves. The shape of the trade is the finding: a backend can be near-zero on the disagreement budget and simultaneously last on code quality, because agreement with a reference and quality of output are orthogonal axes. A fast, small, exactly-agreeing baseline compiler is a useful artifact even when its optimized output loses - provided you say which axis each claim lives on.

There is also a shipping boundary, stated so nobody over-reads the result: t27b emits object files only (ELF64 or Mach-O), it does not link, and it refuses module-level variables in an object file. It is a test runner and a code generator, not yet a producer of runnable executables.

Limitations

The honest list, so the next reader does not have to rediscover it:

  1. One shared front end. Everything above reduces to "the backends agree about the front end." Independence would require a second, separately written front end; none exists.
  2. The fuzzer is small. 1,000 programs per run against one reference is a smoke test, not a falsification campaign. The 7-per-1,000 disagreement rate is a live number to drive down, not a solved quantity.
  3. Coverage gap. 15 files behind the reference, with a known blocker list (i128, escaping frame addresses, empty typed array literals, @cos, anytype, std.*, a frame-size limit). The 98.9% figure is coverage of the reference-passing set, not of the corpus.
  4. No executables. Object-only output, no link, no module-level variables; on Linux, t27b runs tests in its JIT.
  5. Self-generated share. At 12% of the code it runs, t27b cannot meaningfully compile itself end to end; the trust story is about agreement with a sibling, not about self-hosting.

The conclusion is not "therefore distrust t27b." It is that a verification claim is only as good as its stated boundary, and every number above is published with the command that produced it. That habit - receipt, command, count - is portable to any compiler project, whatever its backends.

Sources

Every claim above, with the file and the command that backs it:

ClaimSource (file)Command
0 of 8,344 tests disagree, t27b vs reference, 1,315 filesdocs/reports/t27b-bench-2026-10.md (disagreements table, "Reading these numbers")./target/release/t27b corpus specs --blockers --reference ./target/release/t27c
Zig Debug vs ReleaseFast: 7 files / 11 tests; clang -O0/-O2: 39 files / 107 tests; gcc -O0/-O2: 4 filessame report, disagreements table./scripts/tri test
specs/a/b_c.t27 missing-return examplesame report, "The C pipelines disagree with themselves"./target/release/t27b test specs/a/b_c.t27 --overflow trap
Corpus 1,819 files; t27b passes 1,296/1,311 (98.9%)same report, "Denominator" and coverage tables./target/release/t27b corpus specs --blockers --reference ./target/release/t27c
DDC_BACKEND_AGREEMENT, not DDC_INDEPENDENT; filter C4 FALSEsame report, trust table (receipt #7706)signed DDC receipt (t27b lab, commit 0e8564505)
Fuzzer: 7 disagreements per 1,000 programs, seed 1791653965same report, trust table (receipt #6442)random-program fuzzer, t27b lab run
12% self-generated sharesame report, trust tableline counts: gen/rust/tri/t27b vs cli/t27b/src + bootstrap/src/compiler.rs
4.8 ms/spec vs 3.05 s zig Debug; 3.08 MB binarysame report, compile-speed table and footprint table./target/release/t27b test <spec> --time / binary size
3.4x gcc -O2 instructions, 1.8x sizesame report, axis C sections and limitations listqemu instruction-count plugin runs
Object only (ELF64/Mach-O), no link, no module-level varssame report, "no ahead-of-time executable"; cli/t27b/src/codegen.rs ("no writable data section")./target/release/t27b --help

Speed and size context: [One spec, five targets: the t27 backend shootout](/blog/one-spec-five-targets-the-t27-backend-shootout/). Everything else about the project: [t27.ai](https://t27.ai).

Test files where compiler settings disagree 1,315 test files; each bar counts files whose results changed between settings 0 10 20 30 40 clang -O0 vs -O2 39 files zig Debug vs ReleaseFast 7 files gcc -O0 vs -O2 4 files t27b vs reference 0 files t27b vs the reference compiler: 0 of 8,344 tests disagree
Compiler-setting disagreements across 1,315 test files - clang -O0 vs -O2 leads; t27b vs reference: 0

What this does not settle

Receipts

Work with me

Want this kind of check on your own design?

I audit RTL and build independent, bit-exact models, then take the result through synthesis and, when useful, onto an Artix-7 board. The first conformance module is free.