T27.AI

Blog

tri test now runs the tests

2026-10-07 · 4 min read

[t27#7400 is open, not merged; the demo is one spec on the t27c lab; t27c test-report still exits 0 on a failing test (t27#7370); the MCP tool tri.test is still vacuous] Until now tri test only listed a spec's tests and printed that they passed, so a planted bug stayed green. t27#7400 makes it run them, read the report and exit non-zero on a failure, a BLOCKED spec or zero tests. It also adds tri mutate plant: one recorded mutant on a copy, a clean baseline first, the failing tests named, the original's hash checked. Nine lessons of the AI numbers course wait on these two commands for their recordings.

Until now, tri test <spec> did not run a spec's tests. It called t27c test, which only lists them, and then printed "tests passed" whatever they did. A planted bug stayed green. gHashTag/t27#7400 changes that: tri test now runs the tests, and a new command, tri mutate plant, records one hand-written mutation and names the tests it turns red. #7400 is open and not merged yet.

What tri test does now

It runs t27c test-report <spec>, prints the report, then one line, tests: N, pass: P, fail: F (names). It exits non-zero when a test fails, when the spec is BLOCKED, or when zero tests ran; a spec with only invariants is told that an invariant is not a test.

The verdict is read from the report's text, because t27c test-report itself exits 0 even when tests fail. That exit code is a separate issue, t27#7370, still open. The parser fails closed: a report with no totals, totals that do not add up, or a FAIL count that disagrees with the FAIL names is an error, not a pass.

In the PR's demo on the t27c lab, gft_relu.t27 passes 4 of 4. A scratch copy with line 12 changed so that a negative input is returned instead of 0 gives 3 of 4, negz fails, and the command exits 1.

tri mutate plant

tri mutate plant --file <spec> --line <N> --from <old> --to <new> [--expect <test>] makes one mutant and reports it:

• The mutant is a copy in the work directory tri mutate spec already uses, so use still resolves. The original is not edited.
• <old> must occur exactly once on line N; an empty --from, --from equal to --to, a newline in --to, or a line out of range is refused.
• It prints the one-line diff and runs a clean baseline copy first. A spec that is red before the plant is refused, because the mutant would mean nothing.
• It names the tests that fail on the mutant. A mutant that is BLOCKED is reported as unviable, not killed.
• The original's sha256 is taken before and after; if it changed, the command fails with ORIGINAL CHANGED whatever the verdict.
• With --expect it exits 0 only if the failing set equals the expected set; without it, 0 if at least one test fails.

In the demo, the relu mutant with --expect negz prints "KILLED as expected by: negz" and exits 0. The same mutant with --expect pos exits 1 and says which test was expected but passed and which failed unexpectedly. The original file's hash is the same before and after both runs.

How the change itself was checked

The PR's Rust tests for mutate:: ran on the lab: 43 passed, 0 failed, 9 of them new. Two mutants were planted in the change and reverted with git: making the green rule ignore FAIL turned 3 tests red, and skipping the restore check turned 1 red. After both reverts, 43 passed again.

Why the course waits on it

Lessons 13 to 21 of the "AI numbers" course (gft_smul, gft_sadd, gft_signed_mac, gft_relu, gft_exp2, gft_argmax4, gft_nll, gft_sgd_step and gft_xornet) each end with a recording: the spec's tests pass, one line is changed, and exactly one named test fails. The recording runs tri test and tri mutate plant. Before #7400 the first could not show a failure at all, and the second did not exist. Until #7400 merges, those nine lessons open a placeholder that says the recording is pending and shows no run.

Not in #7400: the MCP tool tri.test in cli/tri-mcp still calls t27c test and is still vacuous, and the exit code of t27c test-report is left to t27#7370.

Work with me

Want this kind of check on your own design?

I audit RTL and build independent, bit-exact models, then take the result through synthesis and, when useful, onto an Artix-7 board. The first conformance module is free.

Discuss your projectSee the verification service