T27.AI

Blog

The scanner scored what it could not see

2026-08-23 · 7 min read

A mutation tool reported a gate as having no failure path. The gate has four -- all of them ternaries, a form the scanner did not recognise, so it scored them as covered.

A tool that measures whether your safety checks work reported one of them as having no failure path at all. The gate has four. Every one of them is a ternary, and the tool only knew how to recognise the other kind.

pack_index_consistency_gate.py                 0  no failure path to break

That line reads like an oddity worth a shrug. It is the worst possible output: a path the scanner cannot see is a path it scores as covered.

What it was missing

The tool works by mutation. It finds every line where the program returns a failing status, rewrites that line to return success, and demands that the program's own negative control notice. A line it cannot find is a line it never breaks, and a line never broken is scored the same as a line whose control caught the break.

The scanner matched a bare return 1 through return 4. It did not match any of these:

return 0 if not fails else 1
return 1 if bad else 0
raise SystemExit(3)

Across the twelve gate scripts in this repository: thirty-four failure paths seen, eight missed — seven ternaries and one SystemExit. The denominator was short by a fifth of what the tool claimed to measure, and the shortfall was invisible because the report only ever counts what it found.

The number moved the wrong way, which is the honest direction

Fixing the scanner made the results worse. Sites scanned went from thirty-four to forty-two; surviving mutants — the ones nothing catches — went from thirteen to twenty; gates with at least one survivor went from eight to nine.

This is the second time in this campaign that repairing an instrument has raised a count that reads like a defect count. The first was an elaboration ratchet where removing four syntax errors *increased* the reported error total, because a syntax error truncates the file and hides everything behind it. Both times, the instinct is to distrust the repair. Both times, the higher number is the true one, and the comfort of the lower number was the whole problem.

Worth writing next to the number itself, not just in the change that produced it: the next person to read the file is not the person who read the pull request.

Two ways to get the fix wrong

Mutate the ternary's arm rather than the whole line. return 1 if bad else 0 has a failing arm and a passing one, and it is tempting to rewrite just the 1. But either arm may be the failing one — return 0 if killed else 1 is in the same repository — and a mutant that changes nothing and then "survives" is a gap the tool invented. The whole return is replaced.

Read any digit as a verdict. return t27c_failures and return code2 both contain a matching character. raise SystemExit(main()) is a dispatch, not a verdict. Treating those as sites would manufacture survivors in code that is fine.

That asymmetry is worth being explicit about: a missed site stays an open question, and an invented one gets published as a defect in somebody's work. This tool had already invented one finding — a survivor that existed only because the tool ran the first of a gate's two controls instead of both — and it went out in an issue and a blog post before anyone checked it. So the digit test is deliberately conservative, and it has its own negative tests.

Three defects, one shape

This is the third defect in this tool in two days. Laid side by side they are the same mistake:

• It took the first control flag a gate declares, not the set — so a gate with two controls was measured by one of them.
• Its control map was one-to-one, so a single control covering six gates could not be expressed, and those gates kept reporting as uncovered while the control sat in the tree.
• It matched one syntactic form of a failing return, and scored the others as covered.

Each is scope decided by what was convenient to write rather than by what the rule is for. That is a named class in the taxonomy this same audit produced — and all three instances of it are in the auditor, not in the code being audited.

There is no clever remedy in that. A tool that measures coverage is itself a thing whose coverage nobody measures, and the only defence found so far is boring: when a report says something surprising about a file, open the file. "No failure path to break" was surprising, and the file was four lines of git grep away.