Blog
Four gates had never been seen red. Writing their negative controls produced one that reported every branch red while the gate it guarded printed OK on a broken catalog.
Four gates in a repository had never been seen to fail. Not "had failed and been fixed" — never once observed red, by anyone, in their lifetimes. Green on the happy path was the whole of the evidence that they worked, which is the same evidence a gate that cannot fail produces.
So each got a negative control: plant a deliberate fault, run the gate, demand it goes red and names the right branch. Thirty-one mutants across the four, all killed. Then I checked the controls themselves, and one of them was the exact defect the batch was written to remove.
The catalog-integrity gate checks that every row in a format catalog still has its spec file on disk. Its control plants seven faults — a dangling reference, a deleted neighbour, a collapsed family — and asserts each is reported by its own message, with its neighbours' markers absent.
I changed one line in the gate: the return 1 that turns a non-empty problem list into a failing exit code became return 0.
main(): return 1 -> return 0
gate, on a catalog with a dangling source=
"OK: 109 catalog rows, every source= resolves..." exit 0
--self-check
"self-check: all branches proven red" exit 0The gate was completely dead. It printed a clean bill of health on a broken catalog. And all seven of its cases still reported success, because every one of them called the checking function directly and inspected the list it returned. The function was covered. The wiring from that function to the process exit code was not — and the exit code is the only thing continuous integration reads.
It would be easy to conclude that controls should always spawn the real program. The comment above this one explains why it doesn't, and the reasoning is sound: the module resolves its repository root from its own file path, so a control that runs it from the wrong working directory gets a root of /, scans nothing, and reports zero problems for entirely the wrong reason.
That is not hypothetical. It had already happened once in this campaign — a control run from a temporary directory "killed" three mutants and the conclusion was nearly reversed before someone re-ran it from the right place, where it killed none. Calling the function directly makes that class of error impossible.
So the fix adds a layer instead of replacing one. Two end-to-end runs now spawn the whole program against a planted tree, with the script copied into that tree — so the root resolves there by the ordinary parent-of-parent rule. No --root flag, no environment variable. Nothing new that could aim a live gate at somewhere harmless.
To check all four controls at once I neutered each gate with a single regular expression: every return 1 through return 4 became return 0. Two controls then "passed vacuously" — the gate was dead and they said nothing.
They hadn't. The regex had also rewritten the returns inside the control functions. Both had detected the mutant correctly and merely lost the ability to report it. Reading the printed output rather than the exit code is what showed the difference; rerunning with one variable changed is what proved it.
This is the broken-ruler error, applied to the experiment rather than the system: I had modified the instrument and the subject in the same step, then read the instrument. It is worth noticing that the discipline against this was already written down, in a document I wrote, and I did it anyway — the rule is easy to hold when diagnosing someone else's system and easy to drop when the system is your own tooling.
The command that found the four uncontrolled gates reports a column called control. It answers: does a negative control exist? After this batch it reads zero missing, twelve of twelve covered.
That number is a label. The property is: can the control fail? The catalog-integrity gate would have been counted as controlled the entire time its control was incapable of noticing a dead gate. Trusting a self-declared label over a measured property is one of the ten ways a gate lies in the taxonomy this same audit produced — and it was sitting in the tool built to find it.
So the sweep got a sibling. It flips each verdict-returning line to return 0, one site at a time, runs the control, and calls the mutant killed only if the control goes non-zero. Survivors are printed by line number. Where sweep asks whether a control exists, this asks whether it works.
gate mutants verdict
check_catalog_count.py 3/3 all killed
check_catalog_integrity.py 1/1 all killed
check_elab_ratchet.py 3/5 SURVIVED at lines 346, 390
check_seal_coverage.py 0/3 SURVIVED at lines 288, 321, 339
check_withdrawn_live.py 0/2 SURVIVED at lines 177, 192Nine of twelve gates had at least one surviving mutant. The tempting headline is that nine gates are broken. It would have been wrong, and checking took ten minutes.
Six of the nine keep a baseline file — a ledger of known, accepted problems. I moved each baseline aside and ran each gate. All six went red, correctly, with the right message. They work today. A surviving mutant does not say a gate is wrong now; it says nothing in the repository proves it will stay right.
That distinction is the difference between a report someone acts on and a report someone learns to discount.
They are preconditions, not verdicts. A control builds a well-formed world and then breaks one fact inside it: a row that points nowhere, a file that lost its data, a count that drifted. It never breaks the world's *existence* — the baseline that isn't there, the tool that wouldn't run, the directory that wouldn't open.
check_elab_ratchet.py:390 -> return 0
gate, with its baseline moved aside
"no baseline; run --update-baseline once" exit 0
its control exit 0The gate announces that it has nothing to check, and passes. Everything downstream reads green. This is the vacuous-pass class, one layer further out than the audit had looked — the audit asked whether a gate could pass without doing its work, and answered it for the data path only.
Six of the nine share that exact shape, which means one control pattern closes the class rather than six bespoke cases.
Two things above are wrong, and using the tool is what found them.
One survivor was invented by the tool. It ran the *first* control flag a gate declares, not all of them. One gate has two, and the one the tool picked does not reach that gate's main verdict at all — the other kills it in a line. So a report that said "nine gates, twenty-one sites" was overstated by one site, published before it was checked. That is the same error the post is about, committed by the instrument the post introduces.
"Six of the nine share that exact shape" was a heuristic, not a reading. I classified by whether a gate keeps a baseline file, rather than by reading what each surviving site actually guards. Reading all twenty: seven are preconditions, across six gates. The other thirteen are ordinary verdict branches whose controls do not reach them — including one gate's *main* verdict. Closing the precondition class does not fix those six gates, which is what the original sentence implied.
The shared control was still worth building, and it found a live defect on its first run: a gate that printed "SKIP: iverilog or t27c missing" and exited zero, greenlighting an unchecked tree while saying so out loud. Survivor sites are now thirteen, and the two remaining preconditions are named in a constant in the file rather than left to be inferred from a count.
Take any gate you rely on and break its failure path — change the line that returns the failing status, nothing else. Then run whatever proves that gate works. If it still passes, you have learned something specific: not that the gate is bad, but that your evidence for it doesn't reach as far as you thought.
And do it one line at a time. Breaking several things at once and reading the result is how a working control gets mistaken for a broken one, which is how a real defect two files over goes unexamined for another week.
Every figure above is measured, and the limits are named with it.