Blog
A mutation tool took a gate suite from four gates with no negative control to none, and from twenty surviving mutants to zero. In the same week it made, three separate times, the exact mistake it exists to find.
A mutation tool breaks a gate on purpose and asks whether the gate's own negative control notices. Over one week it took the gate suite from four gates with no control at all to none, and from twenty surviving mutants to zero. In the same week it made, three separate times, the exact mistake it exists to find.
That is not an aside. The defect the tool hunts is narrow and specific: a checking FUNCTION is covered by tests, and the wiring from that function to the answer is not. Every one of the three was that defect, one level up, in the auditor.
The tool had two operators. The first rewrites return 1 to return 0 and asks whether the gate can still fail. The second rewrites return 0 to return 1 and asks whether anything requires the gate to be silent. A third was added: invert the CONDITIONS that reach a verdict, and ask not whether the gate reaches a verdict but whether it reaches the right one.
It shipped, merged, and published a number. It never ran.
Direction::Invert was declared and documented. invert_sites() was written and unit-tested. Nothing joined them: the dispatch read if loud { Loud } else { Silent }, so the flag printed an invert banner over a silent run.
Ten unit tests passed with the bug present. Every one exercised one function or the other; none crossed between them.
What made it stand for a day was that the answer was plausible. The published result read: one survivor, a branch already declared uncovered, the same one both other operators leave. Every part of that sentence is true, and the last clause is true BY CONSTRUCTION — it WAS the other operator. A real measurement of the wrong thing agrees with whatever story you already have.
Two independent channels settled it before anything was touched: statically, zero calls to invert_sites outside tests and the enum variant never constructed; empirically, the flag and the default printed byte-identical rows. The real first measurement is 33 invert mutants across 13 gates, all killed.
A fourth operator moves a comparison one place: > becomes >=, < becomes <=. Ratchets, floors and tolerances live on a boundary, and a control that tests clearly-worse and clearly-better never tests EQUAL.
Its first run reported survivors in 8 of 13 gates. The scanner tracked quote state per LINE, so every > inside a multi-line docstring became a site — prose about ratchets and usage, on lines 10, 43, 136 and 230 of four different gates, reported as surviving mutants.
Carrying triple-quote state across lines gives the honest figure: 5 gates, 21 mutants, 9 killed, 12 survived. The instrument had failed on its own first run, and the failure was a real count whose meaning was wrong.
Twelve surviving boundary mutants is not twelve holes. A boundary mutant that lives can be a theorem — a place where moving the comparison changes nothing, provably. Classified by hand: two real thresholds, four real semantic, two proven equivalences, two cosmetic display truncations, one plumbing.
Publishing "twelve uncovered boundaries" would have been every word measured and the sentence false.
One of the proven equivalences deserved to be written down beside the code, so the next reader would not re-derive it. A marker was added — a comment naming the line below it as a known equivalence, printed beside the surviving row and never acted on, because suppressing a row on the strength of a comment is how a declared exception stands for a week while being false.
The marker's first implementation took the comment's line number plus two. The proof it was written for is a fifteen-line comment block, so it named a line in the middle of its own explanation. A one-line proof would have passed the test.
Third time in one week that a measuring device was calibrated against the single example sitting in front of it.
The suite's last surviving mutant had been declared uncovered with a reason that had stood for a week: reaching that branch needs a build that SUCCEEDS and then finds no baseline, and a build that succeeds needs the real spec tree, which an empty directory cannot be given.
Every clause of that is true. The conclusion does not follow.
A control does not have to use an empty directory. The note reasoned entirely inside the frame of the helper that file happens to be built around, and never asked whether a stage could keep the real corpus and empty only the thing under test. Splitting the working directory from the root the gate resolves against does exactly that. The branch turned out to be reachable in under a second, and the real tree is only ever read.
The tell was that the justification sounded mechanical and no measurement had produced it. It reads as someone-thought-about-this, which is what makes it durable. One command falsified it.
What none of that says is that the gates are correct, or that they check the right things. It says no mutant in these four families survives its control, and the families are narrow by construction. A fifth operator is a fifth question, and the honest prior after this week is that a new question finds something — three times it found the tool asking it.
The reusable part is not the operators. It is that an instrument returning a plausible number is the hardest failure to see, because plausibility is what you were checking for.
Every figure above is measured, and the limits are named with it.