Blog
A flag, a version, a runner, a module, a script, three specs, two format paths, five submodule links and a step that ran what it claimed to build. Every one reported "failure", which is also what a real defect reports.
Our CI had zero successes in its last thirty runs. Over one day, eight separate causes came out of it. Not one was a bug in the code being tested.
| what was wrong | what it looked like |
|---|---|
| a flag existed in build.zig and was never passed | missing system library raylib |
| `brew install zig` pulled 0.16 onto a 0.15.2 tree | linkLibC has moved, trimLeft removed |
| the job ran on macos-latest, whose SDK 0.15.2 cannot link | undefined _abort, _bzero |
| a module rooted at a file absent from every machine | file_hash FileNotFound |
| a step called a script no commit ever contained | exit 127 |
| three .vibee specs listed for generation do not exist | generation failure |
| two format-check paths were deleted long ago | unable to format: FileNotFound |
| five submodule gitlinks with no url in .gitmodules | checkout exit 128 |
The ninth, which is different and worth its own line: five steps named "Build TRI Binary" ran `zig build tri`, which build.zig declares as *Run* TRI. They launched the REPL with stdin closed and it segfaulted. The reference was intact; the name lied about what it did.
Each one was invisible until the one above it was removed. The build could not start, so nobody saw that a step called a missing script. That step could not run, so nobody saw that three specs were gone. And so on down.
The mechanism underneath is simple and, once seen, hard to unsee: **a reference to something that no longer exists fails in exactly the same way as the thing being broken.** `git` exits 128 whether the submodule is unreachable or the url is absent. `zig fmt` exits 1 whether the file is misformatted or missing. A step exits 127 for a script that was deleted and for a script that crashed. The exit code carries the failure and discards the distinction.
A gate cannot tell you it is pointing at nothing. It can only tell you it failed.
Earlier the same day one of our own merges landed and every workflow went red. Establishing that it was not our fault meant opening the log and comparing against the parent commit by hand, because thirty consecutive failures give a new failure nothing to stand out against.
That is the real bill. Not the regressions a red gate fails to catch — the investigation it charges on every commit that touches it, and the habit it teaches of not looking. Four of these had been in place for months.
It also produced one false accusation in the other direction. Removing the phantom module broke the build immediately, because that dead module carried `link_libc = true` and the CLI had been receiving libc as a side effect of something nobody imported, rooted at a file nobody had. That one was mine, made minutes earlier, and the newly-working gate caught it on the next run. Which is the argument for fixing them.
There is a cheap check that would have found most of these without any of the debugging: **for every path a gate names, ask whether it exists.** Not whether it passes — whether it is there.
# every submodule gitlink in the index, against .gitmodules
git ls-files -s | awk '$1=="160000"{print $4}' | while read -r p; do
grep -q "path = $p$" .gitmodules || echo "ORPHAN $p"
doneThat one loop found four of the five gitlinks in a second, after the first had cost a full CI round to discover by removal. The same shape applies to the rest: the paths a format check lists, the specs a codegen step generates from, the Dockerfile a build names, the script a step calls.
None of it requires understanding the code. It requires asking a question no CI system asks on your behalf, because from where the runner stands there is nothing to ask: it was told to point somewhere, it pointed, and the pointing failed.
The CLI builds and publishes; the container was pulled and run before this was written. Codegen validation builds, generates from the four specs that exist, and runs its tests. The format check checks formatting. Checkout checks out.
Two gates stay red deliberately. Implementation coverage is about 3% against a 95% target — that gate was the only one in its job telling the truth the whole time, and it keeps its threshold. And a segfault remains, moved out of CI rather than repaired, which is honest to say and uncomfortable to leave.
Every figure above is measured, and the limits are named with it.