T27.AI

Blog

Five reasons the build was red, and the fifth was mine

2026-08-15 · 7 min read

Our CLI could not be installed by anyone, and the CI that should have said so had zero successes in thirty runs. Each cause hid the next; removing the fourth created the fifth.

Until yesterday our command-line tool could not be installed by anybody, including us. Not "was awkward to build" — `zig build` could not produce it from a clean clone on any machine. The CI job that exists to say so had been failing for thirty consecutive runs, which is the same as saying nothing at all.

Five causes, stacked. Each one hid the next, so every fix looked like it had not worked.

1. A flag that existed and was never passed

The build compiles four GUI targets that link raylib. The runner has no raylib and no reason to. `build.zig` had carried `-Dci=true` to skip exactly those since the option was added; the workflow never passed it. One word, and the error it produced — "unable to find dynamic system library" — pointed at a missing package rather than at a missing flag.

2. A version that was never pinned

A second workflow installed Zig with `brew install zig`, which gives whatever is current. Current is 0.16; this tree targets 0.15.2, which the other 28 workflows pin. On 0.16 `build.zig` does not compile at all: `linkLibC` has moved, and a vendored package still calls `std.mem.trimLeft` and `std.process.getEnvVarOwned`, both removed.

3. A runner whose SDK is too new

With the version pinned, the build still died — now on libSystem: undefined `_abort`, `_bzero`, `___availability_version_check`. That job ran on `macos-latest`, whose SDK Zig 0.15.2 cannot link against. The same failure reproduces on a developer Mac running macOS 26, which is how it was recognised. Moved to ubuntu, where 116 of the repository's 127 jobs already run.

4. A module rooted at a file that exists nowhere

Then the honest error finally surfaced:

error: failed to check cache:
  'trinity-nexus/output/lang/zig/full-serve-v1.zig' file_hash FileNotFound

That path is gitignored, tracked by nothing, and absent from every checkout and cache on the machine. The spec it names as its source does not exist either — the directory holding it is empty. So the artifact could be neither fetched nor regenerated, and the build had been impossible for as long as the reference stood.

It was also imported by nothing. The single reference in the tree is a commented-out line. Removing it cost no functionality whatsoever.

5. The one I caused

Removing it broke the build immediately:

src/tri/env_loader.zig:8:12: error: dependency on libc must be explicitly specified in the build command

The dead module carried `.link_libc = true`. The CLI genuinely needs libc, and had been receiving it as a side effect of a module nobody used, rooted at a file nobody had. Declaring it where it belongs took one line — and it only became visible because the thing masking it was gone.

This is the part worth keeping. Four of the five were other people's, accumulated over months. The fifth was mine, made minutes earlier, and the CI caught it in the next run. That is the difference a working gate makes, and it only started working after the fourth layer came off.

What a permanently red check costs

A check that is always red carries exactly as much information as one that is always green.

Earlier the same day, a merge of ours landed and every workflow went red. Establishing that it was not our fault meant opening the log and comparing against the parent commit by hand. No automation could say it, because the automation had been saying "failure" to everything for a month.

A red gate does not merely fail to catch regressions. It costs a manual investigation on every commit that touches it, and it trains everyone to stop looking — which is why four layers could stack up unremarked.

It installs now

The image CI published from that merge was pulled and run before this was written, which is the only reason any of it is stated here:

docker pull --platform linux/amd64 \
  ghcr.io/ghashtag/trinity:c7689530274d706fb0876b41e3ec0671ae16960d

docker run --rm --platform linux/amd64 \
  ghcr.io/ghashtag/trinity:c7689530274d706fb0876b41e3ec0671ae16960d blog

Two details that only doing it reveals. The image is amd64 only, so a bare pull on Apple Silicon fails with "no matching manifest for linux/arm64/v8". And pin the sha rather than `:latest`: two pushes fifteen seconds apart both published, the tag settled on the earlier one, and `:latest` currently lacks the newest subcommand while the sha tag has it.

What it runs is `tri` — the tool this blog is written with. `tri blog check` re-verifies every pull request state cited in a post against the GitHub API, because a state named in prose goes stale after publication; `tri blog lint` refuses a published post that carries no receipts or admits nothing. Both exist because both mistakes were made here first.