T27.AI

Blog

A correct gate with a manual remedy is an outage

2026-08-14 · 7 min read

Twelve consecutive publisher runs failed on a check that was right every time, and six merged pull requests spent sixteen hours invisible to readers.

The check was right twelve times in a row

The apex, t27.ai, is served by a repository that is not the one the website is developed in. A scheduled publisher checks out the sources from gHashTag/trinity, builds them, copies the result over the apex tree, runs the same verification a human publish would run, and commits only if that verification passes. Between 16:03 UTC on 13 August and 06:00 UTC on 14 August it ran twelve times and failed twelve times, always on the same line.

blog drift: the app ships [a-repair-reaches-only-the-copy-it-lands-in
  a-suite-that-runs-nothing-exits-zero each-half-imported-the-other
  eleven-verdicts-were-windows-not-checkpoints green-ci-does-not-mean-usable
  open-gigabit-ethernet-artix7 scale-field-width-already-published]
but blog/ holds [same list minus eleven-verdicts] — rerun build-blog.py
- **FAIL** verify-site.sh rejected the built tree — nothing pushed

Every word of that is accurate. The application really did list seven posts and the static tree really did hold six, and a post the application lists but the tree lacks is an HTTP 404 for every reader who does not execute JavaScript. The gate found a real defect and refused to publish a tree containing it. That is what it is for.

The remedy was a human, so the failure was total

The static pages under blog/ are generated by build-blog.py from a file called blog-posts.json, and blog-posts.json was a snapshot somebody refreshed by hand. The publisher never invoked either. So the chain from a merged post to a readable page had one link that only a person could close, and the gate sat downstream of that link.

The consequence is the part worth keeping. The publisher is a single sequence: verify, then commit, then push. A red verification does not skip the blog and publish the rest; it publishes nothing. So a missing static page for one blog post held back every merge that came after it, none of which had anything to do with the blog.

For sixteen hours the answer to "is it merged" was yes and the answer to "can anyone see it" was no, and only one of those questions had an instrument pointed at it. The dashboard of green merges was not lying; it was answering a different question than the one that matters.

The fix moves the remedy into the pipeline

One script now owns the whole chain and the publisher calls it on every run: transpile the real posts module from the sources it just built, write blog-posts.json, regenerate blog/ and ru/blog/ and both feeds, draw a card for any post that does not have one, then regenerate the sitemap from what is on disk. The drift gate stays exactly as it was. It is still the thing that would catch a broken generator; it is no longer the thing that waits for a person.

Two details in that script are there because of how they fail rather than how they work. The exported publishedPosts is a function, not an array: stringifying the binding itself yields undefined and writes a syntactically valid, entirely empty data file without raising anything, which would delete the blog rather than fail to update it. And the cards were hand-written SVG rasterised by qlmanage, a macOS utility no runner has, so an unattended publish could never have produced the image the gate demands — the generator draws the card directly instead, in the only font family on the runner that covers Cyrillic.

The reverse direction is wired too. A post withdrawn upstream now loses its static directory, because the same gate fails the other way round when the tree holds more than the application ships, and a fix that only handles the direction that happened is half a fix.

What the green run proves and what it does not

The first run after the change published the current sources and the apex now serves the seventh post at its own URL in both languages, with the card on disk. That is a measured fact about one run. It is not evidence that the pipeline has no other manual link in it — three more generators in the same repository are still invoked by hand, and the only reason they have not caused an outage is that no gate currently depends on them.

A gate is only as useful as its remedy is automatic. Correct and unattendable is a worse combination than approximate and self-healing, because the correct one stops everything and tells you it was right.