Blog
The tail that had never run
A bitstream-generating CI job was red for eleven days. Underneath: thirteen stacked defects, each invisible until the one above it was cured — a fake chip database, a dependency list nobody had ever executed, and a success step that rejected the first real bitstream in the job’s history. The job now emits a 3,822,704-byte xc7a100t bitstream through a fully open flow.
- THE RED JOBEleven days red.
- THE STACKThirteen stacked defects.
- THE BITSTREAMThe first real bitstream in its history.
A CI job that generates an FPGA bitstream had been red for eleven days. The fix turned out not to be a fix, plural: thirteen distinct defects, stacked so that each one could only be discovered by curing the one in front of it. The job's tail — synthesis, place-and-route, frame generation, bitstream assembly — had never executed. Not once, in the entire history of the job.
This is worth writing down not because thirteen is a large number, but because every layer was invisible from outside while the layer above it was failing. A pipeline that dies at step 3 tells you nothing about steps 4 through 14 — and "nothing" here is literal: no log line, no type error, no lint warning. Code that has never run is not code that works, and it is not code that fails. It is a blank.
The onion, compressed
| layer | what was wrong | why the layer below was invisible |
|---|---|---|
| 1–2 | the yosys lint loop could never pass: 1 of 32 modules survived; missing -sv and -DSIMULATION flags | everything after lint was skipped |
| 3–6 | four code-generator bugs: Verilog keywords used as identifiers, escaping applied before field-flattening instead of after, call sites and return-lvalues unescaped | generated files never reached synthesis |
| 7 | the workflow cloned YosysHQ/nextpnr — which does not contain a xilinx architecture at all; it lives in the openXC7/nextpnr-xilinx fork | cmake died before anything else could |
| 8 | the chip database was one megabyte of /dev/zero, so place-and-route had nothing true to route against | a fake artifact let downstream steps 'pass' into a trap |
| 9 | pip3 install sby — SymbiYosys is not on PyPI and never was; the formal job failed at install on every run since the job existed | nothing after the install line had ever run |
| 10 | timeout ceilings: a 45-minute limit killed a healthy 46-minute run; separately, apt hangs on a dead mirror burned whole ceilings in silence | a killed job looks like a failed job |
| 11 | the build driver hardcodes build/fpga/prjxray, its PYTHONPATH, and the prjxray-db path — while the workflow staged all three under ~/ | the first run ever to finish P&R died one step later |
| 12 | prjxray's requirements.txt pins editable submodule paths and tools incompatible with current Python; the real import chain is six packages | the dependency list had never been executed |
| 13 | the metrics step demanded a file literally named bitstream.bit; the driver names output after the top module — zerodsp_top.bit | the first real bitstream in the job's history failed the step that reports success |
Layer 8 deserves its own sentence. A fake artifact is worse than a missing one: the /dev/zero chip database allowed later steps to run and fail for the wrong reason, which is how a genuinely broken pin constraint hid behind a database that could not have routed anything. When a step cannot yet produce a real input for the next step, the honest move is to fail loudly, not to manufacture a placeholder.
What ended it: pre-verifying the next layer before pushing
The first eight layers were peeled the expensive way — one CI round-trip each, forty-five to ninety minutes per lesson. The last three were peeled in one push, because each was checkable locally before committing anything:
- Layer 11: grep the build driver's source for its path joins. The driver defines the contract; the workflow either satisfies those exact paths or passes flags. Guessing standard locations satisfies neither.
- Layer 12: don't read the dependency file — execute the tool. PYTHONPATH=<clone> python3 fasm2frames.py --help against the exact candidate package list is a one-minute local experiment that replaces a forty-five-minute CI round-trip.
- Layer 13 (and one mine defused before it detonated): two upload-artifact@v4 steps shared one artifact name. v4 errors on duplicate names — the second step only ever passed because the first found nothing to upload. Fixing the first would have armed the second.
=== FASM → Frames ===
Frames: build/fpga/synth/design.frames
=== Frames → Bitstream ===
Bitstream: build/fpga/zerodsp_top.bit (3822704 bytes)
=== FPGA E2E build finished ===3,822,704 bytes is the exact configuration size for an xc7a100t. The flow that produced it is fully open: yosys for synthesis, nextpnr-xilinx for place-and-route against a database generated by bbaexport from the openXC7 tree, prjxray's fasm2frames for frame assembly, xc7frames2bit for the container. No Vivado anywhere in the path.
The rule the whole story compresses into
A job that has always failed early contains a tail that has never run, and that tail should be treated as unwritten code — reviewed, executed locally where possible, and never trusted on the grounds that it looks plausible. The corollary cuts the cost: while fixing layer N, spend the extra ten minutes checking layer N+1 by hand. Every one of the last three layers was visible in advance to anyone who looked — one grep, one --help, one directory listing on a public repository.
What this does not settle
- The bitstream has never been loaded onto a board. The CI machine has no FPGA attached, so correctness beyond the toolchain’s own internal checks — including whether the design does anything at all on silicon — is unproven. Flashing on real hardware is the only test that counts, and it has not happened for this artifact.
- Timing comes from nextpnr’s model, not from measurement. No Fmax claim is made and none should be inferred.
- The flow builds the minimal profile only; the larger profiles have not been through the repaired pipeline.
- The apt-mirror hangs that burned whole timeout ceilings are bounded, not fixed — a retry-with-timeout patch for the install steps exists but has not landed.
Receipts
- gHashTag/t27 PR #2216 — the pull request carrying all thirteen layers
- The first run to produce a real bitstream — fpga-bitstream job log
- Issue #2214 — the red-since-birth diagnosis across jobs
- Issue #2215 — fpga-build: every job red, root causes enumerated
- openXC7/nextpnr-xilinx — the fork that actually contains the xilinx architecture
Work with me
Want this kind of check on your own design?
I audit RTL and build independent, bit-exact models, then take the result through synthesis and, when useful, onto an Artix-7 board. The first conformance module is free.