Blog
A bitstream-generating CI job was red for eleven days. Underneath: thirteen stacked defects, each invisible until the one above it was cured — a fake chip database, a dependency list nobody had ever executed, and a success step that rejected the first real bitstream in the job’s history. The job now emits a 3,822,704-byte xc7a100t bitstream through a fully open flow.
A CI job that generates an FPGA bitstream had been red for eleven days. The fix turned out not to be a fix, plural: thirteen distinct defects, stacked so that each one could only be discovered by curing the one in front of it. The job's tail — synthesis, place-and-route, frame generation, bitstream assembly — had never executed. Not once, in the entire history of the job.
This is worth writing down not because thirteen is a large number, but because every layer was invisible from outside while the layer above it was failing. A pipeline that dies at step 3 tells you nothing about steps 4 through 14 — and "nothing" here is literal: no log line, no type error, no lint warning. Code that has never run is not code that works, and it is not code that fails. It is a blank.
| layer | what was wrong | why the layer below was invisible |
|---|---|---|
| 1–2 | the yosys lint loop could never pass: 1 of 32 modules survived; missing -sv and -DSIMULATION flags | everything after lint was skipped |
| 3–6 | four code-generator bugs: Verilog keywords used as identifiers, escaping applied before field-flattening instead of after, call sites and return-lvalues unescaped | generated files never reached synthesis |
| 7 | the workflow cloned YosysHQ/nextpnr — which does not contain a xilinx architecture at all; it lives in the openXC7/nextpnr-xilinx fork | cmake died before anything else could |
| 8 | the chip database was one megabyte of /dev/zero, so place-and-route had nothing true to route against | a fake artifact let downstream steps 'pass' into a trap |
| 9 | pip3 install sby — SymbiYosys is not on PyPI and never was; the formal job failed at install on every run since the job existed | nothing after the install line had ever run |
| 10 | timeout ceilings: a 45-minute limit killed a healthy 46-minute run; separately, apt hangs on a dead mirror burned whole ceilings in silence | a killed job looks like a failed job |
| 11 | the build driver hardcodes build/fpga/prjxray, its PYTHONPATH, and the prjxray-db path — while the workflow staged all three under ~/ | the first run ever to finish P&R died one step later |
| 12 | prjxray's requirements.txt pins editable submodule paths and tools incompatible with current Python; the real import chain is six packages | the dependency list had never been executed |
| 13 | the metrics step demanded a file literally named bitstream.bit; the driver names output after the top module — zerodsp_top.bit | the first real bitstream in the job's history failed the step that reports success |
Layer 8 deserves its own sentence. A fake artifact is worse than a missing one: the /dev/zero chip database allowed later steps to run and fail for the wrong reason, which is how a genuinely broken pin constraint hid behind a database that could not have routed anything. When a step cannot yet produce a real input for the next step, the honest move is to fail loudly, not to manufacture a placeholder.
The first eight layers were peeled the expensive way — one CI round-trip each, forty-five to ninety minutes per lesson. The last three were peeled in one push, because each was checkable locally before committing anything:
=== FASM → Frames ===
Frames: build/fpga/synth/design.frames
=== Frames → Bitstream ===
Bitstream: build/fpga/zerodsp_top.bit (3822704 bytes)
=== FPGA E2E build finished ===3,822,704 bytes is the exact configuration size for an xc7a100t. The flow that produced it is fully open: yosys for synthesis, nextpnr-xilinx for place-and-route against a database generated by bbaexport from the openXC7 tree, prjxray's fasm2frames for frame assembly, xc7frames2bit for the container. No Vivado anywhere in the path.
A job that has always failed early contains a tail that has never run, and that tail should be treated as unwritten code — reviewed, executed locally where possible, and never trusted on the grounds that it looks plausible. The corollary cuts the cost: while fixing layer N, spend the extra ten minutes checking layer N+1 by hand. Every one of the last three layers was visible in advance to anyone who looked — one grep, one --help, one directory listing on a public repository.
Every figure above is measured, and the limits are named with it.