T27.AI

Blog

Four hundred and twelve tests that were sentences

2026-08-22 · 7 min read

Thirty-four conformance vector files, 512 cases, zero ever executed — and when counted, 412 of those cases carried no inputs and no expected values at all. This is how a corpus becomes decorative, why the first classifier reported 147 where an independent pass said 100, and the three-verdict registry (executed / numbered debt / aspirational) that lets a small true number replace a large false one.

A repository had thirty-four conformance vector files. Five hundred and twelve test cases. Zero of them had ever been executed against the hardware they described — and when we finally counted what was inside, four hundred and twelve of those cases carried no data at all: an identifier, a sentence, and nothing else. No inputs. No expected values. Documentation wearing the shape of a test.

How a corpus becomes decorative

Nothing dramatic happened. A file landed in the conformance directory. A CI job read its stored 'verdict' field — a string someone typed once — and printed it into the run summary. Another file landed. The count grew. 'Thirty-four vector files' became a sentence people said about the project, and every part of it was true except the part that mattered: no runner ever opened them.

The gap is easy to miss because both halves look like work. Writing a vector file IS work. Reading a summary that says CLEAN is reassuring. The missing step — actually applying the vectors to the RTL — leaves no trace when it is absent, because absence has no output.

The count, and the counter that lied about it

The first classifier we wrote reported 147 data-carrying cases. An independent pass in another language reported 100. Two instruments disagreeing is not a rounding difference to average away: one of them is wrong, and until you know which, neither number can be written down.

Ours was wrong. It split each JSON object on commas to find fields — and descriptions contain commas. 'Sync FIFO reports is_sync=true, is_async=false' splits into two fragments, the second of which looks exactly like an unrecognised field. Every prose case with a comma in its sentence was promoted to data-carrying. The rewrite tracks string state and collects only real keys; it now reproduces the independent count on all thirty-four files.

$ tri vectors debt        # 2026-08-22, as first measured
1 executed, 9 debt, 24 prose-only (34 files);
512 cases, 100 carrying data (19%).

$ tri vectors debt        # after spi joined the executed set
2 executed, 8 debt, 24 prose-only (34 files);

Three verdicts, never two

The instinct on discovering this is to say 'run them all'. That instinct is the same one that produced the problem: it treats every artifact as a test that merely hasn't been run yet. Most of these cannot be run at all — not because of a bug, but because they describe behaviour that no interface in the design exposes. One module's vectors check LED output patterns; the specification contains no LED function of any kind.

So the registry sorts every file into exactly one of three verdicts, and the third one is the important one:

• Executed — a call template maps the case shape onto real entry points. Gate hard on it, and prove the gate reacts: plant a fault, see it fail.
• Debt — executable in principle, blocked by a numbered defect. Print it, link the issue, never count it as coverage.
• Aspirational — describes behaviour no current interface exposes. Classify it honestly and stop. Running it would test an invention.

Two verdicts force every unexecutable artifact to masquerade as one of them, and 'not yet' is where they all end up. The third category is what lets you state a real number: the honest ceiling for execution here is ten files, not thirty-four.

The ratchet

Fixing twenty-four files is a decision someone else owns. Stopping the twenty-fifth is not. A gate now freezes the existing prose-only files as named debt and fails when a new one lands — the same baseline-as-debt pattern the repository already uses elsewhere. A planted prose-only file is caught with its case count; a planted file with a single input field passes. The class cannot grow again without someone deliberately recording that they want it to.

Two modules now execute for real: eighteen cases across seven groups in one, three in another, each with a planted-fault control proving the check reacts. That is a small number and it is the true one, which is the only property that makes it worth having. (This sentence first said nineteen. The runner prints eighteen; the correction is recorded here rather than made silently, because a post about counting things honestly has no business rounding its own.)

What this generalises to

Any artifact whose presence is counted but whose execution is not — fixtures, golden files, property sets, benchmark suites — drifts toward decoration at exactly the rate nobody checks. The check is cheap and specific: for each artifact, name the code path that consumes it. If you cannot, you have documentation, and calling it something else is the only real defect.

Work with me

Want this kind of check on your own design?

I audit RTL and build independent, bit-exact models, then take the result through synthesis and, when useful, onto an Artix-7 board. The first conformance module is free.

Discuss your projectSee the verification service