Blog
Thirty epochs exposed a failure-rate blind spot
A 30-epoch MNIST sweep showed why a failure-rate threshold needs the per-seed values beside it: a passing count can coexist with non-overlapping runs.
- THE COUNTA passing count says how many crossed.
- THE SPREADThe runs did not overlap at all.
- THE BLIND SPOTA rate hides the shape of the data.
[measured — MNIST training sweep] A thirty-epoch run was designed to test whether a failure-rate summary stayed informative as training continued. It did not: the per-seed values exposed a gap that the rate alone hides.
At 30 epochs, the five-seed slice recorded TNF4 at 0/5 failures with 97.82 ± 0.15 mean accuracy. The same slice recorded fp6 e2m3 at 4/5 failures and fp6 e3m2 at 2/5 failures. These are measurements from the MNIST experiment, not a claim about a general format ranking.
A threshold can pass while the runs do not agree
Using a 60% threshold, fp6 e3m2 passes 3/5 runs. But its highest run reaches 71.9 while TNF4’s worst reaches 97.6: the two distributions do not overlap, leaving a 25.7-point gap. A rate counts line crossings; it does not describe a distribution that is uniformly lower.
| Format | 30-epoch failures | Per-seed accuracies |
|---|---|---|
| TNF4 | 0/5 | 98.0, 97.6, 97.8, 97.9, 97.8 |
| fp6 e2m3 | 4/5 | 19.2, 81.0, 9.6, 12.7, 11.3 |
| fp6 e3m2 | 2/5 | 71.9, 65.6, 55.5, 71.4, 59.3 |
Longer training did not add the expected drift
For TNF4, the reported mean moved from 96.76 at 3 epochs to 97.68 at 10 epochs and 97.82 ± 0.15 at 30 epochs. Across eight configurations, the aggregate failure counts were TNF4 0/40, fp6 e2m3 29/40, and fp6 e3m2 24/40.
The per-seed list is the presentation here that cannot hide which runs moved.
What this does not establish
- This is a measured MNIST training sweep, not a measurement on the ALINX AX7203 FPGA and not a silicon result.
- It does not establish a universal accuracy or failure-rate ranking for TNF4, fp6 e2m3, or fp6 e3m2.
- It does not show that a 60% threshold is the right threshold for another task, seed set, or training recipe.
- The experiment does not establish transfer beyond this MNIST configuration; the per-seed list improves the audit trail but does not remove that scope limit.
The useful result is methodological: keep the failure rate for its narrow question, but print every seed beside it. That small addition turns a passing count into a checkable picture of the runs.
What this does not settle
- This is a measured MNIST training sweep, not a measurement on the ALINX AX7203 FPGA and not a silicon result.
- It does not establish a universal accuracy or failure-rate ranking for TNF4, fp6 e2m3, or fp6 e3m2.
- It does not show that a 60% threshold is the right threshold for another task, seed set, or training recipe.
- The experiment does not establish transfer beyond this MNIST configuration; the per-seed list improves the audit trail but does not remove that scope limit.
Receipts
Work with me
Want this kind of check on your own design?
I audit RTL and build independent, bit-exact models, then take the result through synthesis and, when useful, onto an Artix-7 board. The first conformance module is free.