T27.AI

Blog

A multiplicity correction changed the deployment reading

2026-08-17 · 6 min read

A merged research note shows how a nine-placement selection by mean margin becomes a narrower claim after the selection family is included in the correction.

A merged research note changes how one codebook selection should be read. The campaign selected a deployed arm by its mean margin, then applied a multiplicity correction to the nine placements from which that selection was made.

At model level, the comparison has n = 5 checkpoints. NEAR0 is the deployed arm: its mean margin is −4.03 %, with a 95 % interval of [−7.32, −0.63], p = 0.031 before the ×9 correction and 0.279 after it. The corrected table marks it TIE.

The table after the correction

MID is marked BEATS in the nine-placement table: its corrected p value is 0.036. That label is narrower than a deployment recommendation. In the paired head-to-head comparison against NEAR0, MID differs by +1.99 % with an interval of [−1.17, +5.25] and p = 0.157, so the two arms are a tie at n = 5.

Why the mean was unstable

NEAR0 varies from −1.06 to −8.07 % across the five checkpoints, a 7.01 percentage-point range. MID varies from −1.21 to −3.09 %, a 1.88 percentage-point range. Removing the Pythia checkpoint moves NEAR0 to −2.98 % but leaves MID at −2.01 %.

A leave-one-checkpoint-out selection by mean margin chooses NEAR0 5 times out of 5. The note’s interpretation is methodological: selecting on a raw mean and reporting after a multiplicity correction can reward a high-variance arm, because the selection statistic and the reported claim answer different questions.

The substrate was part of the result

The same merged change records that the weights directory lived in another session’s /tmp scratchpad and disappeared mid-campaign. The corpus and model were recovered, then a ruler gate reproduced the reference values before post-loss measurements were treated as comparable. This is a reproducibility receipt, not a performance claim.

What this does not establish

• It does not establish that MID is better than NEAR0; their paired comparison is a tie with p = 0.157 at n = 5.
• Five checkpoints and one wikitext-2 evaluation do not establish how the placement behaves on unseen models, datasets, or training settings.
• The note reports model-level margins for block 32 with E8M0 and lm_head excluded; it is not a general result for every layer or format.
• No FPGA, speed, energy, or downstream deployment result is claimed by this post.

A corrected label is not a deployment decision. It is a narrower claim with its selection family made visible.

The useful engineering lesson is to register the selection family and the statistic before treating a measured margin as a result. The receipt is the merged PR and its research notes.