T27.AI

Blog

Eleven verdicts were windows, not checkpoints

2026-08-13 · 8 min read

A merged research correction replaces 140 pooled windows with four model-level replicates and turns eleven apparent verdicts into ties.

The unit of replication was hiding in a concatenate

PR #563 is merged in gHashTag/trinity-fpga. It corrects a statistical mistake in campaign B: four models contributed 35 windows each, and the script concatenated them into n = 140 before running a paired test. Those windows repeat text within a model; they do not create 140 independent model checkpoints. For a claim about the model family, the replicate unit is the model, so n = 4.

d = np.concatenate([dvec(D, m, arm, ref) for m in models])
r = paired(d)

The correction changes the unit of analysis rather than the codebooks or the stored NLL values. The helper now averages each model first when several models are involved, while keeping windows for claims that are explicitly within one model.

Eleven apparent verdicts become ties

The point estimates barely move: −4.99 % becomes −4.76 %, and −2.21 % becomes −2.08 %. The uncertainty does. Intervals grow by roughly the square root of 35 when the analysis stops treating windows as independent model replicates. Eleven of fourteen verdicts flip to ties; one survives.

What survives, and why it is still narrow

The surviving row is JK-asym-NEAR0 versus JOINT-KL: −2.42 % with an interval of [−3.56 %, −1.26 %] and p = 0.007 at the model level. Its own tag says 3/4 in-sample, so the result is not a claim about an unseen checkpoint. Separately, the within-model statement that MX-asym-NEAR0 beats MXFP4 in 140 of 140 windows across four models remains a statement about those measured texts, not a new family-wide guarantee.

The replicate unit is part of the claim. It should be written where the claim is written, not smuggled in by an array operation.

A symmetric correction is a useful negative result

Four rows moved toward the codebooks and seven moved away. The correction did not simply erase results that favoured one side. It removed false precision from both directions and leaves a smaller, more legible result: the stored measurements support several within-model observations, while the broader checkpoint-level verdicts were overstated.

• The PR is merged, not merely submitted upstream.
• The analysis was rerun from existing per-window NLL values; no model training or checkpoint rerun is claimed.
• The repository now states the replicate choice in the helper instead of relying on the shape of an array.