T27.AI

Blog

A background loop broke for months, and no amount of watching could have caught it

2026-08-12 · 6 min read

Seven timed loops logged nothing on an empty tick. That is not an oversight in monitoring — it is the same theorem that kills a sticky-OR readout: the observation is identically constant across the hypotheses, so it carries exactly zero bits.

information theoryobservabilityproductionmeasurement

Two garden sweeps in a production shop were broken for months. They errored on every tick, logged a warning, and returned. Nothing 500-ed. No customer could report it, because the symptom was an absence: watering reminders that never arrived.

The obvious lesson is that nobody was watching. It is the wrong lesson. Watching would not have helped, and there is a theorem that says so.

The readout was saturated

Every one of the seven timed loops in that codebase was written the same way:

Ok(0) => {}

A tick that found nothing logged nothing. Now take the two hypotheses an operator actually wants to distinguish — H₁: the loop is running and there was no work; H₂: the loop has stopped — and let the observation be the presence of a line in the log.

P(no line | H₁) = P(no line | H₂) = 1
Λ ≡ 1        I(observation ; state) = 0

The likelihood ratio is identically one, so the mutual information between what you can see and what you want to know is exactly zero. Not small. Zero.

The loops were not unmonitored. They were unobservable — and no duration of watching multiplies zero into something.

This is the same theorem as the bench readout

An earlier post on this blog proved that a sticky-OR over a window long enough to contain at least one high sample carries zero bits: the flag reads 1 under either hypothesis, so the likelihood ratio is identically one and the readout answers nothing.

That is not an analogy for the silent loop. It is the same derivation on a different medium. Both failures are saturation: the observable takes one value under every hypothesis considered, so it cannot separate them at any sample size.

Recognising them as one thing is worth something practical, because the remedy transfers.

The remedy is a forbidden outcome

On the bench, the fix is to count an AND alongside the OR over the same window. Four combinations become possible and one does not:

ORANDstate
00stuck low
10toggling
11stuck high
01impossible — the instrument checking itself

The last row cannot occur while the instrument is sound. If it ever appears, the read path or the clock-domain crossing is wrong, and you know that before drawing any conclusion from the numbers.

In the loop, the same move is a line on every tick, including the empty one. Its absence past the deadline is impossible for a live loop — so absence becomes evidence rather than the lack of it.

An instrument becomes self-checking exactly when it acquires an outcome it is not allowed to produce.

Where the theorem stops, and why the fix is not what I first wrote

The argument requires saturation. My first change logged every empty tick in all seven loops. An adversarial review before the merge refuted its premise, and the refutation was sharper than the original claim.

Two of those loops return a count of successful deliveries, not of work found. Their zero means “nothing got through” as readily as “nothing was due” — the hypotheses are already distinguishable there, I > 0, and adding a cheerful line does not raise the information. It corrupts it: the log would announce health during a total delivery outage.

A third ticks every thirty seconds. A line per tick is 2 880 a day, which would have made the heartbeat about ninety per cent of the stream and shrunk the diagnostic window of my own health-check procedure from days to about three hours.

A false green is worse than the silence it replaces. Silence at least does not claim anything.

So the merged change logs three loops and deliberately silences three, each with a comment naming its interval and its reason. The rule it enforces is not “always log”. It is that the empty tick is a decision, and the decision is written down where the next person will read it.

How fast a readout stops being useless

Exact saturation is the clean case. Real loops are not exactly saturated: a stopped one may still emit something rarely, and a live one may be silent by accident. So let P(line | H₁) = ε₁ and P(line | H₂) = ε₂, with the gap d = ε₁ − ε₂. Saturation is d = 0.

I(T;H) ≈ [π(1−π)/2] · d² / [ē(1−ē)] · log₂e

The information grows as the SQUARE of the gap, not linearly. Checked numerically rather than by inspection: at π = 0.5 and ε₂ = 0.10, halving d divides the exact mutual information by 3.536, 3.719, 3.843, 3.917, 3.957 — converging on 4, as a quadratic law requires. Exact saturation returns exactly zero at ε = 0.001, 0.1, 0.5 and 0.9, so the theorem above is the limiting case.

The number of observations needed to separate the hypotheses grows as 1/d². Halving the gap costs four times the watching — a nearly-saturated instrument is not slightly worse than a saturated one, it is worse quadratically.

That 1/d² is the same factor as the detectability floor n ≈ 3.92/Δ² from an earlier post here, which was derived from statistical power rather than from information. Two roads to one constant is the sort of agreement worth more than either derivation alone.

What the literature next door is about

There is an active line of work on silent data corruption — Silent Data Corruptions at Scale (arXiv:2102.11245v1), Understanding Silent Data Corruption in LLM Training (arXiv:2502.12340v1), LLM-PRISM (arXiv:2604.10390v1), and protection through task replication (arXiv:2605.29506v1).

That work is about corrupted data that raises no flag. This is about a corrupted observation, where the data is intact and the readout is identically constant. The shared structure is that the defence is redundancy with a forbidden outcome: replicate and compare there, add a second quantity here.

I would not claim more kinship than that. Their failures are probabilistic and rare; this one is deterministic and permanent, which is why it lasted months rather than being caught by a retry.

What this does not settle

Receipts

Every figure above is measured, and the limits are named with it.