T27.AI

Blog

Half of your activations are negative. They carry 1.8% of the energy

2026-08-11 · 6 min read

Post-activation tensors are half negative by count and 1.8–6.2% by energy, while weights are symmetric on both measures — which decides how a 4-bit alphabet should spend its codes.

This is an observation about transformers, not about any particular numeric format. It takes two lines to check on a model you already have.

The table

Two columns describing the same tensor. For weights they agree. For GELU outputs they differ by a factor of twenty-seven.

What it means

One-sidedness is a property of a tensor’s energy, not of its count. GELU passes almost half its values into the negative region, but they sit near zero: nearly everything carrying magnitude is positive.

The count view says the distribution is symmetric, so spend the sign bit evenly. The energy view says the negative half is nearly empty. A quantiser that allocates on the count spends half its code space on 2% of the signal.

Note where the disagreement stops. Weights and post-LayerNorm tensors are symmetric on both measures, so the rule is not “transformers are one-sided” but the sharper one: post-activation tensors are, and only by energy. The same network needs different treatment for weights and activations at the same layer.

It predicts a result, in both directions

An asymmetric 4-bit alphabet, compared against DialectFP4 using the codebook from that paper’s own Figure 4 and its scale rule, wins on every activation class and loses on weights:

The loss is in the table on purpose. It wins exactly where the asymmetry exists and pays about 6% where it does not — which is a prediction landing, not a lucky sweep.

Check it on your own model

(x < 0).float().mean()              # fraction negative
(x[x<0]**2).sum() / (x**2).sum()    # energy share of the negative half

If the second number is much smaller than the first, you have the same asymmetry and your quantiser does not know about it. If they are equal — as they will be for weights — an asymmetric alphabet has nothing to offer you, and that is visible before you build one.