A bend at zero
You will learn
Why a ReLU needs a test with a negative input before anyone can say it bends.
Recording pending: it waits on tri test and tri mutate plant from gHashTag/t27#7400, the two commands the recording runs, and until then the widget below is a placeholder that shows no run. A neuron passes its sum through an activation, and in gft_relu.t27 that is relu(x) = max(0, x). on_comb returns 0 for zero and for any input with the sign bit set, and x itself otherwise. The header says ReLU has an exact 0/1 gradient and that a 2-layer ReLU net solves XOR, which a linear GF-T model cannot. The browser skips all 4 tests because its runner does not know assert_eq yet; the native t27c runs all 4, all pass, none vacuous. The recording lets negative inputs through, which turns relu into the identity, a straight line. Exactly one test fails, negz; no other test has a negative input. Every byte in the recording was printed by the command; only the typing is staged.
Try it
In the recording, find the changed line and the value negz expects; then in the spec frame find the comment on that line and check whether it still describes the code after the change.
Recording pending: waits on tri test and tri mutate plant from gHashTag/t27#7400. Until then this page is a placeholder and shows no run.
specs/ternary/gft_relu.t27
module GftRelu;
// #1764 + GF-T: a GF-T ReLU activation -- relu(x) = max(0, x). For a signed GF-T16
// input, returns x if x > 0 else 0 (raw 0). Unlike the sign/trit quantizer, ReLU
// has an EXACT 0/1 gradient (no straight-through estimate needed), so it enables
// clean multi-layer backprop on GF-T (see tools/gft_deep_demo.py: a 2-layer ReLU
// net solves XOR, which a linear GF-T model cannot).
//
// Input: x signed GF-T16 (u32). Output: max(0,x) as GF-T16 (u32).
fn on_comb(x: u32) -> u32 {
if (x == 0) { return 0; }
if ((x >> 16) == 1) { return 0; } // negative -> 0
return x; // positive -> identity
}
// relu(+1.0) = +1.0 (0x5000).
test pos { assert_eq(on_comb(20480), 20480); }
// relu(-1.0) = 0.
test negz { assert_eq(on_comb(86016), 0); }
// relu(0) = 0.
test zero { assert_eq(on_comb(0), 0); }
// relu(+2.0) = +2.0 (0x5200).
test pos2 { assert_eq(on_comb(20992), 20992); }
endmodule
All lessons
Module 1 · Lab: our own research
A number format of our own, an honest scoreboard, and a model's tables multiplied on the board.
Module 2 · AI numbers: the MX block
How AI chips keep weights in a few bits: one shared scale per block, the scale byte itself, and what one outlier does to its neighbours.
Module 3 · Ternary weights
Weights that are only minus, zero or plus a scale, the five rules a ternary alphabet must pass, and a test pass that checked nothing.
Module 4 · The Ternary Network Float
A rule the compiler enforces before any test runs, and a 17-bit float whose exponent is four balanced trits.
Module 5 · Arithmetic on signed numbers
Multiply two signed numbers, add them when their signs differ, and do both at once in a multiply-accumulate.
Module 6 · Parts of a neuron
A ReLU that bends at zero, a power of two for softmax, and an argmax that names the answer.
Module 7 · Learning from a mistake
A loss that prices a wrong guess in bits, one step that moves a weight against its gradient, and the hidden layer that XOR needs.
Module 8 · BitNet: ternary networks
A threshold that squeezes a sum back to three values, one neuron that becomes a different function when its weights change, and a neuron that reads its inputs 27 trits at a time.
Module 9 · The ternary MAC as a chip
The 27-trit dot product as wires with no register, the same sum added into a register on every clock, and a small whole network to close the course.