ELU: a floor, not a cliff
You will learn
How ELU keeps a soft negative side with a floor at -alpha, and why its slope never quite dies.
ELU keeps the positive side of ReLU and replaces the cliff on the negative side with alpha (e^x - 1), a curve that flattens toward -alpha. With alpha 1, x = -3 gives -0.9502, already near the floor of -1, and the slope there is 0.0498: small, but not 0, so the unit can still move. Because the negative outputs are not thrown away, a layer's outputs average closer to 0. The spec elu_activation.t27 computes the curve, the slope and the floor, and its own batch at -1, 0 and 2: -0.632, 0 and 2.
Try it
Set alpha to 0.5 and read the floor; then find how far left x must go before the slope is under 0.01.
Drag along ELU: x above zero, alpha (e^x - 1) below. The negative side never falls under -alpha, so outputs average closer to zero, and its slope never quite dies.
specs/ml/activation/elu_activation.t27
// SPDX-License-Identifier: Apache-2.0
// t27/specs/
// Derivative | φ² + 1/φ² = 3 | TRINITY
// #8980: `math::exp` named a module this spec never imports, forward_batch grew
// an array with `+`, and every test called a two-argument function with one
// argument and asserted `result != undefined`. Now each test checks a value.
module Elu;
use base::types;
use math::constants;
// ═══════════════════════════════════════════════════════════
// 1. Constants
// ═══════════════════════════════════════════════════════════
const DEFAULT_ALPHA : f32 = 1.0;
// ═══════════════════════════════════════════════════════════
// 2. Types
// ═══════════════════════════════════════════════════════════
pub const ELUConfig = struct {
alpha : f32,
};
// ═══════════════════════════════════════════════════════════
// 3. Core Functions
// ═══════════════════════════════════════════════════════════
// forward(x, alpha) = x for x > 0, alpha * (exp(x) - 1) otherwise
fn forward(x: f32, alpha: f32) -> f32 {
if (x > 0.0) {
return x;
}
return alpha * (exp(x) - 1.0);
}
// forward_batch: output[i] = forward(input[i], alpha)
fn forward_batch(input: []const f32, alpha: f32, output: []f32) -> void {
var i : usize = 0;
while (i < input.len) {
output[i] = forward(input[i], alpha);
i = i + 1;
}
}
// derivative(x, alpha) = 1 for x > 0, alpha * exp(x) otherwise
fn derivative(x: f32, alpha: f32) -> f32 {
if (x > 0.0) {
return 1.0;
}
return alpha * exp(x);
}
// The inputs the tests and invariants share: one negative, zero, one positive.
fn negative_input() -> f32 {
return -1.0;
}
fn valid_input() -> f32 {
return 2.0;
}
fn batch_at(index: usize) -> f32 {
const input : [3]f32 = [-1.0, 0.0, 2.0];
var output : [3]f32 = [0.0] ** 3;
forward_batch(&input, DEFAULT_ALPHA, &output);
return output[index];
}
// ═══════════════════════════════════════════════════════════
// TDD: Tests (from .tri behaviors)
// ═══════════════════════════════════════════════════════════
// ELU(-1) = exp(-1) - 1 = -0.6321206; ELU(2) = 2
test "forward_basic_case"
given input = negative_input()
when result = forward(input, DEFAULT_ALPHA)
then @abs(result - (-0.6321206)) < 0.000001 and forward(valid_input(), DEFAULT_ALPHA) == 2.0
// alpha scales only the negative side, and ELU(x) never goes below -alpha (in f32
// ELU(-20) rounds to exactly -alpha)
test "forward_alpha_scales_negative_side"
given input = negative_input()
when result = forward(input, 0.5)
then @abs(result - 0.5 * forward(input, 1.0)) < 0.000001 and forward(-20.0, 0.5) >= -0.5
test "forward_batch_basic_case"
given input = negative_input()
when result = batch_at(0)
then result == forward(input, DEFAULT_ALPHA) and batch_at(1) == 0.0 and batch_at(2) == 2.0
// ELU'(-1) = exp(-1) = 0.3678794; ELU'(2) = 1
test "derivative_basic_case"
given input = negative_input()
when result = derivative(input, DEFAULT_ALPHA)
then @abs(result - 0.3678794) < 0.000001 and derivative(valid_input(), DEFAULT_ALPHA) == 1.0
test "derivative_matches_finite_difference"
given slope = (forward(-1.0 + 0.001, DEFAULT_ALPHA) - forward(-1.0 - 0.001, DEFAULT_ALPHA)) / 0.002
when result = derivative(-1.0, DEFAULT_ALPHA)
then @abs(result - slope) < 0.001
// ═══════════════════════════════════════════════════════════
// TDD: Invariants (from .tri constraints)
// ═══════════════════════════════════════════════════════════
invariant elu_constraint_0
assert DEFAULT_ALPHA > 0.0
// name: elu_formula -- identity on the positive side
invariant elu_constraint_1
assert forward(valid_input(), DEFAULT_ALPHA) == valid_input()
// name: derivative -- 1 on the positive side
invariant elu_constraint_2
assert derivative(valid_input(), DEFAULT_ALPHA) == 1.0
// "Clevert et al. (2016) - Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs)":
// ELU is continuous at 0
invariant elu_constraint_3
assert forward(0.0, DEFAULT_ALPHA) == 0.0
All lessons
Module 1 · One neuron
A neuron sums its inputs and bends the sum: ReLU, the sigmoid and tanh, and where each one stops passing a gradient back.
Module 2 · New bends
The smooth curves of current models: GELU with its dip, Swish with its knob, and ELU with a floor instead of a cliff.
Module 3 · A layer
What sits around the neurons: softmax turns scores into a choice, an embedding turns a word into numbers, dropout switches units off.
Module 4 · Pictures
Reading a picture in patches: a window averaged over its patch, max pooling, and batch normalisation of features on different scales.
Module 5 · The error
How wrong a guess is, as one number: mean squared error, cross-entropy for probabilities, and the Huber loss for outliers.
Module 6 · Downhill
Walking down the loss: gradient descent with momentum, RMSProp that ignores the gradient's size, and Adam that combines them.
Module 7 · Learning
Where the gradients come from and what they achieve: backpropagation through a layer, a network that learns XOR, and LAMB for large batches.
Module 8 · Memory and attention
Networks that read a sequence: a recurrent cell and its fading echo, the gates of an LSTM, and attention over every position at once.
Module 9 · The transformer and the reward
The parts of a transformer and of an agent: positions as clocks, one encoder block, and PPO, which learns from reward without lurching.