Dropout
You will learn
Why switching units off while training helps, and why the survivors must be scaled.
While training, dropout switches off a share p of the units at random, so the rest cannot lean on any one of them. The units that survive must then be scaled, or the next layer would see the signal at another size in training than at inference. Inverted dropout scales them by 1 / (1 - p) while training: with p = 0.5, input 1, 2, 3, 4 becomes 2, 0, 6, 0, and inference is a plain pass. Classic dropout keeps 1, 0, 3, 0 and scales by 0.5 at inference instead. The spec dropout_layer.t27 uses a fixed mask, keep, drop, keep, drop, so the numbers are the same every time.
Try it
Compare the outputs of the four modes; then read the gradient of a dropped unit in training.
Switch between training and inference, inverted and classic dropout: units 1 and 3 are dropped, the survivors are scaled so that the layer's output keeps its expected size.
specs/ml/layers/dropout_layer.t27
// SPDX-License-Identifier: Apache-2.0
// t27/specs/
// Prevent co-adaptation of neurons | φ² + 1/φ² = 3 | TRINITY
// #8980: the tests called `allocf32` and `allocbool`, which no spec declares, and
// asserted `result != undefined`. Inference also scaled by (1 - p) even for
// inverted dropout, which already scaled by 1 / (1 - p) in training, so the
// expected activation shrank by (1 - p) between training and inference. Inference
// is now the identity when scale_during_training is set.
module Dropout;
use base::types;
use math::constants;
// ═══════════════════════════════════════════════════════════
// 1. Constants
// ═══════════════════════════════════════════════════════════
const DEFAULT_DROP_RATE : f32 = 0.5;
// ═══════════════════════════════════════════════════════════
// 2. Types
// ═══════════════════════════════════════════════════════════
pub const DropoutConfig = struct {
p : f32,
inplace : bool,
scale_during_training : bool,
};
// ═══════════════════════════════════════════════════════════
// 3. Core Functions
// ═══════════════════════════════════════════════════════════
// forward(input: []const f32, output: []f32, mask: []bool, training: bool, config: DropoutConfig) → void
fn forward(input: []const f32, output: []f32, mask: []bool, training: bool, config: DropoutConfig) -> void {
const n = input.len;
var i : usize = 0;
while (i < n) {
if (training) {
if (mask[i]) {
// Neuron is kept, scale during training if configured
if (config.scale_during_training) {
output[i] = input[i] * (1.0 / (1.0 - config.p));
} else {
output[i] = input[i];
}
} else {
// Neuron is dropped
output[i] = 0.0;
}
} else {
if (config.scale_during_training) {
// Inverted dropout already scaled in training: inference is the identity
output[i] = input[i];
} else {
// Classic dropout: scale by the keep probability at inference
output[i] = input[i] * (1.0 - config.p);
}
}
i = i + 1;
}
}
// backward(grad_output: []const f32, mask: []const bool, grad_input: []f32, config: DropoutConfig) → void
fn backward(grad_output: []const f32, mask: []const bool, grad_input: []f32, config: DropoutConfig) -> void {
const n = grad_output.len;
var i : usize = 0;
while (i < n) {
if (mask[i]) {
// Gradient flows through kept neurons
if (config.scale_during_training) {
grad_input[i] = grad_output[i] * (1.0 / (1.0 - config.p));
} else {
grad_input[i] = grad_output[i];
}
} else {
// Gradient is zero for dropped neurons
grad_input[i] = 0.0;
}
i = i + 1;
}
}
// ═══════════════════════════════════════════════════════════
// TDD: Tests (from .tri behaviors)
// ═══════════════════════════════════════════════════════════
fn cfg(scaled: bool) -> DropoutConfig {
return DropoutConfig{ .p = DEFAULT_DROP_RATE, .inplace = false, .scale_during_training = scaled };
}
// forward on input (1, 2, 3, 4) with mask (keep, drop, keep, drop)
fn forward_at(training: bool, scaled: bool, index: usize) -> f32 {
const input : [4]f32 = [1.0, 2.0, 3.0, 4.0];
var output : [4]f32 = [9.0] ** 4;
var mask : [4]bool = [true, false, true, false];
forward(&input, &output, &mask, training, cfg(scaled));
return output[index];
}
fn backward_at(scaled: bool, index: usize) -> f32 {
const grad_output : [4]f32 = [1.0, 1.0, 1.0, 1.0];
const mask : [4]bool = [true, false, true, false];
var grad_input : [4]f32 = [9.0] ** 4;
backward(&grad_output, &mask, &grad_input, cfg(scaled));
return grad_input[index];
}
// ═══════════════════════════════════════════════════════════
// TDD: Tests (from .tri behaviors)
// ═══════════════════════════════════════════════════════════
// p = 0.5, inverted: kept units are doubled, dropped ones are 0
test forward_basic_case
given kept = forward_at(true, true, 2)
when dropped = forward_at(true, true, 1)
then kept == 6.0 and dropped == 0.0 and forward_at(true, true, 0) == 2.0
// The expected activation is the same in training and inference
test forward_inference_keeps_the_expectation
given inverted = forward_at(false, true, 2)
when classic = forward_at(false, false, 2)
then inverted == 3.0 and classic == 1.5 and forward_at(true, false, 2) == 3.0
// The gradient flows only through kept units, with the same scale
test backward_basic_case
given kept = backward_at(true, 0)
when dropped = backward_at(true, 3)
then kept == 2.0 and dropped == 0.0 and backward_at(false, 2) == 1.0
invariant dropout_rate_is_a_probability
assert DEFAULT_DROP_RATE >= 0.0 and DEFAULT_DROP_RATE < 1.0
All lessons
Module 1 · One neuron
A neuron sums its inputs and bends the sum: ReLU, the sigmoid and tanh, and where each one stops passing a gradient back.
Module 2 · New bends
The smooth curves of current models: GELU with its dip, Swish with its knob, and ELU with a floor instead of a cliff.
Module 3 · A layer
What sits around the neurons: softmax turns scores into a choice, an embedding turns a word into numbers, dropout switches units off.
Module 4 · Pictures
Reading a picture in patches: a window averaged over its patch, max pooling, and batch normalisation of features on different scales.
Module 5 · The error
How wrong a guess is, as one number: mean squared error, cross-entropy for probabilities, and the Huber loss for outliers.
Module 6 · Downhill
Walking down the loss: gradient descent with momentum, RMSProp that ignores the gradient's size, and Adam that combines them.
Module 7 · Learning
Where the gradients come from and what they achieve: backpropagation through a layer, a network that learns XOR, and LAMB for large batches.
Module 8 · Memory and attention
Networks that read a sequence: a recurrent cell and its fading echo, the gates of an LSTM, and attention over every position at once.
Module 9 · The transformer and the reward
The parts of a transformer and of an agent: positions as clocks, one encoder block, and PPO, which learns from reward without lurching.