RMSProp

You will learn

How RMSProp divides each step by a running root mean square, so the step no longer depends on the gradient's size.

Try it

Switch between the three gradients and check that the lines do not move; then raise the decay to 0.99.

specs/ml/optimizer/rmsprop.t27

// SPDX-License-Identifier: Apache-2.0
// t27/specs/
// RMSprop optimizer with moving average of squared gradients | φ² + 1/φ² = 3 | TRINITY
// Restored 2026-10-05 from trinity specs/algo/rmsprop.tri. The port kept only
// `grad`, squared grad[0], and threw it away; the test read an undefined
// default_input().

module Rmsprop;
    use base::types;
    use math::constants;

    // ═══════════════════════════════════════════════════════════
    // 1. Constants
    // ═══════════════════════════════════════════════════════════

    const DEFAULT_DECAY : f32 = 0.9;

    // ═══════════════════════════════════════════════════════════
    // 2. Types
    // ═══════════════════════════════════════════════════════════

    pub const RMSpropConfig = struct {
        learning_rate : f32,
        decay : f32,
        epsilon : f32,
    };

    // ═══════════════════════════════════════════════════════════
    // 3. Core Functions
    // ═══════════════════════════════════════════════════════════

    // step(grad, param, cache, config) -> void, in place:
    //   cache = decay * cache + (1 - decay) * grad^2
    //   param -= lr * grad / (sqrt(cache) + epsilon)
    // The .tri formula reads lr, decay and epsilon but its parameter list
    // omits them; they come from config here.
    fn step(grad: []const f32, param: []f32, cache: []f32, config: RMSpropConfig) -> void {
        var i : usize = 0;
        while (i < grad.len) {
            const g = grad[i];
            cache[i] = config.decay * cache[i] + (1.0 - config.decay) * g * g;
            param[i] = param[i] - config.learning_rate * g / (@sqrt(cache[i]) + config.epsilon);
            i = i + 1;
        }
    }

    // ═══════════════════════════════════════════════════════════
    // TDD: Tests
    // ═══════════════════════════════════════════════════════════

    // run `steps` steps of constant gradient g on one parameter from 1.0
    fn param_after(g: f32, steps: usize, decay: f32) -> f32 {
        const grad : [1]f32 = [g];
        var param : [1]f32 = [1.0];
        var cache : [1]f32 = [0.0];
        var k : usize = 0;
        while (k < steps) {
            step(&grad, &param, &cache, RMSpropConfig{ .learning_rate = 0.01, .decay = decay, .epsilon = 0.00000001 });
            k = k + 1;
        }
        return param[0];
    }

    fn cache_after(g: f32, steps: usize) -> f32 {
        const grad : [1]f32 = [g];
        var param : [1]f32 = [1.0];
        var cache : [1]f32 = [0.0];
        var k : usize = 0;
        while (k < steps) {
            step(&grad, &param, &cache, RMSpropConfig{ .learning_rate = 0.01, .decay = DEFAULT_DECAY, .epsilon = 0.00000001 });
            k = k + 1;
        }
        return cache[0];
    }

    // first step from cache 0: cache = 0.1 g^2, so the move is lr / sqrt(0.1)
    // = 0.0316228 whatever the size of g
    test first_step_is_scale_free
        given small = param_after(0.001, 1, DEFAULT_DECAY)
        when large = param_after(1000.0, 1, DEFAULT_DECAY)
        then @abs(small - (1.0 - 0.0316228)) < 0.00001 and @abs(large - small) < 0.00001

    // moving average of a constant: cache_n = (1 - 0.9^n) g^2
    test cache_is_moving_average
        given result = cache_after(2.0, 3)
        when expected = (1.0 - 0.729) * 4.0
        then @abs(result - expected) < 0.00001

    test moves_against_the_gradient
        given up = param_after(0.5, 2, DEFAULT_DECAY)
        when down = param_after(-0.5, 2, DEFAULT_DECAY)
        then up < 1.0 and down > 1.0 and @abs((1.0 - up) - (down - 1.0)) < 0.00001

    // ═══════════════════════════════════════════════════════════
    // TDD: Invariants
    // ═══════════════════════════════════════════════════════════

    invariant rmsprop_decay_in_unit_interval
        assert DEFAULT_DECAY > 0.0 and DEFAULT_DECAY < 1.0

Open the lesson's spec in the player ↗

All lessons

• Module 1 · One neuron

  A neuron sums its inputs and bends the sum: ReLU, the sigmoid and tanh, and where each one stops passing a gradient back.

  1. 01 ReLU: keep the positive part
  2. 02 The sigmoid: a number into a probability
  3. 03 tanh and saturation

• Module 2 · New bends

  The smooth curves of current models: GELU with its dip, Swish with its knob, and ELU with a floor instead of a cliff.

  1. 04 GELU: a dip below zero
  2. 05 Swish and SiLU
  3. 06 ELU: a floor, not a cliff

• Module 3 · A layer

  What sits around the neurons: softmax turns scores into a choice, an embedding turns a word into numbers, dropout switches units off.

  1. 07 Softmax and temperature
  2. 08 Embeddings
  3. 09 Dropout

• Module 4 · Pictures

  Reading a picture in patches: a window averaged over its patch, max pooling, and batch normalisation of features on different scales.

  1. 10 Patches of a picture
  2. 11 Max pooling
  3. 12 Batch normalisation

• Module 5 · The error

  How wrong a guess is, as one number: mean squared error, cross-entropy for probabilities, and the Huber loss for outliers.

  1. 13 Mean squared error
  2. 14 Cross-entropy
  3. 15 The Huber loss

• Module 6 · Downhill

  Walking down the loss: gradient descent with momentum, RMSProp that ignores the gradient's size, and Adam that combines them.

  1. 16 Gradient descent
  2. 17 RMSProp
  3. 18 Adam

• Module 7 · Learning

  Where the gradients come from and what they achieve: backpropagation through a layer, a network that learns XOR, and LAMB for large batches.

  1. 19 Backpropagation
  2. 20 Learning XOR
  3. 21 LAMB and large batches

• Module 8 · Memory and attention

  Networks that read a sequence: a recurrent cell and its fading echo, the gates of an LSTM, and attention over every position at once.

  1. 22 A recurrent cell
  2. 23 LSTM gates
  3. 24 Attention

• Module 9 · The transformer and the reward

  The parts of a transformer and of an agent: positions as clocks, one encoder block, and PPO, which learns from reward without lurching.

  1. 25 Positional encoding
  2. 26 An encoder block
  3. 27 PPO and the reward