t27.aiРусский

The sigmoid: a number into a probability

You will learn

How the sigmoid maps any input to a probability, and why its slope limits deep networks.

The sigmoid 1 / (1 + e^-x) squeezes the whole number line into 0 to 1, which is why it reads as a probability: 2 becomes 0.8808 and -4 becomes 0.018. Its slope is s (1 - s), computed from the output s itself, and it is largest at 0, where it is 0.25; at x = 8 it is 0.000335. A gradient that passes back through a stack of sigmoids is multiplied by at most 0.25 at each one, which is why deep sigmoid networks trained slowly. The spec sigmoid_activation.t27 computes the curve and the slope.

Try it

Find the input where the probability is 0.5; then drag to 6 and read how little slope is left.

Open the interactive lesson →

Sigmoid: turn any number into a probability
Sigmoid: turn any number into a probability ↗

Drag along the S-curve: every input lands between 0 and 1, and the slope peaks at 0.25 in the middle and dies away at both ends.

specs/ml/activation/sigmoid_activation.t27

// SPDX-License-Identifier: Apache-2.0
// t27/specs/
// Derivative (compute from output) | φ² + 1/φ² = 3 | TRINITY

module Sigmoid;
    use base::types;
    use math::constants;

    // ═══════════════════════════════════════════════════════════
    // 2. Types
    // ═══════════════════════════════════════════════════════════

    pub const SigmoidConfig = struct {
        none : void,
    };

    // ═══════════════════════════════════════════════════════════
    // 3. Core Functions
    // ═══════════════════════════════════════════════════════════

    // forward(x: f32) → f32
    fn forward(x: f32) -> f32 {
        // Numerically stable split to avoid overflow:
        // For x >= 0: 1 / (1 + exp(-x))
        // For x < 0: exp(x) / (1 + exp(x))
        // Both branches are mathematically equivalent but the second avoids exp(large positive)
        if (x >= 0.0) {
            return 1.0 / (1.0 + @exp(-x));
        } else {
            return @exp(x) / (1.0 + @exp(x));
        }
    }

    // forward_batch(input: []f32) → []f32
    fn forward_batch(input: []f32) -> []f32 {
        // In-place map over input: alias input and return it
        // This is safe because we're only reading from input and writing to the same locations
        let i = 0;
        while (i < input.len) {
            input[i] = forward(input[i]);
            i += 1;
        }
        return input;
    }

    // derivative(sigmoid_x: f32) → f32
    fn derivative(sigmoid_x: f32) -> f32 {
        // Derivative computed from output: σ'(x) = σ(x) * (1 - σ(x))
        return sigmoid_x * (1.0 - sigmoid_x);
    }

    // ═══════════════════════════════════════════════════════════
    // TDD: Tests (from .tri behaviors)
    // ═══════════════════════════════════════════════════════════

    // #8980: through a function, so the check runs and is counted (it folded at compile
    // time before, and the report counted it as a vacuous pass).
    fn config_size() -> usize {
        return @sizeOf(SigmoidConfig);
    }

    test sigmoid_has_no_tunable_parameter
        given size = config_size()
        then size == 0

    // sigmoid_config_is_a_placeholder_not_an_empty_struct read std.meta.fields, which t27
    // cannot lower; the parser discarded it. The test above already pins @sizeOf == 0.

    test forward_at_zero_is_one_half
        given y = forward(0.0)
        then @abs(y - 0.5) < 1e-6

    test forward_confined_to_unit_interval
        given pos_large = forward(10.0)
        given neg_large = forward(-10.0)
        then (pos_large > 0.0) and (pos_large < 1.0) and (neg_large > 0.0) and (neg_large < 1.0)

    test derivative_at_zero_is_maximum
        given dy = derivative(forward(0.0))
        then @abs(dy - 0.25) < 1e-6

    test forward_batch_preserves_length
        given input = []f32{0.0, 1.0, -1.0, 2.0, -2.0}
        given result = forward_batch(input)
        then result.len == input.len

    // sigmoid(-x) = 1 - sigmoid(x); sigmoid(2) = 0.8807971
    test forward_is_symmetric_about_one_half
        given up = forward(2.0)
        when down = forward(-2.0)
        then @abs(up + down - 1.0) < 0.000001 and @abs(up - 0.8807971) < 0.000001

    // The negative branch never computes exp(+x): no overflow at -100
    test forward_far_left_is_zero_not_nan
        given y = forward(-100.0)
        then y >= 0.0 and y < 0.000001

Open the lesson's spec in the player ↗

All lessons