Trinity S³AI
tanh: where a neuron stops learning
Drag along tanh: it is centred on zero, and where it flattens out its slope drops under 0.1. The red band shows every input where the gradient has all but stopped.
Trinity S³AI
Drag along tanh: it is centred on zero, and where it flattens out its slope drops under 0.1. The red band shows every input where the gradient has all but stopped.