Trinity S³AI
GELU: why transformers bend a little below zero
Drag along GELU: x weighed by the chance a normal draw is below it. It dips to about -0.17 before it rises, and the tanh shortcut stays within a few thousandths of the exact form.
Trinity S³AI
Drag along GELU: x weighed by the chance a normal draw is below it. It dips to about -0.17 before it rises, and the tanh shortcut stays within a few thousandths of the exact form.