Trinity S³AI
Swish: one knob between a smooth curve and ReLU
Drag along x times sigmoid(beta x). At beta 1 it is SiLU, with its dip of -0.28; turn beta up and the curve closes on ReLU, turn it down and it flattens toward x / 2.
Trinity S³AI
Drag along x times sigmoid(beta x). At beta 1 it is SiLU, with its dip of -0.28; turn beta up and the curve closes on ReLU, turn it down and it flattens toward x / 2.