Trinity S³AI
LAMB: one learning rate for layers of every size
Scale a layer's weights and watch its LAMB step: it grows with the weight norm, so every layer moves by the same fraction of itself, until the norm passes 10 and the step stops growing.
Trinity S³AI
Scale a layer's weights and watch its LAMB step: it grows with the weight norm, so every layer moves by the same fraction of itself, until the norm passes 10 and the step stops growing.