SPSA
Technique
Training a 1 billion parameter LSTM for next-token prediction without backpropagation
"The learning procedure, the update rule that worked the best, is something called SPSA that pretty much no one knows about except James Spall and his students that invented this learning update rule... What's cool about this, the other property is there's n..."