Let's Scale Step by Step — predicting the learning rate for a 155B MoE
37 minutes ago
Training a large model gives you one attempt at its learning rate. That is the dial that sets how big a step the model takes each time it corrects itself. Set it too high and the run falls apart. Set it too low and the run crawls. This team picked that dial for a training run of ten trillion tokens, the pieces of text a model reads, before the run started. They read it off a line fitted to much smaller runs.
Ask
Ask about this presentation
Answers are generated from this presentation.
