Olmo-core 3 is Ai2's open code for training one giant model on 512 GPUs
On October 1st, Ai2 released Olmo-core 3, the open-source code it trains its Olmo language models with, rebuilt for very large mixture-of-experts models. To explain it, Ai2 made this interactive walkthrough. It starts with a question: how do 512 GPUs train one model? Here they are, and we'll follow one of them, GPU 1.
Each token uses only 2 of a layer's 16 experts, so the experts can live on different GPUs
Look inside one layer of a mixture-of-experts model. Most of its weights sit in small networks called experts, 16 in this drawing. For each token, a router picks just two of them, here experts 3 and 12, and only those two do any work. So the experts can be split up. GPU 1 keeps experts 1 to 8, GPU 2 keeps 9 to 16, and each one needs half the memory. That's expert parallelism.
The catch: a token whose expert lives on the other GPU has to travel there and back
The catch is traffic. When a token needs an expert on the other GPU, its numbers have to travel over and back, in every layer, once on the way up and again on the way down. Spread a big model over hundreds of GPUs, and that traffic can eat up most of what you saved by using only a few experts per token.
16 times the experts, and training slowed by less than 5%
Olmo-core 3 is built to keep that cost down. In one test, Ai2 picked 4 experts per token and kept the work per token about the same, 3.2 billion active parameters. Then it grew the pool from 8 experts, under 5 billion parameters in all, to 128 experts and 47 billion. Ten times the model, and training slowed by less than 5 percent.
Against Ai2's previous version, the same model trains 2.7 times faster
Against Ai2's own previous version, the gap is wide. In a first test on eight of Nvidia's B300 GPUs, a 47-billion-parameter model trained at about 19,000 tokens per second on each GPU with the old code, and 52,000 with the new: 2.7 times as fast. The old code kept gathering every expert's weights onto each GPU. The new one leaves each expert where it lives and sends the tokens to it.
Too deep for one GPU: stack the layers in stages and keep every stage busy
A model can also be too deep for one GPU. So it's split by height into stages: GPUs 1 and 2 keep layers 1 to 4, and two new GPUs above them take layers 5 to 8. Each stage hands its results up, and the other one waits. So the texts go in back to back, in small batches, and both stages stay busy. That's pipeline parallelism.
64 GPUs make one copy of the model; 8 copies make 512
Now scale it up. 32 layers in 8 stages. 64 experts in every layer, split 8 ways. That's 64 GPUs holding one copy of the model. Then copy that team 8 times, each copy reading different text, and you have 512 GPUs training one model.
On 512 GPUs, a 1.2-trillion-parameter model ran at 858 teraflops per GPU
That's the shape of Ai2's biggest test. On 512 B300 GPUs, a model with 1.2 trillion parameters, 58 billion of them active for each token, reached about 860 teraflops of useful work per GPU. These runs sent tokens to random experts, so they measure the machinery, not how good a model it trains. A shorter test reached 2.4 trillion parameters.
“Token gerrymandering”: the balance score kept improving while the load got more lopsided
Ai2's 168-page report also records what went wrong. One failure they named token gerrymandering. The score meant to keep the experts evenly loaded kept getting better, while the real imbalance climbed from under 3 to almost 7. The router had learned to game the score.
Olmo-core 3 is open source, and Ai2's next Olmo will train on it
Released 1 October 2026 (Ai2's post; GitHub release v3.0.0, Apache 2.0). Ai2's interactive walkthrough is at narrative.allen.ai/scaling-up-training.
Olmo-core 3 came out on October 1st, 2026, open source under the Apache license. Ai2 says its next Olmo will be a mixture-of-experts model trained on it. The code, the interactive walkthrough, and the full report are linked below.
