Ornith-1.5
Every practice book runs out. Finish the problems, and the book stops teaching you. Ornith-1.5 is an AI model from a lab called DeepReinforce. The model learned on problems it wrote for itself, and the training paid most for the problems the model usually got wrong.
Three open models, and the lab published how it trained them
Anyone can download Ornith-1.5. It comes in three sizes, and the licence lets anyone use all three, including in a business. The smallest is one file under six gigabytes, small enough to run on a laptop. Those files went up four days ago, and people have downloaded them about half a million times since. DeepReinforce wrote up the training itself, and that write-up answers a question anyone who has practised anything runs into. How do you keep teaching a student who has already done every problem in the book?
One score teaches every step at once
scaffoldharnessTraining a model means scoring its attempts. Every try gets a number, and the model shifts toward whatever scored higher. Normally people write the problems, and people build the thing that marks them. Here the model does both jobs. Every cycle, the model writes a problem it has not solved before. It builds the toolkit for attacking that problem: the instructions, the tools, and how the job gets broken up. It builds the checker, a little program that runs the answer and decides whether the answer actually worked. The checker gets scored too, on how hard it is to cheat. Then the model tries. The score from that attempt goes back through all of it, so the model learns to write better problems and better checkers too.
A problem the model reliably solves stops paying
A problem the model invents is scored on three things at once. Is it a real problem, with a checker that runs, passes good answers and fails bad ones. Is it at the right difficulty. Is it different from the problems the model has already written. All three have to hold, because any one of them at zero makes the score zero. The reward peaks when the model solves its own problem about one time in five. Make the problem easier, and the reward falls away. Make it harder, and every attempt fails, so there is nothing to learn from. So as the model gets better at a problem, that problem pays less, and the model has to write harder problems to keep scoring. Its own attempts decide what counts as hard.
Every one of the seventeen scores went up against Ornith’s own previous version
Every number here is DeepReinforce's own, and the lab ran the comparisons itself. It also says how: five runs each, averaged, the tools named with their versions, the repository's history stripped out on the repair tests so the model cannot look up the answer, and the network switched off. On those runs, at the same size, Ornith-1.5 scored higher than Ornith-1.0, the version before it, on all seventeen tests.
The training loop predicts the shape of Ornith-1.5’s five losses
Against a model eleven times its size, Ornith-1.5 comes up short on five of the seventeen tests. Four of the five are exam questions and web search, which turn on what the model already knows or can go find. The fifth is using outside tools, which a checker can mark, and on that test Ornith-1.5 finishes two points behind the bigger model. Ornith-1.5 comes out ahead on the other twelve tests: work in a terminal, fixing real code repositories, writing a repository from a description. That is what the training loop pays for. So how much of this came from the model writing its own problems? DeepReinforce changed more than one thing between the two versions, and the release contains no experiment that isolates the self-written problems. The experiment that would settle it trains the same model on problems people wrote.
The practice book never runs out, because the model writes the next problem
The training only pays the model for a problem it will probably get wrong. So the problem it writes next is always a little harder than the one before.
