On-policy distillation punishes the student for stopping

Every model ends an answer by producing one invisible character that means I am done. Nothing else stops it. This answer never produced one. It was already correct at token one thousand and ninety four, then repeated its own final answer line seven hundred and four times until the length ceiling cut it off. Tokens are the word pieces a model reads and writes. The cause is one character. The model writing and the model teaching it spelled that stop character two different ways, and the training loop pushed the writer's own spelling down every time it used one.

The one writing is the student, Qwen3 one point seven billion, base. Base means the raw model, trained on text off the internet. The one teaching is the teacher, Qwen3 four billion, the same family after being taught to hold a conversation. The base student declares one stop character. The post-trained teacher declares two, and it stops with the chat character the student does not use. The claim is that this mismatch alone produces the runaway length, and repairing it brings the length down.

The recipe is called on-policy distillation. The student writes its own answer. The teacher reads each word the student wrote and says how likely the teacher thought that word was. Words the teacher liked get more likely. Words it disliked get less likely. The whole paper turns on one property of that recipe. The teacher's opinion only ever reaches words the student actually produced. A word the student never writes is never scored, so it never gets more likely either.

The teacher's score on a word is a subtraction. How likely the teacher thought that word, minus how likely the student thought it. Positive pushes the word up. Negative pushes it down. The Qwen3 student ends its answers with the base stop character. The teacher puts almost none of its stopping weight there, because the teacher stops with the chat character instead. Subtract, and the score on the one action the student uses to stop comes out negative. So the training pushes stopping down. And the teacher's own stop character is one the student almost never writes, so it is never scored and never becomes more likely. The student is punished for stopping and never taught the replacement. Over two hundred steps the student's probability of writing its own stop character falls from about zero point eight to near zero.

A settings file would be the easy explanation. Gemma 3 rules it out. Gemma 3 base and Gemma 3 chat declare the same two stop characters, the same two token numbers. Their learned preferences still split. And after the repair, when both models stop reliably again, they still stop through different characters. The disagreement is a property of what each model learned, and the repair works by no longer asking them to agree.

The mechanism makes a prediction. The collapse should only happen when the student cannot already write the teacher's stop character. K2-Horizon publishes checkpoints, saved copies of the model from four points in its own training. Start the student from the mid-training copy or the fine-tuned copy, where it already writes the teacher's character often enough to be scored on it, and the plain recipe shows no collapse. Those runs settle on the teacher's own average length, about four thousand five hundred tokens, because that teacher is long-winded itself, so no collapse there means matching a long-winded teacher.

Four repairs, and one does nothing. A one point seven billion parameter student, a four billion parameter teacher, two hundred steps, and a seven thousand one hundred and sixty eight token budget for each answer. Repair one registers both stop characters as valid endings while the model generates, and leaves the scoring rule alone. That orange curve climbs to the budget line and sits there with every answer cut off, which is where the uncorrected run ends up too. Accepting a character neither makes the student write it nor removes the punishment on the one the student does write. The three repairs that touch the probabilities work. Move the teacher's stopping weight onto the character the student already uses, treat every stopping character as one action called stop, or delete the alternatives from the student's options. All three land near three thousand tokens against the teacher's two thousand three hundred and fifty, which the paper calls substantially closer. Across model families the paper defaults to the middle one because it needs no agreed character. The three tie on the numbers, so that default is a choice about assumptions.

The repair holds for the first two hundred steps of the longest run. That run starts from K2-Horizon's raw pretrained copy and goes four hundred steps in all. Then the answers grow again, the probability of stopping decays to near zero, and almost every answer hits the budget, with the repair applied throughout. The authors mark it a third phase and write that they leave its origin to future work. So the open question is what drives a second collapse once both models already agree on a stop character. The largest model trained is K2-Horizon seven B. And even after the repair, the students still write longer answers than their teachers, most of all Llama 3.2.

Read it. Section three is the mechanism and it is short, and the check it implies takes a minute on any distillation run you own. Look at where your teacher puts its stopping weight, and where your student puts its own, before you rewrite the scoring rule. One point seven billion parameters up to K2-Horizon seven B, single maths questions, one collapse still unexplained. Back to that runaway answer. The model had been taught that stopping was the wrong move, and it learned. The paper is When EOS Tokens Disagree, by Yuxiao Yang and eight colleagues at UNC Chapel Hill, Brigham Young University and Microsoft. It is an arXiv preprint, number 2609.20511.