FP8 RL training switches off the penalty on garbled answers

Train an AI model with eight-bit numbers, and a hundred rounds in, it starts writing garbage. That format, FP8, stores every number in eight bits instead of sixteen, so the math runs faster and each number gets rounded more coarsely. The training is reinforcement learning. The model tries an answer, gets a grade, and gets nudged toward what scored well and away from what scored badly. The model is Qwen three from Alibaba, practising on math. Qwen three has eight billion parameters, the numbers a model adjusts as it learns. The standard sixteen-bit run, called BF16, stays steady. This paper traces the blow-up to rounding noise that switches off the penalty for garbled answers.

Slide text

8-bit AI training runs fine for 100 steps, then writes garbage

Qwen3 8B on math · full-precision BF16 stays flat while three FP8 runs surge

the blow-up

Tensorwise · Rowwise · Blockwise = three ways of setting the rounding, coarse to fine

FP8-Rollout-BF16-Train = 8 bits only for writing answers

Eval Accuracy = test score · Training Step = one round of grading and nudging

Clip Fraction = share of words the safety check stops pushing

BF16 · full precision, the flat blue line

Entropy = how scattered the word choices are

Figure 1 · Qwen3-8B-Base learning math (GRPO, DeepScaleR) · the first 300 of 500 training steps

Chen, Jiang, Wei, Zhong, Li, Zhang, Lin, Venkataraman · UW-Madison, ByteDance Seed
“Towards Full Pipeline FP8 Reinforcement Learning for LLMs” · arXiv 2609.22870 v1 · Sep 2026 · preprint

In this kind of training, one copy of the model writes the answers and a second copy learns from the grades. Earlier work put only the writing copy in eight bits, and a standard correction handled the small disagreement between the two copies. That setup trains as steadily as full precision. This paper keeps the correction on in every run and puts the learning copy in eight bits too. The blow-up comes anyway.

The recipe here, GRPO, grades each answer against other answers to the same question. Its safety check comes from an older method called PPO. For every word in an answer, PPO compares two probabilities. The first is how likely the updated model makes that word. The second is how likely the same model made it before this round of updates. Dividing the first by the second gives the ratio. When an answer is graded worse than average, training pushes each of its words down. Once a word's ratio falls to zero point eight, a drop of twenty percent, PPO stops pushing it. Good answers get a matching limit on the way up.

In full eight-bit training, both probabilities in the ratio come out of rounded math, each a little off. Dividing one rounded number by another stacks the two errors together. Take a word from a badly graded answer, with a true ratio of, say, zero point eight two. That word is still inside the limit, so training should keep pushing it down. Rounding noise can make the ratio read zero point seven nine instead. PPO reads that as a word that has already dropped enough, and stops pushing. The word escapes its penalty. The same slip repeats across many garbled answers, so the model never unlearns them. The garbled answers pile up until the run falls apart. The paper calls this over-clipping.

The three FP8 runs round differently: one scale for a whole table of numbers, one for each row, or one for each small block. Smaller blocks round more precisely. In the ratio, the rounding error grows by one point six five times with the finest rounding, and by two point nine times with the coarsest. On the row-by-row run, late in training, about seventy percent of the wrongly stopped words sit at the lower limit, the punishment side. Most of those words, often eighty to ninety percent, come from the garbled answers. Only six point six four percent of the garbled answers were graded better than average, so nearly all of them were due a penalty.

To test the lower limit as the cause, the authors dropped it from zero point eight to zero point six on the row-by-row run. The blow-up never came. The training reward, the grade the model earns while it learns, stayed below full precision. The authors read the lower reward as punishing too hard, so the limit has to land in the right place.

Slide text

Moving the limit to 0.6 stops the blow-up and lowers the training reward

The model now gets punished too hard, so the limit has to move by the right amount

no blow-up

clip_low=0.6 · lower limit moved from 0.8 to 0.6

Reward = the grade it earns while learning · step 300: about 0.55 against 0.65

Entropy stays flat, like full precision

Response Length = how long its answers run · shorter

Figure 5 · Qwen3-8B-Base, row-by-row FP8 with the lower limit at 0.6, against BF16 · reward is measured during training

Chen, Jiang, Wei, Zhong, Li, Zhang, Lin, Venkataraman · UW-Madison, ByteDance Seed
“Towards Full Pipeline FP8 Reinforcement Learning for LLMs” · arXiv 2609.22870 v1 · Sep 2026 · preprint

The paper's fix is called Calibrated Clipping. Every twenty rounds of training, the method runs the batch through a sixteen-bit copy of the model. The method counts the share of badly graded words that full precision would stop pushing at zero point eight. Then it sets the eight-bit limit to stop the same share. Noise pushes extra words past the old limit, so the new limit sits lower. With the finest rounding at eight billion parameters, the lower limit lands between about zero point five four and zero point six six. The method also sets the upper limit to give good and bad answers the same balance of pushes as full precision.

On Qwen three at eight billion parameters, averaged over eight math tests, full precision scores fifty-seven point six. Plain eight-bit scores forty-six point one, forty-seven, and fifty-four point one, from the coarsest rounding to the finest. With Calibrated Clipping, the same three settings score fifty-five point nine, fifty-six point five, and fifty-eight point six. Each score is a single number, and the paper reports no repeats or error bars. Changing only how often the fix rechecks moves one score by almost two points, more than the finest rounding's one-point lead.

A second recipe, DAPO, trains Qwen three at fourteen billion parameters. On AIME, a hard American high-school math competition, full precision solves fifty point nine percent of the problems. Plain eight-bit solves thirty-five point seven to forty-one point six percent. With the fix, eight-bit solves forty-six point five to forty-seven point nine percent. The fix wins back most of the drop and still stops three to four and a half points short.

The paper's speed claim, up to one and a half times faster training, comes from a separate test of plain eight-bit training. The figure's caption says that test leaves out the full-precision checks. The introduction calls the fix's extra cost negligible, and the paper reports no full training time with the fix switched on. The finest rounding, the one that came out level with full precision, is also the slowest, at one point one to one point two times as fast.

The fix still needs a sixteen-bit copy of the model for its checks. On the two coarser roundings at eight billion parameters, the lower limit slides to zero point five, the bottom of its search range, and stays there. The upper limit climbs to two, the top of its range. From then on, the edges of the range set both limits, and the match to full precision stops steering them. Every model in the paper is a Qwen, from eight to thirty-two billion parameters, tested on math and one coding set.

Read it if you train models with low-precision numbers. Section three gives two measurements to repeat on your own training: how often the check stops pushing, and how far the ratio drifts from full precision. Hold off on implementing it, because the paper links no code. The paper is by Fanchao Chen and seven co-authors, a preprint on arXiv, the open archive for research papers, posted before peer review.