Puro-2B trains 2 billion parameters from scratch on consumer cards, at this point on the chart
Building a language model from scratch is treated as a six-figure job. The usual recipe uses data-center graphics cards. Puro-2B is a 2-billion-parameter model trained from random numbers on consumer RTX 5090 cards. From scratch means the weights start random and train on more than a trillion tokens of text, chunks a little shorter than a word. This chart plots a 15-task quality average against dollars of card time. Puro-2B sits at about seven thousand dollars, in the same quality neighborhood as Qwen2-1.5B.
Two runs share the first half and land at four and seven thousand dollars of rental equivalent
| Checkpoint | Phase 1 | Phase 2 | GPU-hours | Cost |
|---|---|---|---|---|
| $4.4K uniform | $1.84K · 439B tokens | $2.53K · 480B tokens | 14,262 | $4.37K |
| $6.9K canonical | $1.84K · 439B tokens | $5.05K · 960B tokens | 22,514 | $6.89K |
Two production runs share the same first half. Uniform phase two mixes the tokens. Canonical phase two doubles the tokens, orders the data by quality, and averages late snapshots, for 1.4 trillion scheduled tokens. The run costs are a rental equivalent: thirty-one cents per GPU-hour, hardware and electricity spread over five years, counted only for the hours the cards actually trained after the training files already existed. A GPU-hour is one graphics card running for one hour. Qwen2-1.5B is Alibaba's open 1.5-billion-parameter model from 2024, the bar they mean to clear. Qwen2.5-1.5B is the 2025 bar they do not clear.
Five bottlenecks, one recipe, and the bars do not multiply
Pretraining cost is five stacked bottlenecks. This recipe attacks a different one at each layer. The first pays less per unit of compute. The second does more compute per second. The third extracts more quality per token. The fourth spends the late tokens where they stick. The fifth picks a cheaper mix of data. The paper's own diagram is the map. It shows consumer cards, two-phase pretraining, a uniform path and a curriculum path, and the precision and optimizer throughout. Data mix sits as a box. Each bar on the next slides has its own baseline.
Pay less per unit of compute, then do more of it in 8-bit
about 2.7 times the compute per dollar
1.34 times quality-adjusted throughput
| Card | Rate |
|---|---|
| RTX 5090 | $0.31/h |
| H200 | $4.00/h |
The authors train on RTX 5090s: consumer gaming cards, 32 gigabytes of memory, no factory high-speed link between cards, instead of data-center H100 or H200 cards. They report about 2.7 times the compute per dollar, in 16-bit and in 8-bit. NVIDIA disables card-to-card shortcuts in software. The authors used a modified driver. The second bottleneck is blockwise FP8. The big matrix multiplies run in 8-bit from the first step, in small blocks, instead of the usual 16-bit. The leftover pieces of the model stay at higher precision. A size ladder shows extra validation loss of about 0.003. They keep 98 percent of 16-bit-equivalent compute. Throughput is 1.36 times at this size. Quality-adjusted throughput is 1.34 times.
The learning rate means what it says, and the late tokens are averaged
1.19 times on a 20-token-per-parameter ladder
curriculum ordering, then a six-snapshot average
The third bottleneck is MuonH, an optimizer that, every step, puts each large matrix back to the size it started at, so the scheduled learning rate is the rate the weights actually feel. Embeddings and the output head stay on a standard optimizer. On a small ladder of 20 tokens per parameter, not the production 700, that is 1.19 times the quality-equivalent compute versus the same optimizer without that projection, a small-ladder transfer. The fourth bottleneck is curriculum model averaging, Luo and colleagues' 2026 averaging recipe. In phase two, each data source is ordered from easier examples to harder ones. The last 29 billion tokens hold the learning rate still. The last six snapshots of the model are averaged. The same quality on the uniform curve would have cost 1.65 times the production bill. Adding the held learning rate and the six-snapshot average, the same quality would have cost 2.40 times the production bill, as a joint recipe.
Puro-2B beats Qwen2 and trails Qwen2.5 on the same 15 tests
| Model | Math and code | Reasoning | 15-task |
|---|---|---|---|
| Puro-2B canonical | 43.50 | 63.02 | 57.81 |
| Puro-2B $4.4K uniform | 38.61 | 61.70 | 55.54 |
| Qwen2-1.5B | 40.29 | 60.54 | 55.14 |
| Qwen2.5-1.5B | 47.52 | 65.53 | 60.73 |
Canonical Puro-2B clears Qwen2-1.5B on math and code, and on reasoning and knowledge, and trails Qwen2.5-1.5B on both panels. Llama-3.2-3B sits at one million four hundred ninety-five thousand dollars under this paper's protocol. SmolLM3-3B sits at seven hundred nineteen thousand dollars. Those comparator dollars are estimated H100 or A100 GPU-hours. Puro-2B is the measured 5090 GPU-hour point.
The Puro Cost Scaling Law is a dollar curve at this fixed 2 billion parameters
The authors resume from the same first-half checkpoint. They vary only how many uniform phase-two tokens they buy, plot the fifteen-test average against rental-equivalent dollars, and fit a curve at this fixed 2 billion parameters. That is the Puro Cost Scaling Law. The 4.4 thousand dollar uniform checkpoint already clears Qwen2-1.5B. The 6.9 thousand dollar canonical point sits above that uniform curve because of the joint curriculum recipe.
The 6.9 thousand dollars is a narrow accelerator ledger, and the curve stays at 2 billion parameters
The 6.9 thousand dollars starts after the training files exist. Thirty-one cents an hour is amortized hardware plus electricity over five years, because a public 5090 rental barely exists. There was no corpus-wide check that the fifteen tests were absent from the training data. Curriculum beating uniform could partly be overlap. The model is overtrained at about 700 tokens of data per parameter, useful as a small deployable base. The driver work is unsupported. Change the utilization, the GPU, or the token counts on Figure 1, and every comparator dollar moves. The authors name 32 gigabytes of on-card memory as the wall against growing past 2 billion parameters. This bundle has no independent reproduction of the 6.9 thousand dollar run.
Read it
Read it. The hour is the cost boundary, the stacked recipe, and the fixed-2B dollar curve. Implement it only if you already have a 5090 cluster and will take the driver risk.
Puro-2B: Poor Lab’s Qwen2-1.5B Trained on RTX 5090 within $5090
Tsinghua University · Pengcheng Laboratory
“Puro-2B: Poor Lab’s Qwen2-1.5B Trained on RTX 5090 within $5090”
arXiv:2608.27370 · submitted 27 August 2026 · preprint
Luo, Cui, Yin, Chen, Yang, Gao, Wang, Li, Zhang, Lyu, and Chen, at Tsinghua University and Pengcheng Laboratory. Puro-2B: Poor Lab's Qwen2-1.5B Trained on RTX 5090 within 5090 dollars. arXiv 2608.27370, submitted 27 August 2026. Preprint.
