NCP-ArchPreview hits OLMo-3-7B loss at 51.3% of the tokens

Language models guess the next word. NCP-ArchPreview also predicts the next idea. Next-token prediction is guessing the next word, one word at a time. OLMo-3-7B is Ai2's open 7 billion-parameter language model, trained on a public mix called Dolma-3. This chart is pretrain loss on the same 5.73 trillion Dolma-3 tokens. The 8.9 billion-parameter NCP-ArchPreview run hits OLMo-3-7B's final loss after 51.3 percent of that token budget. How the idea is built comes next, then the table.

NCP-ArchPreview is 8.94 billion parameters, trained from random weights on Dolma-3. Training is joint next-token prediction and next-concept prediction. A concept here is a discrete label for a short span of words, predicted as the next idea. At generation time the model still writes words. The concept forecast stays inside. These numbers are the authors' own eval.

A forward pass still starts as words. The model has three stacks. The front half reads the words. The middle forecasts the next idea. The back half still writes the next word. The paper names the front half the Token Encoder, 16 layers. Each layer writes a hidden state, an internal vector for every word. Every four of those vectors are averaged into one shorter sequence. That is one idea-slot for four words. The pooling is a straight average of those four vectors. That averaging is the pooling step.

Each pooled vector is split into 32 short pieces, 128 numbers each. Each piece picks a codeword from a book of 128 codewords. That split is product quantization. Combinations of 32 books of 128 entries each give the idea its vocabulary. Thirty-two times 128 matches the width of each word vector. The middle stack is the Concept Module, 8 layers. The Concept Module forecasts the next idea as a weighted mix of those codebook entries. Training can still send a learning signal through the mix. A training loss pulls the codewords onto the pooled states.

The predicted idea is repeated four times, once per word in the span. The sequence is shifted by four positions, matching the pool width. The Token Decoder never sees an idea built from future words. That write-back is causal inject. The idea is added to the token hidden state. The back half is the Token Decoder, 16 layers. The Token Decoder still scores the next word. Training is joint next-word loss, codebook loss, and next-idea loss. Generation still emits words one at a time. Extra skip routes start almost off and learn.

Stage-1 is the long pretrain on mixed web text. Both runs use the same Dolma-3 mix, 5.73 trillion tokens. NCP-ArchPreview reaches OLMo-3-7B's final Stage-1 loss at 51.3 percent of those tokens. That is 1.95 times in token budget. Over the last 120 thousand training steps the Stage-1 loss gap is 0.091. The red pencil in the margin marks the 51.3 percent intercept. Stage-2 is a shorter mid-train that adds more code and math. On Stage-2, NCP-ArchPreview matches OLMo's final loss at 66.2 percent of the tokens, 1.51 times. The 0.027 final Stage-2 gap is a loss number. Later recipe variants in the paper show lower loss with worse test scores.

Vanilla is the released OLMo-3 checkpoint at each stage. Stage-1 overall average is 49.04 for NCP-ArchPreview against 46.59 for Vanilla, a gain of 2.45 points. Grade-school math, the GSM8K test, is 45.26 against 39.27, a gain of 5.99 points. Stage-1 code average also rises. Stage-2 overall average rises 0.59 points. Stage-2 code average is 38.77 against 39.42, a drop of 0.65 points. The authors say the Stage-2 mix is about 10 percent code. These scores are the authors' own harness.

The 51.3 percent intercept is against the OLMo-3-7B run they trained, a 32-block model. NCP-ArchPreview is 8.94 billion parameters. The size-fair match is a 40-block Vanilla. Figure 3 covers the first 200 billion tokens. The full NCP-ArchPreview model approaches that 40-block loss at 85 percent of the counted compute. Concept blocks add parameters while running on a sequence about one quarter as long. They add about one quarter of the compute. The paper counts NCP-ArchPreview as 40 parameter-blocks and 34 compute-blocks, against Vanilla at 32 and 32. The Stage-2 code drop lives in Table 1. Max train length is 8,192 tokens. Long-context training is not in this preview. ConceptLM already showed next-concept prediction at smaller scale. The load-bearing claim here is from-scratch 8.9 billion parameters over 5.73 trillion tokens. A scaling fit prints 1.74 times versus OLMo-3. That 1.74 is a fitted curve. Every score here is the authors' eval.

Read it. The hour is Figure 1a and Table 1 Stage-1, if you train large models or care about objectives above the next word. Implement it only if you already train at OLMo-3 scale and will take the 8.94 billion-parameter, 40-block recipe. Checkpoints are released. That is for the reader. The paper is NCP-ArchPreview Technical Report, from the Intern-NCP Team. The arXiv id is 2609.10715. The labs are Shanghai AI Lab and LUMIA Lab at Shanghai Jiao Tong University. The preprint was submitted on 9 September 2026. On the same Dolma-3 data, a second target reaches OLMo-3-7B's loss at 51.3 percent of the tokens. Against a size-matched 40-block Vanilla, the counted compute is 85 percent.