NestOne cuts a Qwen3.5 decision model to 24 layers at nearly full accuracy

NestOne trains one Qwen3.5 decision model that can answer after 12, 16, 24 or all 32 layers. At 24 layers it saves about a quarter of the time per decision and stays within a point of the full model. Minh-Anh Nguyen built it at Nanyang Technological University.

Slide text

NestOne cuts a Qwen3.5 decision model to 24 layers at nearly full accuracy

Minh-Anh Nguyen · Nanyang Technological University

Nguyen · From LLMs to Efficient System One Models via Adaptive Computation · arXiv 2611.16321v1 · submitted 12 Nov 2026

Agents make many small choices: which tool to call, where to route a request, whether a prompt is unsafe. A System One model answers each in one pass, and every answer still runs all 32 layers.

Probe each layer for the right option, and accuracy climbs through the middle of the network, from layer 15 to layer 23. After layer 23 it goes flat. The last layers add no decision information a probe can find.

Slide text

01 02 03 04 Where the decision is

The answer forms in layers 15 to 23, then stops changing

Nguyen · From LLMs to Efficient System One Models via Adaptive Computation · arXiv 2611.16321v1 · Figure 2, §3.1 · seven decision benchmarks, macro average

Take a released decision model, Intern-Decision-4B, and delete layers 24 to 29 with no retraining. Accuracy doesn't move. Start the cut at layer 22 instead, and it loses almost 12 points.

Training a 12 or 16 layer model from scratch gives you a weaker version of the full one. Its own wins sit barely above the noise between training seeds, and it loses three to four decisions for every one it uniquely gets right.

So NestOne trains every depth together. The shorter models are prefixes of the full model. They share its last two layers and its decision head, and they learn from the full model's own answers as well as the labels.

Slide text

01 02 03 04 One checkpoint

NestOne trains the 12-, 16- and 24-layer models inside the full one

Nguyen · From LLMs to Efficient System One Models via Adaptive Computation · arXiv 2611.16321v1 · Figure 3a, §4.1–4.2

One training run, about 80 minutes on a single H200, replaces four. The deep exits tie the separate models. The shallow ones come out ahead, and clearly so at 12 layers, by about three points.

At each exit, a gate reads the hidden state and decides whether the answer is ready. Checking costs one dot product. Stopping at 24 layers takes about 35 milliseconds, against about 47 for the full model.

Slide text

01 02 03 04 Choosing depth

A small gate at each exit decides whether to answer there

Nguyen · From LLMs to Efficient System One Models via Adaptive Computation · arXiv 2611.16321v1 · Figure 3b, §4.3, §5 setup · latency at batch size one

How much you save depends on your traffic. On easy decisions, like news topics and jailbreak checks, the gate averages under 18 layers. Hard ones still run all 24, and accuracy on the four largest benchmarks holds.

Slide text

01 02 03 04 Choosing depth

Easy traffic saves the most layers, and accuracy on the big benchmarks holds

Nguyen · From LLMs to Efficient System One Models via Adaptive Computation · arXiv 2611.16321v1 · Figure 1, Table 4, §5.2 · gate over exits 9, 13 and 21 · easy: AGNews, WildJailbreak

Every test uses one model, Qwen3.5-4B, timed at batch size one. It scores about 84 percent, below Intern-Decision-4B at 90. And the gate's threshold shifts between data splits, so tune it on traffic like yours.

If I served a fine-tuned decision model, I'd try the 24 layer exit first: a quarter less time, under a point lost, and no gate to tune. The code is on GitHub as NestOne.