EmbodiedCLBench: one practice task gives AI agents most of their gain, and more practice doesn’t add to it

3 hours ago

@ai-papersSubscribe

EmbodiedCLBench tests whether agents in Claude Code, Codex, OpenClaw and Hermes get better at unseen household chores after practicing simpler ones. Practice lifts advanced-task success from about 10% to about 30%; the first practice task gives most of that gain, and more practice adds no consistent improvement. From Peking University, BIGAI and the University of Hong Kong (arXiv 2610.13021v1, submitted 9 Oct 2026). Paper: https://arxiv.org/abs/2610.13021 Code: https://github.com/PKU-VaLuE-Lab/EmbodiedCLBench Data: https://huggingface.co/datasets/PKU-VaLuE-Lab/EmbodiedCLBench

Ask

Ask about this presentation

Answers are generated from this presentation.

Chapters

  1. 0:00One practice task, then a plateau
  2. 0:19Basic chores, then unseen advanced ones
  3. 0:35Each step is a four-way choice
  4. 0:48Without practice
  5. 1:04Two practice chores, every harness
  6. 1:20Full log or a skill note
  7. 1:32Fewer impossible moves
  8. 1:45The gain holds on longer chores
  9. 2:00More practice, no further gain
  10. 2:13Where the result stops
  11. 2:30Code and data
Show transcript

One practice task, then a plateau

AI Papers

EmbodiedCLBench: one practice task gives AI agents most of their gain, and more practice doesn’t add to it

Jie Huang, Yanan Chen, Ruixun Liu, Yiwu Zhong and colleagues · Peking University, BIGAI Beijing, University of Hong Kong

Huang et al. · EmbodiedCLBench · arXiv 2610.13021v1 · submitted 9 Oct 2026 · Figure 5

AI agents running in Claude Code, Codex, OpenClaw and Hermes get much better at unseen chores after practicing simpler ones. The first practice chore gives most of that gain, and more adds no consistent improvement. The benchmark is EmbodiedCLBench, from Peking University.

Basic chores, then unseen advanced ones

010203The test

168 basic chores to practice on, then 100 advanced chores the agent has never seen

Basic tasks such as pick up object and open drawer combine into advanced tasks such as clean up the room
Huang et al. · EmbodiedCLBench · arXiv 2610.13021v1 · §3.2, §4.1 · Figure 1

The agent works in a simulated home. It practices basic chores, like opening a drawer. Then it gets an unseen advanced chore that chains four such actions, like tidying a room before bed.

Each step is a four-way choice

010203The test

Each step shows a picture of the room and four possible moves

Evaluation: the agent sees the state and picks among options that reach the goal, detour, or are distractors
Huang et al. · EmbodiedCLBench · arXiv 2610.13021v1 · §3.3, §3.4, §4.1 · Figure 2

At every step the agent sees a picture of the room and picks one of four moves. One heads for the goal, and some are impossible from where it stands. Advanced chores take twelve steps.

Without practice

010203What it found

Without practice, agents finish about half the basic chores and fewer than one in ten advanced ones

Basic chores
51.5%
Advanced chores
7.9%

Mean of 8 harness and model pairs · basic 42.9 to 58.9% · advanced 3.0 to 12.0%

Huang et al. · EmbodiedCLBench · arXiv 2610.13021v1 · Table 1, zero-shot columns

With no practice, the agents finish about half of the basic chores. That holds across all four harnesses and both models, Qwen3.8-Flash and GPT5.6-Terra. On the advanced chores they finish fewer than one in ten.

Two practice chores, every harness

010203What it found

After two practice chores, every harness and model finishes more advanced chores

HarnessModelColdFull logSkill note
HermesQwen3.8-Flash8.044.033.0
HermesGPT5.6-Terra11.016.034.0
OpenClawQwen3.8-Flash5.035.038.0
OpenClawGPT5.6-Terra12.039.022.0
CodexQwen3.8-Flash4.034.033.0
CodexGPT5.6-Terra11.052.026.0
Claude CodeQwen3.8-Flash3.027.031.0
Claude CodeGPT5.6-Terra9.031.025.0
Huang et al. · EmbodiedCLBench · arXiv 2610.13021v1 · Table 1 · advanced-task success rate, %

Now give the agent two related basic chores before the advanced one. Advanced success climbs from about one in ten to around one in three, in every harness. Codex with GPT5.6-Terra goes furthest, to about half.

Full log or a skill note

010203What it found

A short Markdown skill note works about as well as the full log of the practice runs

Full log in contextevery step of both practice choresahead on 5 of 8 pairs
Skill notea Markdown summary the agent writes after practiceahead on 3 of 8 pairs
Huang et al. · EmbodiedCLBench · arXiv 2610.13021v1 · §4.1, §4.2 · Table 1

One setup keeps the full log of every practice step in context. The other keeps only a short Markdown skill note the agent wrote for itself. Neither has a consistent advantage.

Fewer impossible moves

010203What it found

Practice mostly stops agents from choosing impossible moves

Change in selection rate versus zero shot: optimal up 9 to 12 points, suboptimal up 3 to 8, invalid down 13 to 20, across Hermes, OpenClaw, Codex and Claude Code

Change versus no practice, Qwen3.8-Flash · red: impossible moves · dark: moves on the shortest path

Huang et al. · EmbodiedCLBench · arXiv 2610.13021v1 · §5.3 · Figure 6

With practice, impossible moves drop by 13 to 20 points in every harness, and moves straight toward the goal rise by about 10. The skill note shifts these choices less than the full log.

The gain holds on longer chores

010203What it found

The gain from practice holds as chores get longer

4%no practice
32%full log
26%skill note

Advanced chores of six chained actions · Hermes with Qwen3.8-Flash

Huang et al. · EmbodiedCLBench · arXiv 2610.13021v1 · §5.1 · Figure 4

Longer chores keep the benefit. At six chained actions, an agent without practice finishes 4 percent. With practice it finishes about a quarter to a third. The full log keeps a gain of roughly 30 points.

More practice, no further gain

010203What it found

The first practice chore does the work

From 1 up to 6 practice chores, the curves wobble and none keeps rising

Hermes and OpenClaw with Qwen3.8-Flash

Success rate versus number of learning samples, 0 to 6, for Hermes and OpenClaw, full log and skill note: a jump from about 10% to 28 to 40% at one sample, then fluctuation
Huang et al. · EmbodiedCLBench · arXiv 2610.13021v1 · §5.2 · Figure 5

A single practice chore lifts success from about one in ten to between 28 and 40 percent. Past that first chore, up to six, the curves wobble and none keeps climbing.

Where the result stops

010203The limits

A narrower setup than a real robot in a real home

A recorded map of the simulatorthe closed-source TongSim build, replayed offline as a graph
Multiple choicefour options per step, written by Qwen3.7-Plus and checked by people
One number per settingTable 1 shows no spread or repeated runs
Plateau on two harnessesHermes and OpenClaw, one model
Huang et al. · EmbodiedCLBench · arXiv 2610.13021v1 · §3.3, §3.4, Appendix B · Table 1 · §5.2

The chores replay a recorded map of a closed-source simulator called TongSim, and every step is a four-way choice. Each result is a single success rate with no spread, and the plateau was measured on two harnesses with one model.

Code and data

010203The limits

The EmbodiedCLBench code and task data are public

Codegithub.com/PKU-VaLuE-Lab/EmbodiedCLBench
Datahuggingface.co/datasets/PKU-VaLuE-Lab/EmbodiedCLBench
Paperarxiv.org/abs/2610.13021

Runs against model APIs with no GPU

Huang et al. · EmbodiedCLBench · arXiv 2610.13021v1 · repository README

The code and task data are public, and it runs against model APIs with no GPU. For agents that learn from their own runs, the open problem is the second practice chore, which adds no consistent gain.

Up next

3:11

Tool-using agents often act before checking the evidence the action needs

@ai-papers · 4 days ago

3:19

Given enough tries, base models solve more agent tasks than their post-trained versions

@ai-papers · 1 week ago

4:57

Coding agents solve fewer SWE-bench issues once a repository's names are changed

@ai-papers · 1 week ago