Accelerate Bench: Claude Sonnet 5.5 leads 16 agents at making programs faster

2 days ago

Accelerate Bench gives 16 coding agents 32 working scientific programs, like GROMACS and OpenFOAM, and eight hours each to make them faster. Claude Sonnet 5.5 in Claude Code cuts about two-thirds of the running time on average and leads the field; most agents keep their programs correct, so speed decides the ranking. One run per agent, CPU tasks only.

Ask about this presentation

Chapters

  1. 0:00Accelerate Bench
  2. 0:15A whole program and eight hours
  3. 0:30Two Claude agents lead
  4. 0:44Opus lost three tasks to safety stops
  5. 0:59Speed separates them
  6. 1:13Agents rewrite the computation
  7. 1:30Same code, different settings
  8. 1:42One change depends on another
  9. 1:56A thread default
  10. 2:10Limits
  11. 2:25Where to start
Show transcript
  1. Claude Sonnet 5.5 in Claude Code makes real scientific programs faster than any of the 16 coding agents on Accelerate Bench. A team at Tencent built the benchmark with HKUST, Tsinghua and Peking University.

    Slide text
    AI Papers

    Accelerate Bench: Claude Sonnet 5.5 leads 16 agents at making programs faster

    Bowen Liu, Yuhan Li, Nuo Chen and colleagues · Tencent Hy Team, HKUST, Tsinghua University, Peking University

    Liu et al. · Accelerate Bench · arXiv 2611.20882v1 · submitted 16 Nov 2026 · Table 1
  2. Each of the 32 tasks gives an agent a working program, like GROMACS or OpenFOAM, and eight hours. It can change code, build flags and thread settings, and a hidden verifier times the result on unseen inputs.

    Slide text
    01 02 03 The ranking

    Each task is a whole working program, a workload and eight hours to make it faster

    Accelerate Bench task flow for SU2: the agent diagnoses, edits source and build files, validates on public tests, and a separate verifier scores the submission
    Liu et al. · Accelerate Bench · arXiv 2611.20882v1 · Figure 1, §2, §3 · 32 CPU tasks from six supercomputing competitions
  3. Across all 32 tasks, Claude Sonnet 5.5 cuts about two-thirds of the running time, and Claude Opus 5.5 comes second. Below those two, one run per agent can't rank models a few points apart.

    Slide text
    01 02 03 The ranking

    Claude Sonnet 5.5 cuts about two-thirds of the running time, Hy3 about a quarter

    AgentHarnessTime savedTasks ≥2×
    Claude Sonnet 5.5Claude Code67.4%23 of 32
    Claude Opus 5.5Claude Code59.4%22 of 32
    GPT-6.1 SolCodex49.2%16 of 32
    GPT-6 AstraCodex47.2%17 of 32
    Kimi K3Claude Code42.5%13 of 32
    Grok 4.6Claude Code33.9%11 of 32
    Hy3Claude Code25.5%9 of 32

    Score@1: mean share of runtime saved, failures count as zero · 7 of 16 agents shown · one run each

    Liu et al. · Accelerate Bench · arXiv 2611.20882v1 · Table 1, §3, §4 · benchmark v1.1
  4. Claude Opus 5.5 lost three tasks when safety classifiers flagged them as cybersecurity or biology work, and the test setup blocked Claude Code's fallback to an older Opus. On its other 29 tasks, it nearly matches Sonnet.

    Slide text
    01 02 03 The ranking

    Safety classifiers stopped three Claude Opus 5.5 runs, and those count as zero

    59.4% time saved on all 32 tasks, with Graph sparsification, PRESTO and VPIC scored as failures
    65.6% on its other 29 tasks
    Liu et al. · Accelerate Bench · arXiv 2611.20882v1 · Table 1, Appendix A · flagged as cybersecurity or biology content
  5. Of 512 submitted programs, 472 pass every correctness check. Speed separates the agents. Both Claude agents double the speed on 22 or more tasks, and the other fourteen on 8 to 17.

    Slide text
    01 02 03 The ranking

    Almost every program still computes the right answer, so speed decides the ranking

    Each agent's 32 outcomes split into incorrect, correct under 1.25×, 1.25 to 2×, 2 to 4× and at least 4× faster

    472 of 512 programs pass every check · 125 of the 412 correct programs from the other fourteen agents stay under 1.25×

    Liu et al. · Accelerate Bench · arXiv 2611.20882v1 · Figure 3, §4 · benchmark v1.1, first run
  6. In 476 runs from five agents on the benchmark's first version, changes to the computation itself show up in 225. Compiler and build tuning shows up in only 40, yet the build can decide the result.

    Slide text
    01 02 03 Where speed comes from

    Agents rewrite the computation and rarely touch the build

    Runs using each technique: computation or reuse 225, parallelism or vectorization 132, runtime or numerical 64, library or backend 46, compiler or build 40, I/O 28
    Liu et al. · Accelerate Bench · arXiv 2611.20882v1 · Figure 4, §5.1 · 476 labeled runs of five agents on benchmark v1.0
  7. In one GROMACS run, the agent changed only build and runtime settings. Replayed four ways, that same source ran anywhere from under a third of the original speed to twice as fast.

    Slide text
    01 02 03 Where speed comes from

    The same GROMACS source runs at 0.29× or 2.12× depending only on build and threads

    Slowest setting
    0.29×
    AVX2, 1 thread
    0.71×
    AVX2, 4 threads
    2.12×

    Speedup over the task's starting program · four correct settings replayed on unchanged source

    Liu et al. · Accelerate Bench · arXiv 2611.20882v1 · Table 31, §5.3
  8. In the quantum simulator QiboTN, deferring the tensor library imports did almost nothing alone. Once circuits also went to a fast exact simulator, the same change made the program more than six times faster.

    Slide text
    01 02 03 Where speed comes from

    In QiboTN, deferring library imports is worth 1.01× alone and 6.4× with a second change

    1.01× deferred loading of tensor libraries on its own
    6.4× the same change once circuits go to a fast exact simulator
    Liu et al. · Accelerate Bench · arXiv 2611.20882v1 · §5.3, Appendix L.2 · six circuits of 9 to 14 qubits, three rounds
  9. In one Palabos run, the default thread count saw all 192 host CPUs and took 11 minutes. Four threads took a second and a half. About half of all runs hit a dependency like this one.

    Slide text
    01 02 03 Where speed comes from

    A default thread count turned a 1.5-second Palabos run into 11 minutes

    11 min OpenMP default, seeing all 192 host CPUs
    1.5 s the same program on four threads

    About 52% of 480 runs (41–63%) met a build, runtime or code-path dependency that changed an optimization's speed or correctness

    Liu et al. · Accelerate Bench · arXiv 2611.20882v1 · §5.3 ④, Appendix K.2 · benchmark v1.0
  10. Each agent ran once, on CPU tasks shrunk to fit one machine, and repeat runs moved scores by two to four points. And each score belongs to a model and its harness together, Codex or Claude Code.

    Slide text
    01 02 03 Limits

    One run per agent, on CPU tasks scaled down to one machine

    CPU only 4 to 8 cores, up to 32 GiB · competition workloads shrunk to one node
    One run each repeat runs of GPT-6 Astra and Hy4-Preview varied by 2.5 and 4.4 points
    Model plus harness GPT models in Codex, the rest in Claude Code · not a base-model ranking
    Starting points vary speedups count from each task's baseline, tuned to different degrees
    Behaviour analysis five agents, version 1.0 runs · it doesn't explain the v1.1 ranking
    Liu et al. · Accelerate Bench · arXiv 2611.20882v1 · §2.2, §5, §7, Appendix A.3
  11. If I had a slow program, I'd hand it to Claude Sonnet 5.5 in Claude Code, as long as it times the whole run and I check its build and thread settings. That comes from one run per agent, on CPU only. The tasks are on GitHub as Accelerate Bench.

    Slide text

    For a slow program, Claude Sonnet 5.5 in Claude Code, with the build settings checked

    Liu et al. · Accelerate Bench · arXiv 2611.20882v1 · Table 1, §5.3, §7 · task packages on Harbor Hub and Hugging Face

What happened in AI today, in your inbox

One short email each evening. Read it and you’re caught up.

Built with ❤️ in New York