OpenVLHarness gives vision models tools and a memory, and seven models each gain about 10 points

OpenVLHarness wraps a vision model in tools for finding objects, reading text, measuring depth, searching the web and running code, and keeps every result on hand. With no retraining, seven models scored ten to thirteen points higher. It comes from UCLA and Apple.

Slide text

OpenVLHarness gives vision models tools and a memory, and seven models each gain about 10 points

Yu Zhou, Cheng-Fu Yang, Rui Sun, Di Wu and colleagues · University of California, Los Angeles and Apple

Zhou et al. · OpenVLHarness · arXiv 2610.26860v1 · submitted 19 Oct 2026

On Qwen3-VL-8B, the smallest model, the average across 22 benchmarks rises from about 34 to 45 out of 100. GPT-6 Astra goes from 60 to 70, and improves on all 22.

Slide text

01 02 03 The result

A small open model goes from a third of the answers right to nearly half

Zhou et al. · OpenVLHarness · arXiv 2610.26860v1 · Table 1, Table 7, Figure 1a · GPT-6 results are single runs

Every tool is called the same way, so a backend such as SAM 3 can be swapped without changing the prompts. The tools cover perception, search and code. Results come back as something the model can look at, like a labelled overlay with box coordinates.

Codex is OpenAI's coding agent. With GPT-6 Luna at medium effort, Codex averaged two points below the bare model, and OpenVLHarness added about eleven. Codex ran with web search off, and ran code on fewer than one question in two hundred.

Every result is stored under a name that later steps can reuse. To count pills, the model searched for pills and for round pills, then merged the two sets of masks in code and got the exact count, 139. Keeping the images in memory, instead of text about them, added three to four and a half points.

Slide text

01 02 03 How it works

Masks, depth maps and web links stay usable for the whole question

+3.1 and +4.5 points from keeping the images in memory, on Qwen3-VL 8B and 32B

Zhou et al. · OpenVLHarness · arXiv 2610.26860v1 · Figure 3c, Table 3

More reasoning helps the bare model far less. At medium effort, OpenVLHarness beats Luna alone, Codex and Visual Sketchpad at their maximum effort by about eight points, for at most 37 percent of their token spend. Tool costs aren't included.

Slide text

01 02 03 How it works

Turning up the reasoning effort didn’t close the gap

Zhou et al. · OpenVLHarness · arXiv 2610.26860v1 · Table 2, Figure 1c · model tokens only, tools not priced

Past about 25 objects, the detector misses or merges them, and three quarters of the grounding failures are wrong tool results. At low effort, the harness trails the bare model on general and spatial questions. And an older, smaller Qwen gains much less.

The code is MIT licensed, with presets for the seven models tested. Object finding, text reading and depth each need their own GPU service, and search runs through Serper, a paid API. The Python agent runs generated code with your permissions, so give it an isolated machine.

The harness, its tools and the evaluation scripts are on GitHub. Which vision model would you put inside it?