OpenVLHarness
OpenVLHarness gives vision models tools and a memory, and seven models each gain about 10 points
Yu Zhou, Cheng-Fu Yang, Rui Sun, Di Wu and colleagues · University of California, Los Angeles and Apple
OpenVLHarness wraps a vision model in tools for finding objects, reading text, measuring depth, searching the web and running code, and keeps every result on hand. With no retraining, seven models scored ten to thirteen points higher. It comes from UCLA and Apple.
Gains on every model family
A small open model goes from a third of the answers right to nearly half
On Qwen3-VL-8B, the smallest model, the average across 22 benchmarks rises from about 34 to 45 out of 100. GPT-6 Astra goes from 60 to 70, and improves on all 22.
Nine tools behind one interface
Nine tools sit behind one interface, and each answers with a picture
Object finding returns an overlay, a count and the box coordinates.
Every tool is called the same way, so a backend such as SAM 3 can be swapped without changing the prompts. The tools cover perception, search and code. Results come back as something the model can look at, like a labelled overlay with box coordinates.
Codex scored below the bare model
Inside Codex, GPT-6 Luna scored lower than on its own
22-benchmark average · medium reasoning effort
Codex ran with web search off, and ran code on 0.4% of questions.
Codex is OpenAI's coding agent. With GPT-6 Luna at medium effort, Codex averaged two points below the bare model, and OpenVLHarness added about eleven. Codex ran with web search off, and ran code on fewer than one question in two hundred.
Memory keeps every result usable
Masks, depth maps and web links stay usable for the whole question
+3.1 and +4.5 points from keeping the images in memory, on Qwen3-VL 8B and 32B
Every result is stored under a name that later steps can reuse. To count pills, the model searched for pills and for round pills, then merged the two sets of masks in code and got the exact count, 139. Keeping the images in memory, instead of text about them, added three to four and a half points.
More reasoning did not close the gap
Turning up the reasoning effort didn’t close the gap
More reasoning helps the bare model far less. At medium effort, OpenVLHarness beats Luna alone, Codex and Visual Sketchpad at their maximum effort by about eight points, for at most 37 percent of their token spend. Tool costs aren't included.
Where it still fails
Crowded scenes are still where it fails
Past about 25 objects, the detector misses or merges them, and three quarters of the grounding failures are wrong tool results. At low effort, the harness trails the bare model on general and spatial questions. And an older, smaller Qwen gains much less.
What running it takes
Running it takes GPU services, a paid search key and an isolated machine
The code is MIT licensed, with presets for the seven models tested. Object finding, text reading and depth each need their own GPU service, and search runs through Serper, a paid API. The Python agent runs generated code with your permissions, so give it an isolated machine.
Which model would you put inside it?
Which vision model would you put inside OpenVLHarness?
The harness, its tools and the evaluation scripts are on GitHub. Which vision model would you put inside it?
