Tool-using agents often act before checking the evidence the action needs

Tool-using agents often issue a refund or change a record before checking the facts that justify it. That's the finding of SafeActBench, from researchers at the National University of Singapore and Princeton, among others.

Slide text

Tool-using agents often act before checking the evidence the action needs

Hongzhan Lin, Shidong Cao and colleagues · National University of Singapore, Hong Kong Baptist University, Amazon Web Services, Princeton University

Lin, Cao et al. · From Evidence to Action · arXiv 2610.07753v1 · submitted 6 Oct 2026

Here's the paper's example. The agent looks up charge C1. Then it refunds charge C2, and that succeeds. C2 was eligible, so the outcome looks right, but the agent never checked it.

SafeActBench has 656 cases in six kinds of work, from customer refunds to smart-home control and healthcare. A fixed program replays every run and checks what the agent knew before each action.

The easiest cases only ask the agent to judge a proposed action. Harder ones make it investigate and hold back, act once, or run a chain where each action depends on the last.

The models were Claude Opus 5, GPT-5.6 Sol, DeepSeek V4 Flash, Qwen 3.8 Flash and GLM 5.2, each in its maker's harness, like Claude Code. Each also ran in one shared harness.

Given the evidence and a proposed action, three setups judged it correctly at least 95 percent of the time. Asked to gather that evidence and act themselves, they got the same cases right about half the time or less.

So where does it break? Across the ten setups, between about a third and two thirds of single actions came before the agent had gathered the evidence the case required.

When the agent did gather everything first, the action itself was reliable. Nine setups got those cases right 93 to 100 percent of the time. DeepSeek in the shared harness reached 83.

Some cases call for no action, once the agent finds out why. Between about a fifth and three fifths of those runs stopped before the required checks, holding back without establishing the reason.

In workflows, an action can come up while a fact it needs is still missing, in up to 80 percent of chains for GLM in ZCode. Or some actions run and the workflow stops short.

Next, the authors removed one record an action depended on. Agents still acted in about half those runs. In 65 of 66, the agent had queried the tool that should have returned it.

A system warning that missing evidence doesn't make an action safe didn't measurably change that in any of the three setups. Neither did an urgency cue in the task.

On the same 570 cases, DeepSeek did about four points better in its own harness. GLM did about seven points worse in ZCode. For Claude, GPT and Qwen, the gap was within the margin of error.

The tasks are synthetic, with fictional records, and the authors say results aren't safety certifications. The evaluator sees tool calls, not what the model relied on. The removed-record tests covered 43 cases.

Ten setups, synthetic tasks. Within that, every setup acted early on at least a third of its single actions. The paper is on arXiv, and the benchmark and evaluator are on GitHub.