Skip to main content
Back to timeline
arXivSource publication:

Treat the model as untrusted: a deterministic monitor blocks injected tool calls, yet in-policy semantic fakes remain

Synopsis

Tool-using AI agents mix instructions and data in one channel, so prompt injection is structural rather than a wording problem. In an executable environment of 13 tasks and 12 tools, one effect-based oracle compared in-prompt guardrails, deterministic monitors outside the model, human approval and taint tracking, with real runs on Qwen2.5-1.5B and Qwen2.5-7B: the task-bound monitor admitted 1.3% and 7.7% of attacks while the models proposed the attacker's call in 64% and 84% of episodes. The residual is in-policy semantic falsification, addressed by taint-triggered human review, and that part is simulation-only.

Source-provided article image: Treating the Model as an Untrusted Component: Deterministic Policy Enforcement, Human Approval and Taint Tracking Against Prompt Injection in Tool-Using Agents, with Validation on Open-Weights Models
arXiv

Interpretation

The security guarantee moves out of the model: a small deterministic mechanism authorises every proposed tool call, and the model itself is treated as an untrusted, non-deterministic component. The prevailing approach makes the model better at ignoring injected text, and its security argument is probabilistic and assumes a non-adaptive attacker. In an executable environment of 13 tasks and 12 tools, judged by an effect-based oracle that never consults a defence, the task-bound monitor admitted 1.3% and 7.7% of attack episodes on Qwen2.5-1.5B and Qwen2.5-7B, while the models proposed the attacker's call in 64% and 84% of episodes.

Monitor implementation quality decides whether argument-level evasions get through, and naive and hardened implementations differ sharply. Of 33 effective attack variants, a tool allow-list alone would admit 27, naive argument checks 18, hardened argument checks 6 and task-bound capabilities 1. Against the argument-fuzzing A3 adaptive attacker, the naive monitor reached 84.6% attack success at 25 attempts and the hardened one 30.8%, while against non-fuzzing attackers the two were indistinguishable.

Human approval helps only behind a deterministic monitor, and it needs a deterministic approval budget to resist approval spamming. With human review plus taint tracking but no monitor, a fully compromised model produced 100.0% severe actions and harm 10.00. In simulation, taint-triggered review cut the task-bound monitor's residual harm from 0.69 to 0.26 and attack success from 23.1% to 8.7%, and an approval budget of 1 to 2 cut the monitor-free variant's harm from 10.0 to 2.3-4.4.

Perspective

The demonstrated scope is 13 author-written task templates, 12 tools, one effect-based oracle and static attack families on the two open-weights models Qwen2.5-1.5B and Qwen2.5-7B; the human-approval and taint-tracking layers were run only in simulation, not against a real model.

The model is mostly simulated: guardrail erosion under adaptive attackers, the reviewer model and taint-tracking failures remain assumptions, and the larger model's detector stack contradicts the assumed erosion law; the oracle requires the attacker's exact phrase, so paraphrased semantic falsification is undercounted; undecodable or empty outputs ended 13% and 1% of attack episodes for Qwen2.5-1.5B and Qwen2.5-7B, so attack rates are lower bounds.