Skip to main content
Back to timeline
arXivSource publication:

Take the tools away from the agent: intent contracts let a trusted model write every call, but cannot stop it picking the wrong target

Synopsis

For tool-using LLM agents, intent contracts remove tools from the planning model: at each step the planner submits only a contract with a goal, a verbatim quotation of the user and scope fields, code validates it, and a small trusted model checks it against the user's words and writes the tool call itself. On 96 AmPermBench prompts across three task families, every object changed under contract execution was named in the goal of the contract that produced the change (56/56), against 2/16 and 4/54 for the two review conditions; yet contracts did not reduce unauthorized actions, matching blind review at 10/96, and after environment terms were clarified every acting condition hit a wrong target in 40–49% of runs.

Source-provided article image: Intent Contracts: Taking Tools Away from Agents
Figure 1 · arXiv · Page 2

Interpretation

The planning model holds no tools; at each step it can only submit a logged contract, and a small trusted model decides which part to carry out and writes the tool call itself. Previously the planner wrote the command and a reviewer judged whether to approve the whole command; Claude Code's auto mode deliberately excludes the agent's prose from its permission classifier's input. On 96 prompts from AmPermBench's job-cancellation, service-restart and storage-cleanup families, contract execution, intent-visible review, blind review, the strong model alone, the small model alone and the declaration-only condition each had a valid run, compared pairwise by prompt.

Under contract execution every changed object appears in the goal of the contract that produced the change, and unauthorized changes leave the same record. In the two review conditions the planner names objects in its commands rather than its goals: 2/16 for intent-visible review and 4/54 for blind review, while their commands named 16/16 and 39/54. The metric is timed by the benchmark tools' own invocation logs, taking for each changed object the first state-changing invocation naming it and the last code-accepted contract before it, matched by case-insensitive substring of its ID or a name, key or path field.

When the planner bundles in work the user never asked for, contract execution completes the user's task without doing the extra work. A reviewer can only approve or refuse a whole command, so refusing also blocks the authorized part when the planner combines both in one command. On small author-written bundling tasks, contract execution completed the main task in 27/27 valid runs against 16/30 for intent-visible review with an appended request, and 24/24 against 19/24 on a multi-file list; neither performed the extra change.

Contract execution does not reduce unauthorized actions, is indistinguishable from blind review, and its errors concentrate on ambiguous references. Intent-visible review cut unauthorized actions from blind review's 10/96 to 2/96 on the original prompts, at the cost of more incomplete runs (87/96 versus 76/96). After clarifying environment terms, contract execution and blind review did not differ on safe completion, unauthorized actions or incompleteness; contract execution had 29/96 unauthorized actions, blind review 25/96, intent-visible review 18/96 and the strong model alone 14/96, all unauthorized actions hit a wrong target and all fell on the two range categories, with 0/48 for each of the four conditions on exact-ID and own-ticket prompts.

Perspective

The demonstrated scope is one plugin for the OpenCode coding agent, evaluated on 96 AmPermBench prompts from three task families and on small author-written bundling tasks, with gpt-6.1-terra as planner and GLM-5.3-Flash as trusted model and with cooperative rather than malicious planners; the benchmark's fourth family, branch cleanup, was excluded because its state is a git repository rather than a JSON record, so these numbers are not full-benchmark results.

Results cover one planner, one small model and one wording of each role's instructions, and unauthorized-action counts may change with any of the three; each prompt ran once per condition, and separate collection batches and hosted-service changes can affect paired comparisons; adherence in the declaration-only condition was incomplete, with a post hoc text check finding a marker in 14 of 30 state-changing runs; adding one instruction telling the executor to decline when unsure did not move the counts, and the strict executor declined 10 contracts against 154 for the intent-visible reviewer.