Skip to main content
Back to timeline
arXivSource publication:

System 1 agents call tools without writing a word: 93.9% fully correct on 1,952 requests and zero invented content in 10,888 calls

Synopsis

Agents for sensitive clinical data usually depend on large language models: cloud models send data off site, while local models invent values inside tool calls, and one invented value silently propagates through every later analysis step. System 1 agents turn each tool call into multiple-choice questions with finite options, answered in one forward pass by a local decision model that generates no text. On 1,952 complete and out-of-scope requests, the best System 1 agent was fully correct on 93.9%, above all 18 local LLM-agent settings (best 91.3%); none of 10,888 System 1 calls contained invented content, against 5.5–19.7% of LLM-agent calls.

Source-provided article image: System 1 agents: a method for local, reliable and fast agentic AI
arXiv

Interpretation

A tool-calling agent can generate no text at all and only choose among a finite set of options, completing imaging and biomedical tool calls locally. The prior approach had a local large language model write each tool call token by token, so it could call a tool that does not exist, use an argument the tool lacks, or fill an argument with a value the user never gave. On 1,952 complete and out-of-scope requests, the best System 1 agent, pplx-decider-27b, was fully correct on 93.9% (95% confidence interval 92.7–94.8), against 91.3% for the best LLM setting, Qwen3.5-27B with reasoning off; across 108 pairs, System 1 led significantly in 73, the LLM in 11, with no significant difference in 24.

Each call is decomposed into multiple-choice questions whose options come from the request text, the tool's argument schema or the agent's own policy, and the model returns only a probability per option. A deterministic compiler turns the chosen tool's JSON schema and the request into questions: an enumeration is asked over its values, a boolean over on, off or not mentioned, paths and numbers over candidates the compiler extracts, and a missing required value is left unfilled and put to the user instead. The evaluation covered 157 tools of 18 MCP servers spanning image classification, segmentation, radiomics, AutoML, proteomics, metabolomics, genomics, single-cell and spatial transcriptomics, clinical trials and chemistry; six decision models of 4 to 35 billion parameters ran without fine-tuning, each locally on one GPU through llama.cpp.

Perspective

The evaluation ran locally on one 48 GB GPU in a single workstation, covering 157 imaging and biomedical tools of 18 MCP servers and 2,126 requests, of which 1,952 complete and out-of-scope requests form the main benchmark and 174 underspecified requests were assessed separately; no request, tool schema or answer was sent to an external service during the runs.

System 1 agents are not error-free: they still make wrong choices among grounded options, and in 22 of 10,888 calls sent a value written in the request in a form the schema rejects; out-of-scope requests where a tool performs the task but not on the given input were the hardest for every agent. On argument filling, five LLM-based agents were not significantly different from the best System 1 agent, and on underspecified requests each System 1 agent filled a missing required value with another value written in the request in 2 to 6 cases, a wrong choice rather than an invention.