Skip to main content
Back to timeline
arXivSource publication:

SIGA wraps an off-the-shelf coding agent in a simulator adapter: on harder GEOS tasks TreeSim rises from 0.720 to 0.789 and run-to-run spread shrinks about 16-fold

Synopsis

Configuring a scientific simulator, turning a modeling goal into a valid, runnable input deck, has long been a time sink for domain scientists, who must satisfy simulator-specific vocabulary, cross-file references and validation rules. SIGA leaves the base model and agent loop frozen and supplies that executable contract at three harness interfaces: retrieval, always-on procedural memory, an agent-callable validator, and a validation-gated stop hook. On harder held-out GEOS multiphysics subsurface tasks, the full self-evolved adapter raises mean TreeSim from 0.720 to 0.789 and cuts across-run standard deviation by about 16-fold, mainly by preventing empty or invalid decks.

AI-generated editorial illustration: SIGA: Self-Evolving Coding-Agent Adapters for Scientific Simulation

Interpretation

Configuring a scientific simulator no longer requires building a bespoke simulator agent from scratch; a lightweight adapter around an off-the-shelf coding agent supplies the simulator's executable contract. Earlier simulator agents mostly wrote their own planning, tool calling, retry logic and termination handling, whereas modern coding agents already navigate files, edit code, run commands and repair outputs, and lack only the simulator's vocabulary, structural rules, validation procedures and completion conditions. On harder held-out GEOS multiphysics subsurface tasks, the full self-evolved adapter raises mean TreeSim from 0.720 to 0.789 and cuts across-run standard deviation by about 16-fold, largely by preventing empty or unparseable decks.

The adapter changes only three interfaces of the coding-agent harness, leaving the model and the agent loop frozen. Retrieval adds semantic search over GEOS documentation, XSD schema entries and example files, covering the case where keyword search is brittle because the agent does not know the right simulator terms; procedural memory is a 775-token cheatsheet injected into the always-on system prompt. The same XML schema check appears in two forms: an agent-callable validation tool and a stop hook at each attempted termination that returns structured repair feedback when the deck is empty, unparseable or schema-invalid; when enabled, the validation tool is called about three times per task on average.

The adapter's own contents can be evolved offline, with a coding-agent proposer reading prior trajectories and rewards to rewrite the adapter text without touching the underlying agent. A hand-designed adapter fixes its contents in advance, while the self-evolved version searches over a richer task primer, cheatsheet and auxiliary skills on a held-out validation-selection split. On held-out GEOS tasks, SE-prose, which substitutes only the self-evolved primer and cheatsheet, reaches 0.775 and the full self-evolved adapter SE reaches 0.789, against 0.720 for the bare agent.

Perspective

The demonstrated scope is GEOS multiphysics subsurface deck authoring plus transfer studies on 30 OpenFOAM tasks and 9 LAMMPS tasks, aimed at researchers and engineers who write configurations for scientific simulators. TreeSim measures structural similarity rather than physical correctness, and a 0.8 deck is not guaranteed to run.

The main GEOS result uses three runs of deepseek-v4-flash and the cross-model panel is single-run; the transfer studies are also single-run, with OpenFOAM scored by a file-text-and-coverage metric and LAMMPS by a non-deterministic LLM judge. The held-out lift is concentrated in two tasks, and the human baseline is two participants on a single task. In the autonomy study, the human supervisor the agent could consult was actually another language model acting as a user simulator.