An EDA scripting agent stops guessing: reading back full execution state lifts Pass@1 from 19.7% to 76.5% across 449 tasks
Synopsis
For three EDA scripting interfaces, PyAether, SKILL and Tcl, an execution-grounded coding agent closes a generate-execute-read-back loop and reaches 76.5% pooled Pass@1 on the 449-task EDA-Scripting-Bench, against 19.7% for a pure LLM. With the model fixed, the observation axis spans 56.8 percentage points; with the harness fixed, the backbone axis spans 23.3 points. The authors frame this as a grounding gap: scripting correctness is bounded by how faithfully the agent observes the live design state, which model scale alone cannot supply.
Interpretation
The accuracy bottleneck for EDA scripting agents is located in observation: how much of the current design state the agent can read back after executing a script decides whether it can repair the script. Earlier custom/analog EDA script generation stopped short of execution, and static-knowledge-base approaches did not run scripts against a live design database; digital-flow agents with execution feedback did not ground against the live layout, schematic or waveform database of custom/analog tools. Across 449 tasks, three languages and four commercial runtimes, the full system reaches 76.5% pooled Pass@1 versus 19.7% for the pure LLM; with the model fixed the observation axis spans 56.8 percentage points, and with the harness fixed the backbone axis spans 23.3 points.
The loop runs on four unified MCP tools: semantic and keyword API retrieval, API documentation lookup, and sandboxed execution that reads back a structured result. Retrieval and execution are separated into components that can be switched on or off by tool registration, while the policy stays fixed, so differences are attributable to the interface rather than the model. Removing retrieval lowers pooled Pass@1 by 10.3 percentage points, and removing sandbox execution drops it from 76.5% to 68.6%; read-back information is ordered as no read-back, binary pass/fail, and full observation, with full observation best pooled and on every slice.
Perspective
The conclusions rest on the 449 tasks of EDA-Scripting-Bench, covering PyAether, SKILL, and Tcl on WaveView and Innovus, scored by error-free execution in the target runtime sandbox with all assertions holding. Pass@1 certifies only the effects its assertions check, not that the produced design passes layout-versus-schematic or design-rule checks. The system is deployed in production EDA flows, but deployment metrics are not given.
The authors state the grounding-gap framework as a testable prediction rather than a fitted explanation, and note that the implemented observation channel covers one facet of L4, leaving organizational conventions and waveform imagery outside it, so the practical grounding gap keeps a positive lower bound. Benchmark size limits statistical power to certify the smallest effects, and the SKILL-slice self-exploration difference remains within noise; per-slice dispersion can be large, so slice-level comparisons need care. Only 0.2% of non-empty SKILL error logs carry a line number, and without a location cue the agent rewrites rather than repairs.
