Skip to main content
Back to timeline
arXivSource publication:

Natural-language agent skills can be repaired like programs: Synapse tops six benchmark settings, beating the strongest baseline by up to 4.54 points

Synopsis

For natural-language skill documents used by large language model agents, Synapse compiles a skill into a source-linked partial program view and then generates and validates patches through fault localization, dependence analysis and regression test selection. Across two single-turn question-answering benchmarks, DocVQA and LiveMath, and three execution models, it reaches the highest held-out success rate in all six settings, 1.60 to 4.54 percentage points above the strongest baseline in each. An ablation that replaces the intermediate-representation repair and its screening with constrained text editing lowers success rates in all four evaluated settings.

Source-provided article image: Synapse: Optimizing Natural-Language Agent Skills through Program Repair
arXiv

Interpretation

Skill optimization no longer relies only on rewriting stretches of natural language: the operations, conditions and data references recognized in a skill document are compiled into SkIR, a source-linked intermediate representation, so each edit lands on the mapped span of an existing instruction. Earlier trace-driven, attribution-guided and structure-aware methods express an edit as rewritten text or act on a structured representation that does not correspond to the authored document, so they cannot say which span of the existing skill an edit replaces. Across two recorded DocVQA repair runs, 292 of 315 repair targets (92.70%) are existing instructions with valid source mappings; of 38 proposals from the repair model, 37 pass type checking and all 37 pass the round-trip observations after write-back.

Patch screening reruns more than the tests that exposed the failure: control-flow and dependence relations pull in tests on related instructions, including tests that never cover the edited instruction. Existing methods check an edit against the motivating failure, source tasks, a fixed task set or the examples affected in a skill graph, so a regression passed through a neighboring instruction can go unseen. In the two DocVQA runs, related instructions add screening tests for 23 of 37 patches (62.16%); in the case study a receipt test enters screening only through a control-flow relation and exposes a regression that coverage of the edited instruction misses.

On DocVQA and LiveMath, skills repaired by Synapse reach the highest held-out success rate with all three execution models, and each final skill results from at most three accepted localized patches. On LiveMath, Trace2Skill and SkillBoost end below the initial skill with GPT-5.6 Luna (36.00 and 32.80 versus 36.80), and Trace2Skill also with Gemini 3.6 Flash (32.00 versus 32.80). Synapse exceeds SkillOpt by 3.74 to 4.54 percentage points on DocVQA and 2.40 to 4.00 on LiveMath and beats all three baselines in all six settings; on DocVQA it removes about half of the initial skill's held-out errors (48% to 58%).

Perspective

Evaluation is confined to the single-turn question-answering benchmarks DocVQA and LiveMath, using the same initial skills as SkillOpt, fixed repair, validation and held-out splits, GPT-6.1 Sol as the repair model, and GPT-5.6 Luna, GPT-6 Sol and Gemini 3.6 Flash as execution models. The mechanism analysis comes from two recorded DocVQA repair runs, and the case study from a separate historical DocVQA run.

Safe test selection holds only when the execution agent is one-hop local, and that assumption is not measured; localization and test selection rely on the agent's self-reported instruction usage, whose accuracy is also not evaluated. The recognizers accept only a fixed lexicon and fixed sentence forms, so skills written differently yield sparser dependence graphs. Validation suites hold only 53 and 17 tests and answers vary across requests, so a patch can be accepted by chance. Effects beyond the one-hop neighborhood are covered only if reports or named tests reach them.