Skip to main content
Back to timeline
arXivSource publication:

Agent training data no longer needs pure expert demos: JudgeMix judges each step and lifts pass@1 by up to 8 points on AppWorld

Synopsis

Supervised fine-tuning of AI agents on expert trajectories leaves training data mismatched with the target model's inference-time state distribution, so the model is brittle once it deviates from the ideal path. JudgeMix uses a judge to decide at each step whether the target model's proposed action is semantically correct, hands control to the expert only when the model struggles, and reweights expert supervision by that judgment. On the AppWorld and AutomationBench tool-use benchmarks, JudgeMix reaches the highest pass@1 for both Qwen3.5-2B and Qwen3.5-9B, up to 8 and 10 points over pure expert demonstrations.

Source-provided article image: JudgeMix: Judge-Guided Mixed-Rollout Data Synthesis for LLM Agents
arXiv

Interpretation

Agent SFT data synthesis moves from expert demonstrations or random mixing to step-wise semantic judging that decides when the expert intervenes, keeping rollouts both successful and close to the target model's own state distribution. Earlier mixed approaches act blind to the target model's behavior: On-policy Expert Correction lets the expert take over at a step sampled uniformly, and DAgger inserts expert actions by chance, so either can disrupt a correct generation or intervene only after an unrecoverable error. On AppWorld and AutomationBench, JudgeMix reaches the highest pass@1 in the table for both Qwen3.5-2B and Qwen3.5-9B; against pure expert demonstrations it gains up to 10 points on AutomationBench Test-N and up to 8 points on AppWorld, and against DAgger, usually the runner-up, it leads by more than 5 and 7 points respectively.

At each step the target model first proposes an action, a judge returns a binary semantic verdict, and an accepted action continues the rollout while a rejected one is replaced by the expert's action to keep the trajectory on track. The judge is instantiated by the expert model, GLM-5.1 for AppWorld and GLM-5.2 for AutomationBench, so the decision rests on whether the step productively advances the task rather than on a random heuristic. Under a fixed budget of 8 trajectories per training task, judge-guided rollouts reach a pass@1 27 points above random intervention; on AppWorld with Qwen3.5-2B, JudgeMix trajectories average 14.7 steps versus 18.4 for DAgger.

Actions rejected by the judge are temporarily committed to the environment to record error-inclusive histories, which are paired with expert corrective actions and added to training as error-recovery supervision. Prior target-model sampling kept only successful trajectories, so the model rarely saw the states that follow its own mistakes and could not learn to recover from them. Removing the error-recovery data lowers Qwen3.5-2B by 7.5 and 5 points on AppWorld Test-N and Test-C, and also lowers Qwen3.5-9B on both test sets.

Perspective

The demonstrated scope is two tool-use benchmarks, AppWorld and AutomationBench, with Qwen3.5-2B and Qwen3.5-9B as target models, GLM-5.1 and GLM-5.2 as expert and judge, 8 trajectories per training task, and pass@1 averaged over 8 evaluation runs. The authors explicitly leave cross-model transfer to future work.

The authors note that the synthesized dataset is coupled to a specific target model, so each target model needs its own data generation; querying the judge at every step adds computational overhead; and effectiveness ultimately depends on judge reliability, since an unreliable judge in open-ended domains with ambiguous progress could hurt rollout quality and supervision reweighting.