AutoScaffold writes temporary training hints, then removes them: students still beat RL baselines by 7.2 points on ALFWorld
Synopsis
Reinforcement learning scores each attempt with a single outcome reward, so the reasoning, actions and feedback inside a rollout are largely discarded, and information in failed trajectories is wasted. AutoScaffold has a frozen teacher model periodically read the student's training log, diagnose bottlenecks, and write class-level strategies plus hints for individual hard tasks, with class-level edits checked on held-out tasks before deployment. Evaluated without scaffolds, students beat the corresponding RL baselines in all four domains, by 7.2 points in ALFWorld success and 4.0 points in average QA exact match.
Interpretation
Text scaffolds written by a teacher model can be added to training prompts temporarily, and the student still scores above the same RL baseline when those scaffolds are removed at evaluation. Earlier scaffolds were mostly prepared by humans as hint tiers, partial solutions or skill libraries, and during training they either stayed fixed or changed through predefined rules. On ALFWorld, AutoScaffold reaches 93.9% success, 7.2 points above the GiGPO baseline; on search-based QA it reaches 46.1% average exact match, 4.0 points above GiGPO.
The teacher decides which training records to read and turns them into two kinds of scaffold: strategies shared across a task class and hints for individual hard tasks. The teacher gathers evidence through three query tools covering class statistics, individual trajectories and all attempts on one task, and keeps a memory of its last three decisions, instead of receiving one fixed sample of failures. Replacing the query tools and memory with a fixed sample of failed trajectories lowers ALFWorld success to 85.3%, 8.6 points below the full system.
A class-level scaffold is validated on held-out tasks before use, and the teacher uses the result to adopt, keep, combine or remove it. Without validation, a strategy inferred from a few trajectories would shape a whole task class for an entire training block. Removing scaffold validation drops ALFWorld success from 93.9% to 79.8%, below the GiGPO baseline.
Perspective
Evaluation covers ALFWorld, search-based QA, mathematical reasoning and KernelBench, with students from Qwen2.5-1.5B-Instruct to Qwen3-8B, and results are averaged over three independent runs; every score is measured with scaffolds removed, so it reflects the student's own capability.
The analysis characterizes the learning signal under the scaffolded prompt and does not model how gains transfer to the unscaffolded policy; the expected class-level signal is linear in the injection probability and would favor always injecting, while in practice always-on injection creates a training-evaluation mismatch, which is why the injection probability is capped. Teacher intervention cost and when learning with scaffolds improves the student's independent capability are left as future work.
