Skip to main content
Back to timeline
arXivSource publication:

Long-sequence distillation stops handing the teacher the whole answer: ATI-OPD rewrites only the earliest divergent segment, and students' average accuracy on their own beats standard on-policy distillation

Synopsis

On-policy distillation trains a student on its own responses using token-level teacher feedback, but as responses grow longer the student's prefix drifts from the teacher's own reasoning and later feedback weakens, bringing gradient spikes and extremely long responses. ATI-OPD has the teacher insert a short replacement at the earliest high-disagreement segment, hands generation back to the student, and excludes persistently divergent suffixes from the distillation loss. Across five mathematical reasoning benchmarks and two 1.5B teacher-student pairs, average student-only accuracy exceeds standard on-policy distillation.

Source-provided article image: ATI-OPD: Adaptive Teacher Intervention for Stable Long-Sequence On-Policy Distillation
arXiv

Interpretation

Training stability in long-sequence on-policy distillation can be improved by revising the prefix the student has already generated, not only by capping response length or reweighting the loss. Earlier length-control and loss-reweighting approaches limit learning from problematic continuations while leaving the generated prefix unchanged, and fixed cutoffs can discard later reasoning. On the JustRL-DeepSeek pair, ATI-OPD keeps the teacher-student absolute gap lower and teacher top-16 coverage higher for longer, reduces the raw gap area under the curve relative to standard on-policy distillation, and shows steadier gradients with lower, less variable student entropy.

The teacher inserts a length-bounded replacement only at the earliest high-disagreement segment, then returns generation to the student, who continues its own reasoning from the revised text. Earlier forms of teacher participation mixed sampling distributions, replaced individual tokens, or took over the remaining suffix; ATI-OPD triggers on segment-averaged divergence and performs a bounded replacement. Pairing student continuations after the original triggering segment with those after the teacher segment, up to 16 valid tokens per side, the mean paired decrease in sampled-token divergence is positive over 217 training steps, with a decrease in 193 of those steps.

The student can diverge again after control returns, so distillation supervision is applied only to the parts of the regenerated continuation that do not diverge persistently. Supervising the entire continuation and stopping supervision at the teacher segment are the two extremes; ATI-OPD selects the supervision range from persistent divergence between them. Across five tasks, both extremes, supervising the entire continuation and stopping at the teacher segment, perform worse, indicating that useful student supervision remains after the handoff and that more interventions do not replace selecting the supervision range.

Perspective

The demonstrated setting is mathematical reasoning distillation on DAPO-Math-17K with two 1.5B teacher-student pairs, one training run each, evaluated on AIME 2024, AIME 2025, AMC 2023, AMO-Bench and HMMT February 2026; an exploratory 7B-student comparison covers four tasks.

The method adds teacher decoding, student regeneration and scoring costs, and selects supervision only after full generation, so it does not reduce those decoding costs; the trigger-threshold sensitivity comparison has one training run per setting and describes observed sensitivity rather than establishing a statistically optimal threshold.