Skip to main content
Search

# On-Policy Distillation

44 related articles

Subscribe to updates
  1. arXiv

    Long-sequence distillation stops handing the teacher the whole answer: ATI-OPD rewrites only the earliest divergent segment, and students' average accuracy on their own beats standard on-policy distillation

    On-policy distillation trains a student on its own responses using token-level teacher feedback, but as responses grow longer the student's prefix drifts from the teacher's own reasoning and later feedback weakens, bringing gradient spikes and extremely long responses. ATI-OPD has the teacher insert a short replacement at the earliest high-disagreement segment, hands generation back to the student, and excludes persistently divergent suffixes from the distillation loss. Across five mathematical reasoning benchmarks and two 1.5B teacher-student pairs, average student-only accuracy exceeds standard on-policy distillation.

  2. arXiv

    Long-horizon agents keep memory in plain files: CAMP-II averages 64.6% across five environments, above the 54.2% task-reward reference CAMP-I

    Long-horizon agents accumulate interaction histories that outgrow the active context, so they must learn what to write down and when to read it back. Earlier on-policy distillation of memory control relied on dedicated memory tools or compaction protocols that neither the teacher nor the student had seen, which weakened guidance on the teacher side and left the student sampling too few meaningful memory actions. CAMP-II instead routes memory through ordinary Bash file operations that both models already have a prior for, training one policy jointly across five long-horizon environments to a 64.6% average task success rate against 54.2% for the task-reward reference CAMP-I, with transfer to Terminal-Bench 2.1 and SWE-Bench Pro V2.

  3. arXiv

    History Forcing teaches few-step video models to remember: a 1.3B student reaches 79.82 on 60-second VBench-Long

    Few-step autoregressive video diffusion models gradually lose subject and scene information over long rollouts, and existing distillation targets do not constrain distant history separately. History Forcing builds a context residual from the difference between one autoregressive teacher's predictions with and without extra history, then weights it into the DMD or OPSD distillation target. A 1.3B student reaches a VBench quality score of 85.07 for 15-second generation and a VBench-Long aggregate of 79.82 for 60-second generation, and raises PlayWorld Evolution Quality from 2.35 to 2.70 when the camera leaves and returns.

  4. arXiv

    Diffusion language models can learn from autoregressive teachers: JustOPD lifts Dream-7B's math reasoning while keeping any-order decoding

    Autoregressive models generate strictly left to right, while diffusion language models denoise in arbitrary order, so the former's token-level supervision does not transfer directly. JustOPD forces the diffusion student to roll out left to right during training, letting a Qwen3-14B teacher supervise every intermediate state, while inference keeps any-order decoding. With Dream-7B as student and Qwen3-14B as teacher on GSM8K and MATH500, scores rise over the base model across several divergences.

  5. arXiv

    When students write wrong, teachers drift too: SPTA first teaches the teacher to continue from a student's half-finished reasoning

    On-policy distillation trains a student on trajectories the student itself generates, but student prefixes can carry reasoning errors or patterns unfamiliar to the teacher, and the teacher's token-level supervision becomes less reliable as prefixes grow longer. Before distillation begins, SPTA trains the teacher on student prefixes of random lengths, rewarding correct final answers, so the teacher learns to recover from student-induced states rather than imitate the student's reasoning; standard on-policy distillation then proceeds with the distillation objective unchanged. On mathematical reasoning benchmarks, students trained with SPTA average better than under standard on-policy distillation and better than under on-policy distillation with task-level RLVR-adapted teachers.

  6. arXiv

    Joint audio-video generation stops letting audio drag: OmniOPD steadies distillation by giving the audio teacher a cleaner video condition

    In joint audio-video generation, audio has far fewer tokens than video, so its supervision is sparse and distillation is unstable, and simply raising the audio loss weight makes things worse. OmniOPD has the audio teacher produce targets from a cleaner visual condition, samples several trajectories per prompt, and weights each timestep by teacher-student transition divergence; on JavisBench it raises video quality VQ from 2.714 for LTX-2.3 to 3.347 while holding audio quality AQ at 5.449, below the audio teacher's 5.576.

  7. arXiv

    Distillation Rollouts Stop Copying the Student: Qwen3-4B Math Average Gains 6.91 Points and Generation Length Drops 41%

    On-policy distillation no longer treats the student's own samples as the only rollout distribution: previously each round sampled from the current student, forcing the teacher to supervise token by token on trajectories the student reaches but the teacher rarely does, so that supervision becomes noisy and degenerate patterns such as repetition are sampled again in later rounds. The new method reweights by relative teacher support, derives a geometric mixture of student and teacher inside a KL trust region, and trains a proposal policy to approximate that mixture in one forward pass. Across 16 math, knowledge QA and medical reasoning benchmarks, Qwen3-4B-Base gains 6.91 average accuracy over OPD on math while generating 41% shorter responses.

  8. arXiv

    Multi-turn agent distillation stops feeding student reasoning to the teacher: a 4B student beats its 32B teacher on ALFWorld Seen at 43.39%

    Multi-turn agent distillation now grounds teacher supervision in the interaction state, keeping only executed actions and observations, and the Qwen3-4B student surpasses its 32B teacher on ALFWorld Seen at 43.39%.

  9. arXiv

    MLP linearization moves to low-degree rational fits: Qwen3-8B drops 14% of parameters and decodes faster

    Instead of relying only on polynomial random features, MLP linearization now approximates activations with low-degree rational functions and splits each layer, via partial fractions, into separable weight-side and input-side inner products. Across synthetic fitting, vision, 3D and language models this route gives lower error at the same feature budget; compressing Qwen3-8B by replacing its last eight feed-forward layers removes 1.17B parameters (14% of the model) and raises decoding throughput from 763.3 to 809.6 tokens/s, with reasoning and knowledge benchmarks roughly at parity with the uncompressed model.

  10. arXiv

    Teacher supervision stops being copied token by token: reward-weighted distillation lifts math benchmarks by up to 6.5 points

    Distillation no longer treats every teacher token alike: each token's weight is set by its predicted effect on the student's post-update reward. Earlier token-selection methods relied on proxies such as uncertainty or teacher-student divergence, which need not identify reward-improving supervision. RAD scores each token by differentiating through a temporary optimizer step, then updates token weights by projected ascent, improving mean accuracy on six competition-math benchmarks by 6.5 points for Qwen3-1.7B and 2.5 points for Qwen3-4B-Instruct-2507 at a matched training-response budget, and reaching 54.69% and 58.67% pass@1 on LiveCodeBench-v6.