Skip to main content
Search

# Data-Free On-Policy Distillation

1 related articles

Subscribe to updates
  1. arXiv

    Eight real prompts match 17k problems: data-free on-policy distillation hits full-data parity with 64 self-generated questions

    Across two representative single-teacher on-policy distillation (OPD) settings, training on 8 real prompts yields performance comparable to training on 17k problems, and datasets differing substantially in measured difficulty and initial distillation gap yield similar outcomes; building on this, the authors propose a data-free OPD (DF-OPD) setting in which 64 self-generated questions obtained without seed examples match full-data OPD in both single-teacher settings, and in multi-teacher OPD 1k generated questions across mathematics, code, and instruction following match training on approximately 7k real post-training examples.

You’re all caught up