Skip to main content
Back to timeline
arXivSource publication:

Agentic on-policy distillation rewrites ReAct habits: an ALFWorld teacher drops customer-service accuracy from 41.5% to 6.6%

Synopsis

On-policy distillation transfers a domain teacher's capability to language-model agents, but across domains it also rewrites how they organize reasoning, actions and observation gathering. Comparing students distilled from different sources on mathematics, ALFWorld and customer-service tool tasks, code-interpreter supervision raised customer-service accuracy from 41.5% to 51.0%, while ALFWorld supervision cut it to 6.6% and brought many tool calls issued before the user replied. Using only source-domain trajectories, the authors then selected supervised positions by the teacher's log-probability change before and after domain RL; keeping the top 95% raised customer-service accuracy from 6.6% to 68.9% at some cost to source-domain performance.

Source-provided article image: Agentic On-Policy Distillation Implicitly Reshapes LLMs' ReAct Patterns
arXiv

Interpretation

On-policy distillation changes more than task scores: it changes how an agent picks a solution route, adjusts actions after environment feedback, and waits for user-supplied information in multi-turn interaction. Earlier on-policy distillation work mainly tracked target-task accuracy, leaving unclear how interaction behavior itself shifts across domains. Training and cross-evaluating the same Qwen3-4B-Instruct-2507 students on mathematics, ALFWorld and customer-service tool tasks, code-interpreter supervision raised ALFWorld accuracy from 0.129 to 0.402 and customer-service accuracy from 0.415 to 0.510, while ALFWorld supervision lowered customer-service accuracy from 0.415 to 0.066.

The failure traces to a specific interaction timing: in one turn the student writes text requesting user information and also issues a tool call that depends on that information. The base model almost never shows this same-turn request-and-call pattern, and the ALFWorld teacher itself rarely does so on customer-service tasks. Among ALFWorld-student outputs containing request-like text, 94.3% also contained a tool call; semantic review found dependency violations in 55.8% of trajectories versus 0.50% for the base, covering 98 of the 100 customer-service tasks.

Source-domain trajectories alone can mitigate this negative transfer: rank positions by the teacher's log-probability change before and after domain RL, then drop the most negative tail. Ordinary on-policy distillation applies teacher supervision uniformly to all student-generated positions, so negative transfer enters the student with it. Using only ALFWorld trajectories, keeping the top 95% of positions raised customer-service accuracy from 6.6% to 68.9% while ALFWorld accuracy stayed at 0.834 versus 0.855 with full retention; randomly keeping the same fraction still harmed customer-service performance.

Perspective

The findings rest on Qwen3-4B-Instruct-2507 students, with teachers that include a code-interpreter teacher from the same 4B family and ALFWorld and customer-service teachers initialized from Qwen3-30B-A3B-Instruct-2507 and given domain-specific RL; they cover tool-free mathematical reasoning, code-interpreter mathematics, ALFWorld, and airline, retail and telecom customer-service tasks. Evaluation uses AIME2026 and BeyondAIME mathematics problems, 140 in-distribution and 134 out-of-distribution ALFWorld tasks, and 100 customer-service tasks. The authors note that extending the analysis to larger students, other model families and a broader range of agent environments would help establish how general these transfer patterns are.

The authors attribute negative transfer to severe overfitting to task-specific ReAct patterns, but that mechanism judgment comes from behavioral observation and position analysis and still needs controlled interventions to isolate the causes of cross-domain transfer. Selective supervision mitigates customer-service degradation while sacrificing some source-domain performance, and retention ratios and alternative ranking rules do not give a consistent source-target balance; the GR-to-code-interpreter extension preserves more tool use at similar mean GR accuracy but has lower overall code-interpreter accuracy.