Agentic RL stops leaning on scalar rewards alone: targeted guidance teaches tool formats and writing styles that plain policy gradients could not
Synopsis
Multi-turn tool-calling agentic reinforcement learning is often limited by the base model's coverage, so some rewards cannot be learned even when the reward function is well designed. Targeted guidance places the textual feedback attached to each reward criterion just before the offending agent turn, builds an expert that shares the policy's weights but sees that feedback in context, and distills toward it only on that turn while combining it with a per-criterion RL reward. In a Qwen3-Coder-30B software-engineering multi-turn setting, this combination taught a new semantic-search tool argument format and a per-turn poem requirement while keeping coding pass rate near that of GRPO training alone.
Interpretation
Agentic reinforcement learning can use the textual feedback that accompanies reward criteria to learn behaviors that policy gradients alone cannot move, without sacrificing pass rate on the primary task. Naive on-policy RL only incentivizes actions the policy already takes; if the actions satisfying a criterion lie outside the base policy's support, the policy will not improve even with a carefully chosen reward function. In a multi-turn software-engineering environment with a supervised-finetuned Qwen3-Coder-30B, introducing a new argument format for the semantic-search tool left the base model not using the format at all even with example usage in the prompt; with targeted guidance the model learned the format without regression on primary coding performance.
Each binary reward criterion also defines a textual feedback string; inserting it before the agent turn that failed yields an expert that shares the current policy's weights and differs only in context. Earlier on-policy self-distillation placed privileged information in a global context and applied distillation pressure to all turns, which can cause reasoning shortening or diversity collapse. The objective minimizes reverse KL between the live policy and that expert on failing turns and masks the loss on all other turns, while keeping a per-criterion RL reward; the distillation term is normalized across all tokens in a rollout to reduce the influence of long rollouts.
Stability comes from two choices: distilling only on turns that miss a criterion, and keeping the distillation weight in a lower range. At high distillation weight the style reward is higher early on, but entropy rises sharply and outputs lengthen, after which both pass rate and style criteria collapse; lower weights improve style more slowly without instability. On the out-of-distribution per-turn poem criterion, a self-distillation-only run without the matching reward did produce poems but was unstable and regressed coding pass rate, and a variant that applied the feedback on every turn regardless of success showed a similar regression.
Perspective
The demonstrated scope is post-training of Qwen3-Coder-30B in a single multi-turn software-engineering environment, with guidance targets centered on auxiliary criteria such as tool format and writing style rather than primary intelligence; each criterion requires a hand-designed static feedback string.
High distillation weight produces an entropy spike followed by failure on both pass rate and style criteria, while lower weights are stable but improve style more slowly; the authors note that many on-policy self-distillation variants were not compared head-to-head under the same experimental setting, and feedback strings currently rely on the policy's zero-shot understanding and human prompting.
