Skip to main content
Back to timeline
arXivSource publication:

RL post-training makes generative recommenders better rankers but worse retrievers: Office hit@500 falls from 52.84% to 47.69%

Synopsis

Generative recommenders are now commonly post-trained with reinforcement learning after supervised fine-tuning, but gains are usually reported on the first ten results. Exact full-catalogue ranking on five datasets shows that RL post-training raises top-of-list accuracy while lowering recall deeper in the candidate list: on Office, NDCG@10 rises from 11.59% to 12.55% while hit@500 falls from 52.84% to 47.69%. The cause is that RL pushes down every sampled candidate except the logged target, though those candidates are often what similar users choose next; one anchoring term outside the rollout restores recall to the supervised level while keeping the top-of-list gain.

Source-provided article image: Better Rankers, Worse Retrievers: The Hidden Cost of RL Post-Training in Generative Recommendation
arXiv

Interpretation

RL post-training makes a generative recommender more accurate at the top of the list but loses recall deep in the candidate list that deployment actually consumes, so the same model becomes a better ranker and a worse retriever. RL gains for these models were mainly reported on the first ten results, and benchmark studies evaluate at 10 or 32 at most, rarely measuring hit rates deeper in the list. Exact full-catalogue ranking for every test user on five datasets, averaged over three RL seeds: on Office, NDCG@10 rises from 11.59% to 12.55%, hit@50 falls from 25.73% to 24.22%, and hit@500 falls from 52.84% to 47.69%, with 22% of the test targets in the SFT model's top 500 no longer there.

The recall loss comes from the negative advantages RL assigns to non-target candidates inside the rollout, not from the KL-regularised objective itself. The exact solution of the KL-regularised RL problem cannot lower expected hit@ at any cut-off; the loss comes from how the update is estimated, since SFT only raises the logged item while GRPO also lowers every sampled non-target candidate. Keeping only positive advantages removes the recall loss and the top-of-list gain with it; keeping only negative advantages lowers hit@500 by 19 to 21 points, three times the unmodified run on Office and more than ten times on Industrial; a per-item rollout record computed from the SFT model before RL identifies in advance the test users who lose the most recall.

Adding one anchoring term outside the rollout keeps the top-of-list gain and restores recall to the supervised fine-tuning level. The KL term of GRPO is also estimated on the rollout, so it does not constrain items outside it, which is where the recall loss occurs. On Office the anchor recovers 4.4 of the 5.1 hit@500 points lost, reaching 52.14% against 47.69% for GRPO and 52.84% for SFT; on the other four datasets hit@500 returns to the SFT level or slightly above, and NDCG@10 does not differ significantly from GRPO on all five.

Perspective

The results cover five public datasets with a Qwen2.5-1.5B-Instruct backbone, the MiniOneRec GRPO configuration, and evaluation after one epoch of training per model; Office and Industrial use MiniOneRec's files and semantic IDs unchanged, while Beauty, Toys and Sports build semantic IDs with the same pipeline and are evaluated on 5,000 sampled users each. The authors propose measuring the trade-off next at larger catalogues and models and with the downstream ranker in the loop.

The authors note that RL hyper-parameters were not tuned for Beauty, Toys and Sports, so RL is probably not at its best there, though the trend is the same on all five datasets; the anchor constrains only the relative order of items outside the rollout, not the order among rollout candidates or absolute ranks; and RL itself still has no learning signal for targets outside the rollout.