跳到正文
arXiv cs.AI· Yuheng Chen, Fan Zhang, Yixuan Xu, Chao Yang, Yang Xiang, Can Yang·· 2 小时前AI 评分51

RL 后训练让生成式推荐器排序更准但召回变差,arXiv 论文提出 out-of-rollout anchoring 修复

Better Rankers, Worse Retrievers: The Hidden Cost of RL Post-Training in Generative Recommendation

AI 导读

arXiv 论文(arXiv:2611.22838)系统研究生成式推荐中 RL 后训练的隐性代价:在全部五个数据集上,RL 提升 small K 的 hit@K 却降低数百 K 的候选列表召回,模型成为更好的排序器但更差的检索器。

正文 · 原文

View PDF HTML (experimental)

Abstract:Generative recommenders decode the semantic ID of a user's next item with a language model, and post-training them with reinforcement learning (RL) after supervised fine-tuning (SFT) has become standard practice. The gains are mostly reported on the first ten results, yet in deployment such models hand hundreds of candidates to a downstream ranker or reward model. We show that RL post-training has a hidden cost: it improves accuracy at the top of the list (hit@K for small K) but, on all five datasets we study, lowers the recall of the candidate list (hit@K for K in the hundreds). The model becomes a better ranker and a worse retriever. Controlled experiments trace the loss to the negative advantages that RL assigns to the candidates it samples, the rollout. The root cause is that only one next item is logged per user, so RL treats every other candidate as wrong. Yet these candidates are the model's most plausible alternatives, and often the next items of other users with similar histories. Penalising them pushes down exactly what similar users choose next, and these items drop out of those users' candidate lists. This diagnosis points directly to a remedy: out-of-rollout anchoring, a single additional loss term that preserves the SFT model's ordering of the items outside the rollout. It keeps the gain at the top and restores recall to the SFT level on all five datasets, and it is effective under other RL configurations as well. To our knowledge, this is the first systematic study of this recall loss: we measure it, identify its cause, and show that, once the cause is known, a simple remedy suffices.
Subjects: Information Retrieval (cs.IR); Artificial Intelligence (cs.AI)
Cite as: arXiv:2611.22838 [cs.IR]
  (or arXiv:2611.22838v1 [cs.IR] for this version)
  https://doi.org/10.48550/arXiv.2611.22838

arXiv-issued DOI via DataCite (pending registration)

Submission history

From: Yuheng Chen [view email]
[v1] Tue, 17 Nov 2026 14:19:22 UTC (1,194 KB)

来源:arXiv cs.AI · arxiv.org