arXiv cs.AI· Tianyang Luo, Tao Feng, Zhigang Hua, Yan Xie, Shuang Yang, Ge Liu, Jiaxuan You·· 4 小时前AI 评分43
MemReward：基于图的经验记忆助力 LLM 奖励预测，标签受限也能高效 RL 微调
MemReward: Graph-Based Experience Memory for LLM Reward Prediction with Limited Labels
AI 导读
MemReward 是一种基于图的经验记忆方法，在在线策略优化中把奖励传播到未标注 rollout：将查询、思考过程和答案存为异构图节点，用 GNN 预测未标注样本的奖励。
正文 · 原文
Abstract:Reinforcement learning (RL) fine-tuning improves large language model (LLM) reasoning using reward signals on sampled rollouts. When reward labels are available for only a subset of rollouts, restricting policy updates to that subset leaves the remaining rollouts unused for policy optimization. Semi-supervised learning propagates labels from labeled to unlabeled samples, but existing methods either diffuse labels with fixed similarity weights or treat each sample independently, ignoring the relations among queries, thinking processes, and answers. We propose MemReward, a graph-based experience memory that propagates rewards to unlabeled rollouts during online policy optimization. MemReward stores queries and the rollouts (thinking processes and final answers) of an initial LLM policy as nodes of a heterogeneous graph connected by similarity and structural edges, and warms up a GNN on labeled rollouts to propagate rewards over this graph. During online RL fine-tuning, unlabeled rollouts are attached to the graph by query similarity, and the policy is updated with ground-truth rewards on labeled rollouts and GNN-predicted rewards on the rest. We evaluate MemReward on 13 benchmarks in mathematics, question answering, and code generation with verifiable rewards, and further with LLM-judge, human-preference, and continuous rewards. With ground-truth rewards on only 20% of rollouts, MemReward achieves 96.6% of the performance of fully supervised training (Oracle) on Qwen2.5-1.5B and 97.3% on Qwen2.5-3B, where it also outperforms label propagation and FixMatch-style self-training. It carries over to Llama backbones up to 8B and closely approaches Oracle on out-of-domain tasks.
| Comments: | Accepted to NeurIPS 2026 |
| Subjects: | Machine Learning (cs.LG); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2603.19310 [cs.LG] |
| (or arXiv:2603.19310v5 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2603.19310 arXiv-issued DOI via DataCite |
Submission history
From: Tianyang Luo [view email]
[v1]
Fri, 13 Mar 2026 18:00:16 UTC (426 KB)
[v2]
Tue, 24 Mar 2026 05:09:14 UTC (370 KB)
[v3]
Thu, 21 May 2026 02:26:31 UTC (339 KB)
[v4]
Fri, 22 May 2026 03:50:49 UTC (325 KB)
[v5]
Wed, 4 Nov 2026 05:07:38 UTC (347 KB)
来源：arXiv cs.AI · arxiv.org