arXiv cs.AI· Rui Jin, Jikai Chen, Chengyue Yu, Yihan Chen, Hao Zhou, Demin Zhu, Kaichen Yang, Dong Wang, Linjian Mo, Chenyi Zhuang·· 5 小时前AI 评分44
设计反馈而不只是奖励：面向开放式强化学习的训练环境研究
Design the Feedback, Not Just the Reward: Environments for Open-Ended Reinforcement Learning
AI 导读
arXiv 论文提出一个面向开放式任务的强化学习框架，用 rubric 编码人类领域知识锚定评估意图，并结合模型推理与分层奖励定位失败区域，再通过 token 级权重指导 GRPO 优化。
正文 · 原文
Abstract:Reinforcement learning has advanced large language models on tasks with reliable verifiers, where checking an outcome provides a clear basis for reward. Extending this approach to open-ended tasks requires environments that can both evaluate complex, partially successful outputs and turn these judgments into effective learning guidance. We propose a reinforcement learning framework built around a training environment that integrates human domain knowledge with model reasoning. Human domain knowledge, encoded in rubrics, anchors evaluation to task intent, while model reasoning combined with hierarchical rewards distinguishes complex open-ended outputs and links diagnosed failures to likely responsible output regions. Furthermore, these diagnostic attributions guide token-level weights in the GRPO objective, thereby translating the ambiguous notion of success in open-ended tasks into concrete guidance for policy optimization. We instantiate this framework in interactive web application generation, a representative open-ended generation task. The resulting model achieves an average pass rate of 41.25\% on MiniAppBench and 76.19\% on ArtifactsBench. Ablation studies show the contributions of individual components, while additional out-of-distribution evaluations support transferable gains without degrading performance on other evaluated tasks.
| Subjects: | Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2608.27906 [cs.AI] |
| (or arXiv:2608.27906v4 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2608.27906 arXiv-issued DOI via DataCite |
Submission history
From: Jikai Chen [view email]
[v1]
Fri, 28 Aug 2026 04:17:05 UTC (5,401 KB)
[v2]
Mon, 31 Aug 2026 09:54:20 UTC (5,401 KB)
[v3]
Thu, 15 Oct 2026 06:11:14 UTC (18,254 KB)
[v4]
Thu, 29 Oct 2026 10:02:25 UTC (18,254 KB)
来源：arXiv cs.AI · arxiv.org