DERL：双演化强化学习实现自动化奖励设计
Dual-Evolving Reinforcement Learning for Automated Reward Design
AI 导读
研究者提出双演化强化学习框架 DERL，用于自动化奖励设计。该框架由外层提议者基于任务原语组合奖励，内层策略用该奖励训练，提议者再根据内层策略验证表现通过策略梯度学习，从而利用历史试错反馈优化奖励提议。DERL 在 ALFWorld、ScienceWorld、GSM8K、MATH 上验证，在多轮智能体基准上超越强基线，尤其在分布外任务上优势明显，数学任务中优于结果奖励。
正文 · 原文
Abstract:Crafting effective reward signals remains a central challenge in reinforcement learning (RL) for complex reasoning. Existing methods for automated reward optimization search with a fixed proposer which never learns from its costly trials. We present Dual-Evolving Reinforcement Learning (DERL), a dual-training framework for autonomous discovery of reward structures. An outer-loop proposer composes a reward from task primitives, and an inner policy trains with this reward. The proposer itself learns by policy gradient from the validation performance of the inner policy. Thus, DERL utilizes past-trial feedback to shape its reward-proposal distribution, an empirical meta-gradient of task success. We validate DERL on embodied (ALFWorld), scientific (ScienceWorld), and mathematical (GSM8K, MATH) reasoning. On multi-turn agent benchmarks, DERL outperforms strong baselines, significantly on out-of-distribution tasks. In mathematics, DERL improves on the outcome reward where directly added primitives hurt. Further analyses under matched compute show that our proposer effectively captures the meta-gradient to discover useful reward signals.
| Comments: | Homepage: this https URL. Code: this https URL. Models: this https URL |
| Subjects: | Artificial Intelligence (cs.AI); Computation and Language (cs.CL) |
| Cite as: | arXiv:2512.13399 [cs.AI] |
| (or arXiv:2512.13399v4 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2512.13399 arXiv-issued DOI via DataCite |
Submission history
From: Sitao Cheng [view email]
[v1]
Mon, 15 Dec 2025 14:50:08 UTC (583 KB)
[v2]
Wed, 13 May 2026 04:43:51 UTC (646 KB)
[v3]
Tue, 13 Oct 2026 21:12:48 UTC (1,275 KB)
[v4]
Tue, 27 Oct 2026 21:15:33 UTC (1,309 KB)
来源：arXiv cs.AI · arxiv.org