Learning to Undo:状态空间变换下的迁移强化学习研究
Learning to Undo: Transfer Reinforcement Learning under State Space Transformations
AI 导读
一项 NeurIPS'26 研究提出:当源 MDP 与目标 MDP 间存在 undo map(施加于目标状态空间可还原源状态空间)时,可通过匹配两 MDP 的状态特征统计量学习该映射,进而将源策略迁移到目标。
正文 · 原文
Abstract:Transfer learning in reinforcement learning (RL) has shown strong empirical success. In this work, we take a more principled perspective by studying when and how transferring knowledge between MDPs (a source and a target) can be provably beneficial. Specifically, we consider the case where there exists an undo map such that applying this map to the target's state space recovers the source exactly. We propose an algorithm that learns this map by matching state feature statistics gathered from both MDPs, and then uses it to transfer the source policy to the target. We theoretically justify the algorithm by analyzing the setting when the undo map is linear and the source is linearly-$Q^\star$ realizable, where our approach has strictly better sample complexity than tabula rasa RL in the target MDP. Empirically, we demonstrate that these benefits extend beyond this regime: on challenging continuous control tasks and Atari games, our method achieves significantly better sample efficiency. Overall, our results highlight how shared structure between tasks can be leveraged for efficient transfer of policies across environments.
| Comments: | NeurIPS'26 paper |
| Subjects: | Machine Learning (cs.LG); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2611.23081 [cs.LG] |
| (or arXiv:2611.23081v1 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2611.23081 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Mridul Mahajan [view email]
[v1]
Tue, 17 Nov 2026 17:06:51 UTC (2,471 KB)
来源:arXiv cs.AI · arxiv.org