潜在世界模型中的评论家引导合成经验生成方法 CGSE
Generating Critic-Guided Synthetic Experience in Latent World Models
AI 导读
研究者提出评论家引导合成经验（CGSE）方法，把世界模型当作局部转移生成器，在离线策略采样的候选动作上用 Q 函数一阶近似在信任域内迭代优化，再用冻结的 latent world model 想象未来状态并由 critic 挑选最优转移加入 replay buffer。该方法无需额外真实环境交互或测试时规划器。在三个 OGBench 操作任务上，CGSE 使离线策略平均成功率先对提升 7.1%。
正文 · 原文
Abstract:Learning from offline datasets is limited in scalability, as collecting new experience is often costly or unsafe in the real world. World models offer an alternative way to generate new experience through a learned dynamics model, yet learning policies in latent space is limited due to compounding errors in imagined rollouts. To this end, we propose using the world model as a local transition generator, leveraging critic guidance to choose actions inside that model to generate synthetic experiences. We study action refinement methodologies, introducing Critic-Guided Synthetic Experience (CGSE) generation in world models. Specifically, CGSE augments an existing offline dataset by sampling candidate actions using an offline policy and refining each to maximize a first-order approximation of the $Q$-function within a trust region. We then use a frozen latent world model to imagine the future state given each updated action, and use the critic to choose the best one, appending this synthetic transition to the replay buffer. Our method requires no additional real-environment interaction or test-time planner, as it generates synthetic experience offline using the world model. On three OGBench manipulation tasks, CGSE improves an offline policy by a relative improvement of $7.1\%$ on the average success rate.
| Subjects: | Machine Learning (cs.LG); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2610.39100 [cs.LG] |
| (or arXiv:2610.39100v1 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2610.39100 arXiv-issued DOI via DataCite |
Submission history
From: Jorge De Freitas [view email]
[v1]
Thu, 29 Oct 2026 00:41:38 UTC (646 KB)
来源：arXiv cs.AI · arxiv.org