RM-VLM：从自然语言任务描述到视觉接地的奖励机器
RM-VLM: From Natural-Language Task Descriptions to Vision-Grounded Reward Machines
AI 导读
论文提出 RM-VLM，用 LLM 从自然语言任务描述生成奖励机器（RM），并用视觉语言模型（VLM）充当标注函数，在图像观测上接地 RM 的命题，同时以 VLM 价值估计器生成稠密奖励信号。该方法针对传统 RM 需人工设计标注函数、只能处理非马尔可夫任务的问题。在网格世界导航、机器人操作和 3D 物体收集三个领域的实验中，RM-VLM 的任务成功率均高于现有基于 VLM 的奖励方法。
正文 · 原文
Abstract:Reward machines (RMs) have been proposed as a reward formulation in reinforcement learning (RL) to capture non-Markovian tasks where credit assignment depends on history. By representing reward functions as automata, RMs expose the task's temporal structure and allow such rewards to be handled within a Markovian framework. However, they assume a labelling function, a proposition-grounding oracle for observations, which must be manually engineered for each environment, limiting their wider applicability. In this work, we address this limitation by proposing RM-VLM, a novel approach in which a large language model (LLM) generates an RM from a natural-language task description, and a vision-language model (VLM) acts as the labelling function, grounding the RM's propositions over image-based observations during RM-guided policy learning. A VLM-based value function estimator is used to assess the agent's progress towards each RM transition and derive a dense reward signal. We evaluate RM-VLM on three domains covering non-Markovian grid-world navigation, robotic manipulation, and 3D object collection. Our experimental results show that RM-VLM consistently achieves higher task success rates than existing VLM-based reward methods across all tested domains.
| Comments: | 20 pages including appendix |
| Subjects: | Artificial Intelligence (cs.AI); Machine Learning (cs.LG) |
| Cite as: | arXiv:2611.08308 [cs.AI] |
| (or arXiv:2611.08308v1 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2611.08308 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Roko Parac [view email]
[v1]
Thu, 5 Nov 2026 17:10:10 UTC (759 KB)
来源：arXiv cs.AI · arxiv.org