Q-MMR：基于递归重加权与矩匹配的离线策略评估方法
Q-MMR: Off-Policy Evaluation via Recursive Reweighting and Moment Matching
AI 导读
Q-MMR 是面向有限时域 MDP 的离线策略评估理论框架，为每条数据学习一个标量权重，使重加权奖励逼近目标策略的期望回报，权重通过对价值函数判别器类的矩匹配目标自顶向下归纳学习。在仅需 Q^π 可实现的条件下，可建立数据相关的有限样本保证，且误差界与函数类的统计复杂度无关。该方法还与重要性采样、线性 FQE 等已有方法建立联系，并对 offline RL 中覆盖性的本质给出新分析。
正文 · 原文
Abstract:We present a novel theoretical framework, Q-MMR, for off-policy evaluation in finite-horizon MDPs. Q-MMR learns a set of scalar weights, one for each data point, such that the reweighted rewards approximate the expected return under the target policy. The weights are learned inductively in a top-down manner via a moment matching objective against a value-function discriminator class. Notably, and perhaps surprisingly, a data-dependent finite-sample guarantee for general function approximation can be established under only the realizability of $Q^\pi$, with a dimension-free bound -- that is, the error does not depend on the statistical complexity of the function class. We also establish connections to several existing methods, such as importance sampling and linear FQE. Further theoretical analyses shed new light on the nature of coverage, a concept of fundamental importance to offline RL.
| Subjects: | Machine Learning (cs.LG); Artificial Intelligence (cs.AI); Machine Learning (stat.ML) |
| Cite as: | arXiv:2605.06474 [cs.LG] |
| (or arXiv:2605.06474v3 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2605.06474 arXiv-issued DOI via DataCite |
Submission history
From: Xiang Li [view email]
[v1]
Thu, 7 May 2026 16:00:04 UTC (122 KB)
[v2]
Fri, 8 May 2026 12:50:37 UTC (122 KB)
[v3]
Fri, 23 Oct 2026 17:57:34 UTC (128 KB)
来源：arXiv cs.AI · arxiv.org