有限模型不确定性下的 Q-learning：arXiv 论文提出鲁棒 Q-learning 算法
Q-Learning under Finite Model Uncertainty
AI 导读
arXiv 论文提出面向马尔可夫决策过程的鲁棒 Q-learning 算法，为每个状态-动作对关联一个由候选转移核组成的有限模糊集，支持竞争性拟合模型和压力情景等自定义不确定性建模。
正文 · 原文
Abstract:We propose a robust Q-learning algorithm for Markov decision processes under model uncertainty when each state--action pair is associated with a finite ambiguity set of candidate transition kernels. This finite-measure framework enables user-designed uncertainty models, including competing fitted models and stress scenarios. {The learner observes successor states and their deterministic rewards from every candidate simulator, without requiring explicit knowledge of the transition probabilities or the reward function.} We establish almost sure convergence of the learned Q-function to the robust optimum, and derive a time-uniform high-probability error certificate that separates an observable empirical Bellman residual from transition-kernel estimation error. We also give quantitative finite-ambiguity approximation results for Wasserstein and parametric uncertainty models on finite state spaces. Numerical examples illustrate the effect of scenario selection and compare the tabular algorithm with robust value iteration.
| Subjects: | Optimization and Control (math.OC); Artificial Intelligence (cs.AI); Machine Learning (cs.LG); Probability (math.PR) |
| Cite as: | arXiv:2407.04259 [math.OC] |
| (or arXiv:2407.04259v4 [math.OC] for this version) | |
| https://doi.org/10.48550/arXiv.2407.04259 arXiv-issued DOI via DataCite |
Submission history
From: Julian Sester [view email]
[v1]
Fri, 5 Jul 2024 05:19:36 UTC (49 KB)
[v2]
Sun, 16 Feb 2025 03:16:16 UTC (53 KB)
[v3]
Mon, 9 Feb 2026 05:38:32 UTC (348 KB)
[v4]
Thu, 5 Nov 2026 07:58:02 UTC (245 KB)
来源：arXiv cs.AI · arxiv.org