面向零和 POSG 的保守策略迭代算法
Conservative Policy Iteration For Zero-Sum POSGs
AI 导读
arXiv 论文提出一种针对有限时域零和部分可观测随机博弈（zs-POSG）的近似策略迭代算法。该方法为每个玩家各维护一个策略并沿占用状态轨迹联合更新，每次更新仅用当前联合策略诱导的单个支撑超平面，而非逼近完整的中心值函数。在零和 POSG 基准上，该算法在小 horizon 下与 PBVI、HSVI、CFR+ 相当，较大 horizon 时往往改善运行时间与可利用性的权衡。
正文 · 原文
Abstract:Occupancy-state reductions make dynamic programming possible for zero-sum partially observable stochastic games (zs-POSGs). Existing point-based solvers usually optimise a central value function. This restores Bellman recursion, but it requires point-based updates over high-dimensional occupancy spaces. We introduce an approximate policy-iteration algorithm for finite-horizon zs-POSGs. The method maintains one policy per player and updates both jointly along occupancy trajectories induced by conservative policy updates. Instead of approximating a full central value function, each update uses a single supporting hyperplane induced by the current joint policy. The algorithm alternates local policy improvements, conservative updates, and persistent exploration. Computation is therefore concentrated on occupancy regions where the current joint policy is exploitable. On finite-horizon zero-sum POSG benchmarks, the method is competitive at small horizons and often improves the runtime-exploitability trade-off at larger horizons compared with PBVI, HSVI, and CFR+.
| Subjects: | Artificial Intelligence (cs.AI); Computer Science and Game Theory (cs.GT) |
| Cite as: | arXiv:2610.36847 [cs.AI] |
| (or arXiv:2610.36847v1 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2610.36847 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Erwan Escudie [view email]
[v1]
Tue, 27 Oct 2026 08:45:33 UTC (321 KB)
来源：arXiv cs.AI · arxiv.org