GENSTRAT：迈向大语言模型策略推理的评测科学
GENSTRAT: Toward a Science of Strategic Reasoning in Large Language Models
AI 导读
研究者推出 GENSTRAT，通过程序化生成二人零和不完全信息博弈游戏，按需产生新题目来评测 LLM 的策略推理能力，避免固定评测集饱和与数据污染。每个游戏按五个复杂度维度刻画，团队构建了 50 个游戏的基准，对十二个前沿及开源权重模型进行锦标赛，收集超 40,000 场 LLM 对战数据。
正文 · 原文
Abstract:Fixed game suites for evaluating strategic reasoning in large language models (LLMs) saturate and are hard to keep free of contamination. We introduce GENSTRAT, a system for evaluating the strategic capabilities of LLMs driven by procedural generation of two-player zero-sum imperfect-information betting games, allowing an evaluator to draw novel, fresh games on demand. Each game is characterized on a scale of five complexity axes (state space, temporal contingency, opponent dependence, private information, precision) and measured by Monte Carlo simulation. Using this generator, we construct a 50-game benchmark and hold a tournament of twelve frontier and open-weight models, gathering data from over 40,000 LLM vs. LLM matches. We further construct a composite complexity axis, and find that, on the eight hardest games, the gap between the best and worst model is 1.7x its value across the whole set of games. Ablations show that a higher thinking budget, disclosing the opponent's strategy, and higher reasoning matched to a game's specific demands (for private information and planning depth) were associated with higher margins for the LLM player.
| Comments: | 33 pages, 8 figures, 17 tables (3 figures, 2 tables in main paper) |
| Subjects: | Artificial Intelligence (cs.AI); Computer Science and Game Theory (cs.GT); Machine Learning (cs.LG); Multiagent Systems (cs.MA) |
| Cite as: | arXiv:2605.23238 [cs.AI] |
| (or arXiv:2605.23238v2 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2605.23238 arXiv-issued DOI via DataCite |
Submission history
From: Vartan Shadarevian [view email]
[v1]
Fri, 22 May 2026 05:13:45 UTC (1,940 KB)
[v2]
Thu, 29 Oct 2026 23:30:39 UTC (1,122 KB)
来源：arXiv cs.AI · arxiv.org