跳到正文
arXiv cs.AI· Xuan Zhang, Zhijian Zhou, Zicheng Hu, Yan Feng, Hongyan Hao, Yuan Qi, Chao Qu·· 56 分钟前AI 评分41

MCTS 在错误地计数:LPC-MCTS 用学习到的探索计数改进 LLM 推理

MCTS Is Counting the Wrong Thing: Learning Exploration Counts for LLM Reasoning

AI 导读

arXiv 论文提出 Learned Pseudo-Count MCTS(LPC-MCTS),指出标准 MCTS 中 UCT 对文本分支独立计数,导致同一推理策略的不同表述重复获得探索奖励、累积 regret 增加。

正文 · 原文

View PDF HTML (experimental)

Abstract:Monte Carlo tree search (MCTS) has powered landmark successes in reinforcement learning, yet its gains in LLM reasoning can remain modest despite substantial search effort. We identify an overlooked bottleneck: what MCTS counts. Standard UCT assigns independent visit counts to textual branches, allowing different formulations of the same reasoning strategy to repeatedly receive exploration bonuses. Our theoretical analysis shows that this count fragmentation can impair exploration efficiency, as quantified by increased cumulative regret. To address this problem, we introduce Learned Pseudo-Count MCTS (LPC-MCTS), which changes only the exploration statistic in UCT. A lightweight Coin-Flipping Network head on the policy's hidden representations estimates representation coverage, enabling related branches to share exploration evidence. Across 5 mathematical-reasoning and 7 search benchmarks, LPC-MCTS consistently outperforms strong MCTS baselines. Holding each LPC-MCTS-trained policy fixed, replacing standard MCTS counts at inference improves average mean@16 accuracy across 5 mathematical benchmarks by 3.79 percentage points on Qwen2.5-7B and 2.71 points on Qwen3-4B. Applying LPC-MCTS during both training and inference yields gains of 4.96 and 4.37 points over standard MCTS, respectively. Together, our theory and experiments show that changing what MCTS counts can substantially improve its exploration efficiency and reasoning performance.
Subjects: Artificial Intelligence (cs.AI)
Cite as: arXiv:2611.22301 [cs.AI]
  (or arXiv:2611.22301v1 [cs.AI] for this version)
  https://doi.org/10.48550/arXiv.2611.22301

arXiv-issued DOI via DataCite (pending registration)

Submission history

From: Xuan Zhang [view email]
[v1] Tue, 17 Nov 2026 06:57:40 UTC (1,619 KB)

来源:arXiv cs.AI · arxiv.org