arXiv cs.AI· Qiaoman Zhang, Shunhua Min, Xiang He, Zhenghan Zhang, Yuting Zhang, Guangxian Li, Yunzhu Kuang, Rui Jiang, Jingyan Zhang, Chaoyue Li, Jiayu Ding·· 3 小时前AI 评分41
TIDE：基于熵门控有界偏差的 token 级信用分配
TIDE: Token-Level Credit Assignment via Entropy-Gated Bounded Deviation
AI 导读
论文提出 TIDE（Token-level Importance via Deviation and Entropy），一种无 critic 的强化学习框架，用有界 Hellinger 距离替代反向 KL 散度，并通过熵门控将序列级优势重新分配到 token 级。
正文 · 原文
Authors:Qiaoman Zhang, Shunhua Min, Xiang He, Zhenghan Zhang, Yuting Zhang, Guangxian Li, Yunzhu Kuang, Rui Jiang, Jingyan Zhang, Chaoyue Li, Jiayu Ding
Abstract:Reinforcement learning (RL) is central for aligning large language models (LLMs) with complex reasoning, yet the leading critic-free algorithm, Group Relative Policy Optimization (GRPO), broadcasts a single sequence-level advantage uniformly across long Chain-of-Thought (CoT) generations and constrains exploration with an unbounded Kullback-Leibler (KL) penalty, which dilutes token-level supervision and induces gradient instability and mode-seeking conservatism. We propose TIDE (Token-level Importance via Deviation and Entropy), a critic-free RL framework that reinterprets distribution deviation as a guiding signal rather than a rigid penalty. TIDE replaces the volatile reverse KL divergence with the bounded Hellinger distance, safely quantifying token-level exploration without risk of gradient explosion. To separate genuine reasoning breakthroughs from confident hallucinations, an entropy gating mechanism scales this deviation by the policy's predictive uncertainty, and the gated scores redistribute the sequence-level advantage across tokens. The resulting updates amplify exploratory steps while suppressing low-entropy deviations, without a value network and with negligible overhead. Across in-domain, held-out, and out-of-domain benchmarks, TIDE outperforms strong critic-free baselines: on the Qwen2.5-32B backbone it reaches 60.0% and 46.0% Avg@32 accuracy on AIME 2024 and AIME 2025, respectively, above competitive baselines such as DAPO. Token-level attribution, causal perturbation, and exploration analyses further support the interpretation that the entropy-gated deviation identifies pivotal reasoning steps rather than merely perturbing optimization dynamics.
| Subjects: | Machine Learning (cs.LG); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2605.03327 [cs.LG] |
| (or arXiv:2605.03327v3 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2605.03327 arXiv-issued DOI via DataCite |
Submission history
From: Hongbo Jin [view email]
[v1]
Tue, 5 May 2026 03:36:27 UTC (2,315 KB)
[v2]
Fri, 8 May 2026 12:51:27 UTC (2,314 KB)
[v3]
Thu, 12 Nov 2026 06:41:56 UTC (2,112 KB)
来源：arXiv cs.AI · arxiv.org