跳到正文
arXiv cs.AI· Sergio Hern\'andez-Guti\'errez, Matteo Merler, Ilze Amanda Auzina, Joschka Str\"uber, Ameya Prabhu, Matthias Bethge·· 4 小时前AI 评分46

QVal:评估长程 LLM Agent 稠密监督信号的无训练测试床

QVal: Evaluating Dense Supervision Signals for Long-Horizon LLM Agents

AI 导读

arXiv 论文提出 QVal,一个免训练测试床,通过衡量稠密监督信号与参考策略回报的 Q-alignment 来评估 LLM Agent 中间步骤信号质量。

正文 · 原文

View PDF HTML (experimental)

Abstract:LLM agents increasingly act over long horizons, where outcome-only rewards say little about which intermediate actions were good. Dense supervision methods address this by scoring intermediate steps, but they are usually evaluated through the downstream performance of a full training pipeline. This is expensive, confounds signal quality with other training choices, and means that methods are rarely compared on common ground. We introduce QVal, a training-free testbed that measures the $Q$-alignment of a dense signal: whether it orders the candidate actions in each state the same way as their returns under a reference policy do. We benchmark 21 methods from seven families with six open-weight backbones in four environments. Simple prompting baselines are the most $Q$-aligned on average, and performance clusters strongly by family. Code-generated signals do well only in small state and action spaces; embedding-based signals are weak; self-distillation signals are anti-aligned; and no method is $Q$-aligned in open-ended environments. When five of these signals guide an actor at test time by choosing among four proposed actions, $Q$-alignment strongly correlates with the gain in task success. QVal is designed to be easily extensible to new environments and methods, enabling researchers to compare new signals cheaply before committing to a training run.
Comments: 10 pages, 6 figures in main text; 67 pages, 13 figures with appendix
Subjects: Machine Learning (cs.LG); Artificial Intelligence (cs.AI); Computation and Language (cs.CL)
Cite as: arXiv:2606.32034 [cs.LG]
  (or arXiv:2606.32034v2 [cs.LG] for this version)
  https://doi.org/10.48550/arXiv.2606.32034

arXiv-issued DOI via DataCite

Submission history

From: Sergio Hernández-Gutiérrez [view email]
[v1] Tue, 30 Jun 2026 17:58:23 UTC (2,071 KB)
[v2] Fri, 13 Nov 2026 17:57:08 UTC (2,310 KB)

来源:arXiv cs.AI · arxiv.org