arXiv cs.AI· Thomson Yen, Gilbert Yang, Naimeng Ye, Minghui Chen, Dong Xia, Tianyi Shen, Yechan Jeon, Yaqin Chen, Daksh Mittal, Fangyu Wu, Hongseok Namkoong·· 2 小时前AI 评分32
Seq@K：在序贯反馈下评估大语言模型的新基准
Seq@K: Evaluating Language Models Under Sequential Feedback
AI 导读
研究者提出 Seq@K 评估协议，衡量模型能否在 K 次序贯尝试、每次失败后获得反馈的条件下解决问题，以捕捉 Pass@1、Pass@K 无法反映的反馈利用能力。该协议在 7 个基准、16 个前沿模型上评测，结果显示模型排名常与 Pass@K 不同，Pareto 前沿也出现更高效的配置。分析模型轨迹还可刻画其迭代修正行为，如响应反馈中问题的速率和对自身答案的固执程度。
正文 · 原文
Authors:Thomson Yen, Gilbert Yang, Naimeng Ye, Minghui Chen, Dong Xia, Tianyi Shen, Yechan Jeon, Yaqin Chen, Daksh Mittal, Fangyu Wu, Hongseok Namkoong
Abstract:Language-model agents differ fundamentally from prior intelligent systems in their ability to learn from natural language in context. This capability enables agents to iteratively refine their answers in response to feedback, a prevalent mode of operation in modern agentic workflows. However, the community primarily ranks models by metrics such as Pass@1 or Pass@K, which cannot capture how they utilize feedback on prior attempts. We introduce Seq@K, a natural evaluation protocol that measures whether a model can solve a problem within K sequential attempts, provided feedback after each previous failure. Unlike multi-turn agentic benchmarks that examine long-horizon planning, Seq@K isolates and formalizes the core capability of refinement from feedback. We evaluate Seq@K on a suite of 7 benchmarks and 16 frontier models, and show that Seq@K reveals a materially different model-selection landscape: under Seq@K, (1) model rankings often differ, (2) the Pareto frontiers admit more efficient configurations and (3) the gains from choosing the optimal model configuration can widen or shrink relative to those of Pass@K. We further show that analyzing model trajectories under Seq@K helps characterize iterative refinement behavior, such as the rate at which models address issues raised in feedback, or how rigidly they hold onto their own answers. Finally, we study how the nature of feedback affects models' iterative refinement, showing that practitioners must carefully design and benchmark the feedback loops of iterative agentic systems. Together, these results establish how Seq@K offers a more faithful metric for intelligent systems that typically operate in such feedback loop, and the necessity to consider Seq@K in evaluating models for these systems.
| Subjects: | Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2610.37859 [cs.AI] |
| (or arXiv:2610.37859v1 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2610.37859 arXiv-issued DOI via DataCite |
Submission history
From: Thomson Yen [view email]
[v1]
Tue, 27 Oct 2026 19:04:51 UTC (595 KB)
来源：arXiv cs.AI · arxiv.org