DynaRev：动态目标修订下的 Web Agent 基准测试
DynaRev: Benchmarking Web Agents under Dynamic Goal Revision
AI 导读
研究者发布 DynaRev 基准，包含 181 个任务，用于评估 Web Agent 在用户目标动态修订场景下的表现。该基准通过语义里程碑在任务定义的阶段下发修订，覆盖增量修订、无条件覆盖和运行时条件覆盖三类修订，并以轨迹为依据评估正确性、响应性和效率。对五个模型的评测显示，智能体常只完成部分任务要求，且保留浏览器状态的指令改写带来收益，而重置浏览器则造成损失。
正文 · 原文
Abstract:Web agents are typically evaluated with goals fully specified before execution and fixed thereafter, whereas real users often revise their requirements during interaction. To study such interactions, we introduce DynaRev, a benchmark of 181 tasks to evaluate web agents under dynamic goal revision. DynaRev uses semantic milestones to deliver revisions at task-defined stages, covers additive revisions, unconditional overrides, and runtime-conditioned overrides, and evaluates correctness, responsiveness, and efficiency through trajectory-grounded evaluation. Five-model evaluations show that agents often satisfy only part of the task requirements, with performance varying across task sources, revision types, and models. Further analyses reveal varied milestone timing, mixed effects of dynamic delivery, and contrasting revision-handling outcomes, with gains from instruction rewriting while preserving browser state and losses from browser resets. DynaRev provides a testbed for studying how web agents adapt to evolving user goals. The code and data are available at this https URL.
| Subjects: | Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2610.40133 [cs.AI] |
| (or arXiv:2610.40133v1 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2610.40133 arXiv-issued DOI via DataCite |
Submission history
From: Shiqi Zhang [view email]
[v1]
Thu, 29 Oct 2026 14:25:54 UTC (972 KB)
来源：arXiv cs.AI · arxiv.org