WhoWins 受控研究：LLM Agent 指令冲突中位置、角色与标签谁起作用
Position, Role or Label? A Controlled Study of Instruction Conflicts in LLM Agents
AI 导读
arXiv 论文发布 WhoWins 基准，含 1,080 个代码评分场景和 7 组受控实验，测试 14 个开源权重模型（0.6B 到 122B，共 1,887,356 条评分回复）。
正文 · 原文
Abstract:LLM agents take instructions from the operator, repository configuration, user, tool results and saved memory, which often conflict. Instruction-hierarchy benchmarks ask whether the higher-ranked source wins, but a source sets position, role and label at once. First, we build WhoWins, 1,080 code-scored scenarios in five task types, each with up to five mutually exclusive instructions, and seven controlled experiments that vary only how instructions are delivered. Second, we study 14 open-weight models (0.6B to 122B, thinking off and on) over 1,887,356 scored replies, checking main findings on held-out scenarios. Inside one message the first of two conflicting lines wins 72.42 to 94.66%, and across messages it keeps 22.95 to 62.65%. A policy sent first in the system (or developer) message loses most conflicts with later lines in every Qwen model but wins most on gpt-oss when it acts. In tool and file-write tasks it wins less often than the same text sent first as a labelled user message, in every model-mode. Third, we test whether pairs forecast larger conflicts; forecasts improve in every model-mode once they know where the task line is. Finally, by our keyword patterns at most 2.92% of decided pair replies flag or hedge a conflict.
| Comments: | 55 pages, 8 figures, 24 tables. Code and data: this https URL (pip install whowins). Website: this https URL |
| Subjects: | Computation and Language (cs.CL); Artificial Intelligence (cs.AI); Software Engineering (cs.SE) |
| Cite as: | arXiv:2610.33201 [cs.CL] |
| (or arXiv:2610.33201v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2610.33201 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Anish Laddha [view email]
[v1]
Fri, 23 Oct 2026 18:48:27 UTC (522 KB)
来源：arXiv cs.AI · arxiv.org