arXiv cs.AI· Sizhe Chen, Sanjay Kariyappa, Liwei Jiang, Chong Xiang, Jihye Choi, Chaowei Xiao, G. Edward Suh·· 2 小时前AI 评分45
从目标-目标冲突学习改进指令层级
Improving Instruction Hierarchy by Learning from Goal-Goal Conflicts
AI 导读
一项 arXiv 研究提出“目标-目标冲突”这一指令层级训练中缺失的冲突类型，现有 OpenAI 与 NVIDIA 的指令层级训练集均不包含该类型。在 Qwen3.5-9B 和 Gemma4-12B-it 上微调此类样本，可显著提升模型在训练未见的真实任务上的指令层级遵循能力，且在八个基准的普通任务上效用变化极小。
正文 · 原文
Abstract:The recent incident in which OpenAI agents autonomously compromised Hugging Face's infrastructure illustrates the importance of instruction hierarchy (IH): an LLM should follow lower-priority instructions, such as user requests, only when they do not conflict with higher-priority instructions, such as system policies. IH can be effectively improved by fine-tuning LLMs on simulated examples of instruction conflicts, whose diversity is key to teaching LLMs to properly handle conflicts at test time. However, our decomposition of a task into goals and constraints reveals a conflict type -- goal-goal conflicts -- that is absent from existing IH training sets from OpenAI and NVIDIA. We show the importance of this conflict type: fine-tuning Qwen3.5-9B and Gemma4-12B-it on examples of this conflict type significantly improves IH on a wide range of real-world tasks unseen in training; adding these examples to existing IH training sets for fine-tuning also gives better IH adherence. The IH gains come with trivial mean utility changes on no-conflict tasks across eight benchmarks. Our work advocates including this new conflict type in IH training, even if these examples do not reflect typical LLM use.
| Subjects: | Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2610.35093 [cs.AI] |
| (or arXiv:2610.35093v1 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2610.35093 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Sizhe Chen [view email]
[v1]
Mon, 26 Oct 2026 09:33:17 UTC (464 KB)
来源：arXiv cs.AI · arxiv.org