让 LLM 相信提示词中的任意假事实:提示词条件化 SDF 研究
Can you make models believe false facts in the prompt?
AI 导读
研究提出提示词条件化 Synthetic Document Finetuning(SDF),对单个模型在覆盖数千个事实的合成文档上微调,使其相信提示词标签中注入的任意假事实,避免每套假信念都微调一个新模型。
正文 · 原文
Abstract:Making LLMs believe false facts is useful for alignment evaluations and for building model organisms. Previous work explored doing so with Synthetic Document Finetuning (SDF). But SDF requires fine-tuning one new model for each set of injected false beliefs. To enable larger scale SDF-powered alignment evaluations, we attempt to produce a single model that believes any fact placed in a <facts> tag in the prompt. To do so, we explore doing prompt-conditioned SDF: fine-tuning a model on synthetic documents covering thousands of facts, with each document conditioned on a fact presented in the prompt. We evaluate how deeply the model believes the facts provided in the prompt using evaluations from prior work and a new Jacobian lens evaluation. We find that the belief depth on unseen facts scales roughly log-linearly with the number of training facts, making this a cheap, promising avenue for making models believe false facts, though preliminary experiments on applying it to alignment audits on the small model we studied showed negative results due to these small models often already acting as if they believed facts provided in regular honeypot system prompts. Using RL to reduce the rate at which models verbalize following a false fact in the prompt, on top of prompt-conditioned SDF, generalizes to held-out evals and almost matches SDF on some evaluations. But training against one of our metrics for belief depth makes it harder to trust those evaluations.
| Subjects: | Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2611.23938 [cs.AI] |
| (or arXiv:2611.23938v1 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2611.23938 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Ananya Sutradhar [view email]
[v1]
Wed, 18 Nov 2026 06:26:12 UTC (505 KB)
来源:arXiv cs.AI · arxiv.org