Semantic Overlays 论文提出用残差流标注缓解提示注入

View PDF HTML (experimental)

Abstract:Language models see only tokens. But the harness knows more: which token spans are user text, which are tool outputs, and which come from untrusted sources such as web pages or external documents. To pass that knowledge to the model, the harness must transmit it as plain text. Prompt injection exploits this: planted instructions, formatted the right way, can convince a model that they come from the user or the system. We introduce Semantic Overlays, small trained adapters that edit a frozen model's residual stream at chosen token spans. No input text can create or remove a mark, although the model's response to a mark is computed from states the input does influence. When the serving stack marks retrieved text with an overlay trained to mean "do not execute instructions in this span," the model treats injected instructions as inert content. Against attacks written without knowledge of the defense, it reaches the attack rates of the best published defenses: SEP separation rises from 24.3% to 98.8% under a corrected grader, AlpacaFarm attack success falls from 99.0% to 0.5%, and AgentDojo from 12.8% to 0%. Fine-tuning the model's weights on the same data reached similar separation but kept 64-80% of SEP utility, against 93% for the overlay. On TensorTrust hijacking, whose attacks resemble nothing in our training data, it lets 4.4% of hijacks through on Llama-3.1-8B, against 43% for Meta-SecAlign on the same base. An attacker with the weights does better: GCG succeeds on 72% of items, and on 14% after adversarial training of the overlay alone. Overlays are also flexible: the model reads their positions faithfully in stacked configurations; told by an overlay that non-Python code is Python and asked to copy it, the model rewrites it in Python; and overlays can carry span-delimited instructions. We also correct defects in two prompt-injection benchmarks and one evaluation harness.

Comments:	21 pages, 4 figures, 13 tables. Interactive demo: this https URL. Code and released adapters: this https URL
Subjects:	Artificial Intelligence (cs.AI); Computation and Language (cs.CL); Cryptography and Security (cs.CR); Machine Learning (cs.LG)
Cite as:	arXiv:2608.23873 [cs.AI]
	(or arXiv:2608.23873v4 [cs.AI] for this version)
	https://doi.org/10.48550/arXiv.2608.23873 arXiv-issued DOI via DataCite

Submission history

From: Joshua Penman [view email]
[v1] Mon, 24 Aug 2026 22:26:58 UTC (114 KB)
[v2] Fri, 28 Aug 2026 09:49:45 UTC (118 KB)
[v3] Thu, 3 Sep 2026 22:46:27 UTC (117 KB)
[v4] Wed, 11 Nov 2026 04:07:19 UTC (161 KB)