arXiv cs.AI· Kaiqu Liang, Jaime Fern\'andez Fisac, Thomas L. Griffiths, Andrew Lampinen·· 4 小时前精选AI 评分69
arXiv 论文提出记忆陷阱：语言智能体会从良性交互中过度泛化
The Memory Trap: Language Agents Overgeneralize from Benign Interactions
AI 导读
arXiv 论文提出记忆陷阱概念，指出语言智能体从良性交互中持久化记忆后，会在后续任务上出现不安全或不可靠行为。
推荐理由
论文指出良性交互留下的记忆会让语言智能体在高风险任务上过度泛化，为智能体记忆系统设计提供了具体失效模式。
正文 · 原文
Abstract:Language agents increasingly use persistent memory to record user preferences and procedures across sessions. We show that memory from benign interactions can cause unsafe or unreliable behavior on later tasks. We call this failure the memory trap. To study this issue, we compare behavior in sessions with and without memory across six frontier models (including Claude Sonnet, Opus, and Fable models and GPT-5.6). We evaluate these models in a range of settings. First, in autonomy experiments, we find that requests to skip confirmation during low-stakes coding chores lead agents to skip confirmation and verification on simulated high-stakes tasks in both coding and everyday life. We next study memories formed when users seek emotional support, or describe harm from uncritical feedback, and find these memories bias later judgments of mathematical work, arguments, and poems toward sycophancy or over-critique (respectively). Memory notes written during legitimate stale-test repairs lead agents to weaken correct tests that catch broken safeguards; these memories also increase cheating on ImpossibleBench, with larger increases for stronger Sonnet and Opus models. We show that memory traps can arise both from overgeneralizations in memory writing, and when acting on memory. These findings demonstrate that even benign interactions can lead to serious risks from memory systems, and thus motivate the need for new research into ensuring that agents preserve the scope of their memories.
| Subjects: | Computation and Language (cs.CL); Artificial Intelligence (cs.AI); Machine Learning (cs.LG) |
| Cite as: | arXiv:2611.07554 [cs.CL] |
| (or arXiv:2611.07554v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2611.07554 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Kaiqu Liang [view email]
[v1]
Thu, 5 Nov 2026 07:01:59 UTC (492 KB)
来源：arXiv cs.AI · arxiv.org