MoE-Context:只给模型看它需要的内容
MoE-Context: Show the Model Only What It Needs
AI 导读
arXiv 论文提出 MoE-Context 方法,把上下文分块,每步先用快速判断挑出模型所需的块,其余以单行笔记形式保留在共享 KV cache 中,隐藏而非删除。
正文 · 原文
Abstract:AI agents read their whole history at every step. People do not work like that: we keep what the moment needs and hold the rest as a rough note we can recall. MoE-Context applies this to language models. The context is split into blocks. Before each step a fast yes or no pass decides which blocks the step needs; the rest stay as one-line notes that can be brought back, and nothing is deleted. In the version we build, one model makes this decision and then answers, over a single key-value cache where hidden blocks are masked rather than removed. On a laptop with Qwen3-1.7B and a synthetic lookup task, showing only the needed block raises accuracy from 93 to 119 of 120, and a masked block is as hidden as a deleted one. The model's own picking is the weak part. A supervised linear probe written into spare output rows lifts its accuracy from 80 to 110 of 120, but simple keyword search picks perfectly on this task, and the probe does not transfer to a second task. We report these results openly, along with two cheap usage patterns that block switching enables: undoing a failed attempt and running subagents without a new context.
| Comments: | 10 pages, 10 figures |
| Subjects: | Artificial Intelligence (cs.AI); Computation and Language (cs.CL); Machine Learning (cs.LG) |
| Cite as: | arXiv:2611.24379 [cs.AI] |
| (or arXiv:2611.24379v1 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2611.24379 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Vedant Navale [view email]
[v1]
Wed, 18 Nov 2026 13:08:14 UTC (68 KB)
来源:arXiv cs.AI · arxiv.org