arXiv cs.AI· Hanze Li, Athan Z. Li, Feiyang Kang, Jiachen T. Wang, Yuhan Chen, Ruishan Liu, Ruoxi Jia·· 2 小时前AI 评分39
CaliMem：通过校准记忆实现搜索高效的数据智能体
CaliMem: Search-Efficient Data Agent via Calibrated Memory
AI 导读
CaliMem 是一种校准记忆的智能体循环，要求智能体在执行每个数据筛选策略前先预测结果，用预测误差对结论做"现实检验"，保留有依据的结论、修正错误并标注适用条件。在覆盖生物领域和 LLM 后训练的 55 个数据集-模型任务上，同一筛选配方可能帮助一个模型却损害另一个模型。CaliMem 持续超越固定筛选基线与已发表方法，最高超过全量数据训练 19.4%，代码已开源。
正文 · 原文
Abstract:Data curation is a first-order design choice in machine learning, yet tailoring training data to a downstream model and task remains costly in labor, expertise, and compute. Datasets are therefore often processed through fixed pipelines and reused across applications. We show that this assumption is fragile: across 55 dataset--model tasks spanning biological domains, the same curation recipe can help one model while hurting another. Effective curation must adapt to the downstream setting, motivating automated search over data recipes. We first build an agentic system that searches for curation policies at deployment, improving data utility via \textit{test-time curation (TTC)}. Although achieving performance gains, its search is driven largely by recent feedback and remains local. Adding persistent memory helps the agent retain evidence across iterations and reveals a deeper problem: it forms claims that extend beyond the conditions in which they were observed, sometimes abandoning an entire strategy after a single unsuccessful trial.
To address this problem, we introduce CaliMem, a calibrated-memory agentic loop that requires the agent to predict the outcome of each proposed policy before executing it. Comparing this prediction with the observed result gives the agent a \textit{reality check} on the claims behind its proposal. It uses the prediction error to retain supported claims, revise mistaken ones, and specify the conditions under which a conclusion holds. This makes the agent's beliefs testable, reduces overgeneralization, and supports more effective long-horizon exploration. Across biological and LLM post-training tasks, CaliMem consistently outperforms fixed curation baselines and published methods, and exceeds full-data training by up to 19.4\%. Code is open-sourced.
| Subjects: | Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2610.34500 [cs.AI] |
| (or arXiv:2610.34500v1 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2610.34500 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Hanze Li [view email]
[v1]
Sun, 25 Oct 2026 18:55:44 UTC (5,499 KB)
来源：arXiv cs.AI · arxiv.org