arXiv cs.AI· Jinbae Seo, Jungin Park, Hyunsung Jang, Namkoo Ha, Jiyoung Lee, Kwanghoon Sohn·· 1 小时前AI 评分34
COTEX:面向多模态 Agent 的持续工具空间扩展学习
Continual Tool-Space Extension Learning for Multimodal Agents
AI 导读
一项研究提出 COTEX,让多模态 Agent 以 adapter 级别的开销持续接入新的专业工具。该方法定位两组参数方向,分别控制工具调用与工具选择,并通过正交约束保证新工具不影响既有工具的选择;同时提出将专业模型的任务标注转化为已验证工具使用轨迹的协议。在两个 Agent 和三个专业工具的实验中,该方法在十个视觉语言 benchmark 上取得提升,既有工具最多仅下降 0.3 分。
正文 · 原文
Abstract:Multimodal agents largely rely on generic tools such as cropping, search, and code execution, leaving specialist vision models outside their tool space. However, bringing such a model into an already trained agent is not straightforward. The challenge is to learn each new tool efficiently while retaining the ability to use existing ones. This work proposes COTEX, continual tool-space extension learning, that adds new tools at efficient adapter-level cost. COTEX locates two sets of parameter directions; one triggers a tool call and thereby opens a multi-turn reasoning trajectory, while the other determines which tool is chosen. The adapter of a new tool is restricted to the calling directions that leave earlier tool choices unaffected by orthogonal constraint. We further introduce a protocol that converts specialist models' existing task-specific annotations into verified tool-use trajectories. Experiments on two representative agents and three specialist tools show gains on ten vision-language benchmarks, with at most a 0.3-point drop on existing tools.
| Subjects: | Artificial Intelligence (cs.AI); Machine Learning (cs.LG) |
| Cite as: | arXiv:2611.23810 [cs.AI] |
| (or arXiv:2611.23810v1 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2611.23810 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Jinbae Seo [view email]
[v1]
Wed, 18 Nov 2026 04:14:02 UTC (12,044 KB)
来源:arXiv cs.AI · arxiv.org