语言模型智能体系统的集合值路由：基于 WildChat 的基准与成本感知评估
Set-Valued Routing for Language-Model Agent Systems: A WildChat Benchmark and Cost-Aware Evaluation
AI 导读
arXiv 论文将语言模型智能体系统的固定目录路由形式化为集合值语言理解问题，构建了覆盖 12 个能力型智能体的 3000 条 WildChat 提示基准。微调文本编码器达到 89.59 F1，优于线性 SVM 的 71.79 和零样本 GPT-4o 的 41.51，在更难的语义组划分下仍保持 88.02 F1。
正文 · 原文
Abstract:Language-model agent systems must map natural-language requests to one or more specialized agents before planning or tool execution. This decision is often treated as top-1 selection, although compositional requests can require several capabilities and unnecessary dispatch raises cost. We formulate fixed-catalog routing as set-valued language understanding and construct a benchmark of 3,000 WildChat prompts over 12 capability-defined agents. Reproducible schema-guided reference sets are balanced by agent frequency and route cardinality. We compare profile retrieval, multilabel classifiers, a fine-tuned text encoder, and catalog-conditioned zero-shot GPT-4o. Evaluation combines set quality, latency, capability coverage, and ordinal-cost utility. The encoder attains 89.59 F1, compared with 71.79 for linear SVM and 41.51 for GPT-4o, and retains 88.02 F1 under a harder semantic-group split. We further introduce Weighted Agent Routing (WAR), a deterministic post-scoring rule that changes the quality-cost operating point without retraining; Encoder+WAR obtains the best constrained utility. The results show that catalog-specific supervision and explicit set construction are important for multi-agent dispatch, while cost-aware decision layers can complement strong language encoders.
| Subjects: | Machine Learning (cs.LG); Artificial Intelligence (cs.AI); Information Retrieval (cs.IR); Multiagent Systems (cs.MA) |
| Cite as: | arXiv:2606.28925 [cs.LG] |
| (or arXiv:2606.28925v3 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2606.28925 arXiv-issued DOI via DataCite |
Submission history
From: Ananto Nayan Bala [view email]
[v1]
Sat, 27 Jun 2026 13:59:46 UTC (1,181 KB)
[v2]
Sun, 12 Jul 2026 14:45:06 UTC (1,181 KB)
[v3]
Thu, 5 Nov 2026 16:02:27 UTC (763 KB)
来源：arXiv cs.AI · arxiv.org