用超时级联降低编程智能体推理成本
Reducing Inference Cost of Coding Agents with Timeout Cascades
AI 导读
一项 arXiv 研究评估了 2024 年以来发布的 36 个开源大小模型在软件工程基准上的表现,发现小模型在简单和中等复杂度任务上准确率已接近大模型,差距集中在少数困难任务上。
正文 · 原文
Abstract:AI coding agents have become one of the most popular applications of large language models. Yet, agentic AI comes at the cost of a tremendous amount of GPU compute resources, which limits its profitability and its viability at scale. In this paper, we investigate whether smaller, cheaper models can cut this cost without sacrificing accuracy.
We evaluate 36 popular large and small open-weight models released since 2024 on a software engineering benchmark, and find that recent small models are closing the accuracy gap compared to their large counterparts. We investigate the structure of this gap and find that the gap is highly concentrated around hard tasks that small language models fail to solve while achieving remarkably good accuracy for easy and medium complexity tasks. Following this observation, we show that by utilizing a hybrid inference strategy that tries a small model first and escalates to a large one only when needed, we can achieve up to 33% reduction in total GPU-hour costs and 16.4% on all tasks from SWE-bench Verified, without sacrificing accuracy. Strikingly, this simple cascade nearly matches an oracle-style task router, highlighting a practical path for deploying cost-aware coding agents in production without complex learned task routers and with minimal engineering complexity.
| Subjects: | Artificial Intelligence (cs.AI); Software Engineering (cs.SE) |
| Cite as: | arXiv:2611.24324 [cs.AI] |
| (or arXiv:2611.24324v1 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2611.24324 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: German Sviridov [view email]
[v1]
Wed, 18 Nov 2026 12:26:42 UTC (180 KB)
来源:arXiv cs.AI · arxiv.org