arXiv cs.AI· Tingxu Han, Yi Zhang, Wei Song, Chunrong Fang, Zhenyu Chen, Youcheng Sun, Lijie Hu·· 4 小时前精选AI 评分60
SWE-Skills-Bench：评测 Agent Skills 在真实软件工程中的实际作用
SWE-Skills-Bench: Do Agent Skills Actually Help in Real-World Software Engineering?
AI 导读
研究者发布 swe-skills-bench，一个衡量单个 Agent Skill 对真实软件工程任务提升的基准，基于 98 个公开技能和固定 commit 的 GitHub 仓库，在 4 种智能体-模型配置下完成 3,920 组配对评测。
推荐理由
论文用 3920 组配对实验量化单个 Agent Skill 的实际收益，发现 98 个技能中 92 个无显著作用，可帮助理性评估技能引入方式。
正文 · 原文
Abstract:Agent skills, structured procedural knowledge packages injected at inference time, are increasingly used to augment LLM agents on software engineering tasks. However, their real utility in end-to-end development settings remains unclear, and no standard procedure tests an individual skill before installation. We present swe-skills-bench, a skill-centric, requirement-driven benchmark that measures how much individual agent skills improve performance on real-world software engineering (SWE) tasks. We start from 98 public SWE skills, each paired with a real GitHub repository pinned at a fixed commit. For each skill, we derive repository-specific requirements, create executable verifiers, and build a reproducible environment. Under each of four agent-model configurations, we run every task once without the skill and once with it, yielding 3,920 paired evaluations. For each requirement, we compute a paired delta, i.e., the change in outcome when the skill is added. We then aggregate these deltas across a skill's ten requirements into a per-skill utility profile, and use a statistical test to classify each skill as Golden (significantly helpful), Ineffective (no significant effect), or Harmful (significantly detrimental). Our results show that skills provide limited benefits: 92 of 98 skills have no statistically significant effect, and adding skills raises the average pass ratio only from 76.6% to 76.9%. The few helpful skills supply procedures that the agent would otherwise omit (gains of up to $+17.7$ points). In contrast, harmful skills anchor the agent on their own templates and override the task's explicit instructions (losses of up to -17.0 points). Our code is available at this https URL.
| Subjects: | Software Engineering (cs.SE); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2603.15401 [cs.SE] |
| (or arXiv:2603.15401v2 [cs.SE] for this version) | |
| https://doi.org/10.48550/arXiv.2603.15401 arXiv-issued DOI via DataCite |
Submission history
From: Tingxu Han [view email]
[v1]
Mon, 16 Mar 2026 15:16:31 UTC (1,343 KB)
[v2]
Tue, 27 Oct 2026 09:54:21 UTC (1,279 KB)
来源：arXiv cs.AI · arxiv.org