SkillCT：面向 Agent 技能评估的多维基准与免执行基线
SkillCT: A Multidimensional Benchmark and Execution-Free Baseline for Agent Skill Assessment
AI 导读
arXiv 论文提出 SkillCT 基准，将 Agent 技能评估从下游任务执行扩展到九个维度的质量刻画，涵盖静态评估、跨骨干模型的动态评估和人工辅助评估。基于该基准，论文还提出免执行的质量评估基线，通过检索相似参考技能预测未见过技能的多维质量画像，降低 token 消耗与评估时间。SkillCT 引导的优化可提升下游任务成功率与执行效率。
正文 · 原文
Abstract:Agent skills package instructions, scripts, and resources for specialized tasks and are increasingly distributed through public Web repositories. However, existing skill benchmarks primarily focus on downstream task execution, insufficiently assessing specification quality, behavioral robustness, semantic alignment, and responsibility boundaries. Their reliance on task-specific tools, environments, and repeated executions limits scalability and pre-deployment assessment. To address these challenges, we introduce SkillCT, a benchmark that broadens skill assessment from downstream utility to a multidimensional characterization of skill quality. SkillCT evaluates skills across nine complementary dimensions spanning three aspects: Static Assessment, which captures intrinsic properties of the skill specification; Dynamic Assessment, which measures whether the induced behavior remains stable across backbone models and request variations; and Human-Assisted Assessment, which examines behavior-chain consistency and task granularity. These dimensions provide a broader view of skill quality than execution success. Applying SkillCT to a public skill corpus establishes a reference benchmark for analyzing quality weaknesses and relationships across dimensions. Building on this benchmark, we propose an execution-free quality evaluation baseline that predicts the multidimensional quality profiles of unseen skills by retrieving and leveraging similar reference skills, without executing the skills under evaluation. This reduces token consumption and evaluation time, while preserving consistency with the full assessment. Furthermore, SkillCT-guided optimization improves downstream task success and execution efficiency, demonstrating its value for skill refinement. These findings highlight the effectiveness of SkillCT in supporting scalable skill assessment, optimization, and repository maintenance.
| Subjects: | Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2610.38147 [cs.AI] |
| (or arXiv:2610.38147v1 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2610.38147 arXiv-issued DOI via DataCite |
Submission history
From: Mengyu Sun [view email]
[v1]
Wed, 28 Oct 2026 01:30:12 UTC (2,036 KB)
来源：arXiv cs.AI · arxiv.org