arXiv cs.AI· Julian Skirzynski, Harry Cheon, Shreyas Kadekodi, Meredith Stewart, Berk Ustun·· 2 小时前AI 评分27
概念瓶颈模型评测新工具:面向可解释 AI 的合成基准(NeurIPS 2026)
Measuring What Matters: Synthetic Benchmarks for Concept Bottleneck Models
AI 导读
研究者为概念瓶颈模型构建了合成基准,可生成带概念标签的数据集,并控制数据模态、概念选择和标注质量等影响性能的因素。该基准用于评估概念瓶颈模型的两大用例——辅助人类专家决策和无人监督的自动化任务,能识别新的失败模式,并将其归因于模型架构或数据本身。基准已在 NeurIPS 2026 评测与数据集赛道发表并公开提供。
正文 · 原文
Abstract:Concept bottleneck models predict using high-level concepts detected in inputs. Although these models provide a simple way to reap benefits from interpretability, they are hard to train and evaluate because very few real-world datasets include concept labels. In this paper, we develop synthetic benchmarks for concept bottleneck models. With our benchmarks, researchers can generate labeled datasets and control properties that affect performance, such as data modality, the choice of concepts, or annotation quality. They can thus find problems that suit these models and isolate the factors behind their failures. We demonstrate how our benchmarks can be used to evaluate major use cases for concept bottleneck models: decision support, i.e., helping human experts make better decisions; and automation, i.e., handling routine tasks without supervision. Our demonstrations show how the benchmarks can identify new failure modes and attribute them to either the architecture or the data.
| Comments: | Camera-ready version, NeurIPS 2026 (Evaluations and Datasets Track). Benchmarks available at this https URL |
| Subjects: | Machine Learning (cs.LG); Artificial Intelligence (cs.AI) |
| ACM classes: | I.2.6 |
| Cite as: | arXiv:2606.04326 [cs.LG] |
| (or arXiv:2606.04326v2 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2606.04326 arXiv-issued DOI via DataCite |
Submission history
From: Julian Skirzynski [view email]
[v1]
Wed, 3 Jun 2026 01:01:05 UTC (1,311 KB)
[v2]
Tue, 17 Nov 2026 19:59:15 UTC (370 KB)
来源:arXiv cs.AI · arxiv.org