Whisper 明智筛选：ASR 微调的智能子集选择方法
Whispering Wisely: Smart Subset Selection for ASR Fine-Tuning
AI 导读
一项研究提出基于嵌入的子集选择方法来降低 Whisper 等 ASR 模型的微调成本。该方法利用 Whisper 预训练编码器将音频映射到嵌入空间，用贪心 k-center 算法选取多样化训练子集。在希伯来语和丹麦语上的评估显示，k-center 选择常优于随机采样，小样本量下收益最明显，且所选子集词汇多样性更高。
正文 · 原文
Abstract:Automatic Speech Recognition (ASR) models such as OpenAIs Whisper have substantially improved speech recognition quality but remain expensive to fine-tune, especially when large amounts of transcribed audio are available. This makes it challenging for many practitioners to adapt these models to new languages or domains. In this work, we introduce an embedding-based subset selection approach for selecting smaller training sets for ASR fine-tuning. Using Whispers pretrained encoder, we map each audio sample into a fixed-dimensional embedding space and apply a greedy k-center algorithm to select diverse and representative subsets of the training data. We evaluate this approach on both Hebrew and Danish, showing that k-center selection often improves upon random sampling, particularly at smaller subset sizes. An analysis of lexical diversity reveals that k-center subsets capture a broader range of linguistic content than random subsets of the same size, which may help explain their observed effectiveness. Overall, our results demonstrate that embedding-based subset selection provides a practical strategy for ASR subset selection in the two evaluated languages, with the clearest recognition benefits observed at smaller sample budgets.
| Comments: | 17 pages, 5 figures |
| Subjects: | Computation and Language (cs.CL); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2610.32997 [cs.CL] |
| (or arXiv:2610.32997v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2610.32997 arXiv-issued DOI via DataCite |
Submission history
From: Roi Yozevitch [view email]
[v1]
Mon, 5 Oct 2026 10:47:16 UTC (2,314 KB)
来源：arXiv cs.AI · arxiv.org