MG-REPA：面向生成式目标说话人提取的多粒度表征对齐方法
MG-REPA: Multi-Granularity Representation Alignment for Generative Target Speaker Extraction
AI 导读
提出多粒度表征对齐方法 MG-REPA，在帧级与 SSL 模型、语句级与说话人验证模型分别对齐隐状态，并通过对帧级目标做时间归一化使各分支专注自身粒度。
正文 · 原文
Abstract:Generative target speaker extraction (TSE) models achieve high-quality extraction, but suffer from considerably slow training convergence. Representation Alignment (REPA) accelerates diffusion transformer training by aligning hidden states with a pre-trained Self-Supervised Learning (SSL) teacher. However, existing alignment in the speech and audio domain operates at a single granularity. For TSE, frame-level alignment leaves the global, time-invariant characteristics of the target speech implicit, whereas utterance-level alignment leaves the time-varying content implicit, even though extracting the target requires both. We propose Multi-Granularity Representation Alignment (MG-REPA), aligning the hidden states with an SSL model at the frame level and a speaker verification model at the utterance level. We further apply temporal normalization to the frame-level target, removing its utterance-level statistics so that each branch concentrates on its granularity. On Libri2Mix, MG-REPA reaches the OVRL and PESQ of a MeanFlow-TSE baseline with $3.0\times$ and $2.1\times$ fewer epochs, respectively. Trained for 750 epochs, it is competitive with the published 2,000-epoch baseline on intrusive metrics while improving non-intrusive metrics. An ablation demonstrates that MG-REPA outperforms alignment at a single granularity.
| Subjects: | Audio and Speech Processing (eess.AS); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2610.32161 [eess.AS] |
| (or arXiv:2610.32161v1 [eess.AS] for this version) | |
| https://doi.org/10.48550/arXiv.2610.32161 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: DoHwan Kim [view email]
[v1]
Fri, 23 Oct 2026 04:51:51 UTC (520 KB)
来源：arXiv cs.AI · arxiv.org