arXiv cs.AI· Guanrou Yang, Tian Tan, Qian Chen, Zhikang Niu, Yakun Song, Ziyang Ma, Yushen Chen, Zeyu Xie, Tianrui Wang, Yifan Yang, Wenxi Chen, Qi Chen, Wenrui Liu, Shan Yang, Xie Chen·· 3 小时前AI 评分44
WavCube：通过语义-声学联合建模统一语音理解与生成表示
WavCube: Unifying Speech Representation for Understanding and Generation via Semantic-Acoustic Joint Modeling
AI 导读
WavCube 是一种由 SSL 语音编码器导出的紧凑连续潜在表示，可同时支持语音理解、重建与生成。它采用两阶段训练：先训练语义瓶颈过滤冗余，再通过端到端重建注入声学细节并以语义锚定损失保持语义稳定。
正文 · 原文
Authors:Guanrou Yang, Tian Tan, Qian Chen, Zhikang Niu, Yakun Song, Ziyang Ma, Yushen Chen, Zeyu Xie, Tianrui Wang, Yifan Yang, Wenxi Chen, Qi Chen, Wenrui Liu, Shan Yang, Xie Chen
Abstract:Integrating speech understanding and generation is a pivotal step toward building unified speech models. However, the different representations required for these two tasks currently pose significant compatibility challenges. Typically, semantics-oriented features are learned from self-supervised learning (SSL), and acoustic-oriented features from reconstruction. Such fragmented representations hinder the realization of truly unified speech systems. We present WavCube, a compact continuous latent derived from an SSL speech encoder that simultaneously supports speech understanding, reconstruction, and generation. WavCube employs a two-stage training scheme. Stage 1 trains a semantic bottleneck to filter off-manifold redundancy that makes raw SSL features intractable for diffusion. Stage 2 injects fine-grained acoustic details via end-to-end reconstruction, while a semantic anchoring loss ensures the representation remains grounded within its original semantic manifold. Comprehensive experiments show that WavCube closely approaches WavLM performance on SUPERB despite an 8x dimensional compression, attains reconstruction quality on par with existing acoustic representations, delivers state-of-the-art zero-shot TTS performance with markedly faster training convergence, and excels in speech enhancement, separation, and voice conversion tasks on the SUPERB-SG benchmark. Systematic ablations reveal that WavCube's two-stage recipe resolves two intrinsic flaws of SSL features for generative modeling, paving the way for future unified speech systems. Codes and checkpoints are available at this https URL.
| Subjects: | Audio and Speech Processing (eess.AS); Artificial Intelligence (cs.AI); Computation and Language (cs.CL) |
| Cite as: | arXiv:2605.06407 [eess.AS] |
| (or arXiv:2605.06407v2 [eess.AS] for this version) | |
| https://doi.org/10.48550/arXiv.2605.06407 arXiv-issued DOI via DataCite |
Submission history
From: Guanrou Yang [view email]
[v1]
Thu, 7 May 2026 15:17:24 UTC (675 KB)
[v2]
Wed, 11 Nov 2026 09:58:10 UTC (274 KB)
来源：arXiv cs.AI · arxiv.org