arXiv cs.AI· Xuan Shi, Chang Zeng, Tiantian Feng, Ye Zhang, Jun Liu, Shih-Heng Wang, Lie Lu, Shrikanth Narayanan·· 3 小时前AI 评分38
Uni-Stream：统一的流式语音语言模型，兼顾语音合成与识别
Uni-Stream: A Unified Streaming Speech Language Model for Speech Synthesis and Recognition
AI 导读
研究者提出 Uni-Stream，一个基于紧凑预训练语音模型构建的统一流式语音语言模型，可在单模型中同时支持语音识别与语音合成，实现实时人机交互。该模型以流式方式处理文本与语音交错输入，通过两阶段训练达到与任务专用模型相当的性能，并将公开发布流式训练用的转写文本和词级时间戳。分析发现语音与文本在统一架构中仍处于不同的潜在特征空间，各模态内部表示在深层更依赖任务。
正文 · 原文
Abstract:Unified streaming speech language models aim to support speech recognition and synthesis within a single model, enabling real-time human-computer interaction. However, jointly supporting these two directions of speech-text conversion while preserving task-specific performance remains challenging. We propose Uni-Stream, a unified streaming speech language model built on compact pretrained speech models. Uni-Stream processes interleaved text and speech in a streaming format and achieves performance comparable to task-specific models through two-stage training. We also develop a data annotation pipeline for streaming training and will publicly release the generated transcriptions and word-level timestamps. We further examine Uni-Stream's representations across tasks and modalities and find that speech and text residence in distinct latent feature space even when trained in a unified model architecture, while representations within each modality become more task-dependent in later layers.
| Subjects: | Audio and Speech Processing (eess.AS); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2610.32187 [eess.AS] |
| (or arXiv:2610.32187v1 [eess.AS] for this version) | |
| https://doi.org/10.48550/arXiv.2610.32187 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Xuan Shi [view email]
[v1]
Fri, 23 Oct 2026 05:32:28 UTC (224 KB)
来源：arXiv cs.AI · arxiv.org