PlanAudio：从自由文本提示统一合成语音与音效
Unified Synthesis of Compositional Speech and Sound from Free-Form Text Prompts
AI 导读
arXiv 论文提出端到端框架 PlanAudio，将无约束的自由文本提示直接映射为融合语音与音效的统一音频，无需中间提示预处理。该框架在单一自回归模型中引入潜在规划，先预测音频布局再合成声学 token，并构建大规模训练语料和 FUAG 基准。实验显示单个 PlanAudio 在语音与音效混合生成上超过现有流水线和统一基线，论文已被 NeurIPS 2026 接收。
正文 · 原文
Abstract:Generating unified audio that naturally integrates speech and sounds from free-form text prompts remains challenging. For example, the prompt ``A fast drum roll ends with a cymbal crash, after which a man shouts: `Come on!'.'' describes several events that should appear in order and form a coherent recording rather than independent tracks that lack coherent temporal and semantic interactions. Current methods broadly follow two paradigms: pipeline methods couple separate text-to-sound and text-to-speech models, while unified models rely on a single generator that still requires structured, multi-field prompt inputs. Both paradigms require upstream prompt decomposition or rewriting, limiting free-form flexibility and risking cascading errors. To enable direct generation from free-form prompts within a single model, we introduce PlanAudio, an end-to-end framework that maps unconstrained free-form text to unified audio without intermediate prompt preprocessing. To address the challenges of direct free-form generation, PlanAudio integrates latent planning into a single autoregressive framework, predicting continuous audio layout before acoustic token synthesis. We construct a large-scale training corpus and introduce Free-Form Unified Audio Generation Benchmark (FUAG) for evaluation, both spanning speech, sound, and joint speech-and-sound generation. Experiments show that a single PlanAudio outperforms existing pipeline and unified baselines on speech-and-sound generation, while remaining competitive with specialist models on speech and sound generation. Further analysis validates the effectiveness of audio layout planning over textual planning, acoustic planning, and direct generation. Sample Available: this https URL.
| Comments: | Accepted by NeurIPS 2026. This version renames concepts, reorganizes sections, and fixes Method notation; adds ablation experiments and core-mechanism analysis; updates Table 1 with larger training data (improved speech performance in speech-and-sound mixed scenarios). The main conclusions remain unchanged |
| Subjects: | Sound (cs.SD); Artificial Intelligence (cs.AI); Multimedia (cs.MM) |
| Cite as: | arXiv:2605.28063 [cs.SD] |
| (or arXiv:2605.28063v2 [cs.SD] for this version) | |
| https://doi.org/10.48550/arXiv.2605.28063 arXiv-issued DOI via DataCite |
Submission history
From: Yuyue Wang [view email]
[v1]
Wed, 27 May 2026 07:15:35 UTC (6,129 KB)
[v2]
Tue, 10 Nov 2026 07:32:24 UTC (1,750 KB)
来源：arXiv cs.AI · arxiv.org