arXiv cs.AI· Oh Hyun-Bin, Kazuki Shimada, Yuhta Takida, Kim Sung-Bin, Toshimitsu Uesaka, Takashi Shibuya, Kyeongyoon Lee, Tae-Hyun Oh, Yuki Mitsufuji·· 4 小时前AI 评分34
面向动态声源的时空音频语言建模：ST-AudioQA 数据集与 ST-AudioLM
Spatio-Temporal Audio Language Modeling for Dynamic Sound Sources
AI 导读
研究者提出时空音频问答数据集与基准 ST-AudioQA，基于静止与移动声源的一阶高保真立体声（FOA）渲染构建，提供声源身份、方向、距离和运动元数据。同时推出联合学习事件语义与时变轨迹的 ST-Audio Encoder，以及将音频 token 输入 LLM 的 ST-AudioLM，用于回答声音是什么、在哪里、如何移动等时空问答。
正文 · 原文
Abstract:Sound events have semantic identities, locations, and trajectories, yet existing audio-language models typically encode clips as global event content. Conversely, sound event localization models track source directions over time but offer limited semantic coverage for language reasoning. To bridge this gap, we introduce ST-AudioQA, a spatio-temporal audio QA dataset and benchmark built from first-order ambisonic (FOA) renderings of static and moving sound sources. Each scene provides source identity, activity, direction, distance, and motion metadata, enabling dense trajectory supervision and QA about what is sounding, where it is, how it moves, and how sources relate. We further introduce ST-Audio Encoder, which jointly learns event semantics and time-resolved source trajectories, and ST-AudioLM, which exposes these audio tokens to an LLM for spatio-temporal audio QA. Experiments show that this representation improves the balance between semantic preservation and localization and yields stronger reasoning than static spatial and localization-oriented baselines. Beyond the controlled benchmark, ST-AudioLM shows zero-shot transfer to real recordings, with further gains after light adaptation. Project page: this https URL.
| Comments: | EMNLP 2026 Main Conference |
| Subjects: | Sound (cs.SD); Artificial Intelligence (cs.AI); Computation and Language (cs.CL) |
| Cite as: | arXiv:2606.14141 [cs.SD] |
| (or arXiv:2606.14141v2 [cs.SD] for this version) | |
| https://doi.org/10.48550/arXiv.2606.14141 arXiv-issued DOI via DataCite |
Submission history
From: Hyun-Bin Oh [view email]
[v1]
Fri, 12 Jun 2026 05:58:31 UTC (273 KB)
[v2]
Tue, 27 Oct 2026 10:34:08 UTC (1,477 KB)
来源：arXiv cs.AI · arxiv.org