arXiv cs.AI· Hansol Park, Minkyung Song, Zinu Kim, Yujun Lee, Hoseong Ahn, Hyunjin Choi, Kyuhong Shim·· 4 小时前AI 评分37
Tap2Listen：流式音视频目标说话人提取的即时目标切换
Tap2Listen: Instant Target Switching for Streaming Audio-Visual Speaker Extraction
AI 导读
Tap2Listen 是一个流式音视频目标说话人提取（AV-TSE）模型，可对面部选择即时响应目标切换，切换仅需 0.12ms 重投影并立即生效。在 LRS3-2mix 测试集上，其 SI-SNRi 从基线的 8.85 提升至 9.68 dB，切换到已知人脸后首帧即输出全质量结果，而基线约需 1 秒恢复。
正文 · 原文
Abstract:Streaming audio-visual target speaker extraction (AV-TSE) uses a target speaker's face to extract the speaker's speech from an input audio mixture in real time. However, existing systems fix the target speaker for an entire clip, while conversation demands frequent and immediate target switches. To address this problem, we propose Tap2Listen, a streaming AV-TSE model designed for instant response to target switches. The model separates computation by modality: the target-agnostic audio encoder processes only the mixture, while the target-specific visual cues of the selected face condition only the decoder. A switch requires only a 0.12ms re-projection and takes effect immediately. We also design component-wise losses: a prediction loss for the encoder and a supervised extraction loss for the decoder. For evaluation, we measure how quickly output quality recovers after a switch. On the standard LRS3-2mix test set, Tap2Listen improves SI-SNRi from 8.85 to 9.68 dB at a comparable parameter count. After a switch to a known face, the model produces full-quality output from the first frame, whereas the baseline needs about 1 second to recover. Finally, we show that the proposed streaming AV-TSE model runs in real time on a single CPU core.
| Comments: | NeurIPS 2026 Workshop on On-Device Intelligence |
| Subjects: | Sound (cs.SD); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2610.36643 [cs.SD] |
| (or arXiv:2610.36643v1 [cs.SD] for this version) | |
| https://doi.org/10.48550/arXiv.2610.36643 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Hansol Park [view email]
[v1]
Tue, 27 Oct 2026 05:03:47 UTC (1,645 KB)
来源：arXiv cs.AI · arxiv.org