AU-Harness：面向 Audio-LLM 高效统一评测的开源工具包
AU-Harness: An Open-Source Toolkit for Efficient and Unified Evaluation of AUdio-LLMs
研究者推出 AU-Harness，一个开源的 大型音频语言模型（LALM）评测框架，针对现有工具流程慢、缺乏多轮对话支持、评测标准不统一三大问题。该框架将评测处理时间减少 77%，提供标准化协议与灵活配置以实现公平比较，并支持多轮对话动态等深度分析，用于研究 LALM 的音频推理能力。论文已被 EMNLP 2026 Findings 接收。
Authors:Hoang Nguyen, Sidharth Surapaneni, Akshay Kalkunte, Jash Mehta, Aman Tiwari, Oluwanifemi Bamgbose, Khyati Mahajan, Jash Shah, Shruthan Radhakrishna, Sathwik Tejaswi Madhusudhan, Vikas Yadav, Sai Rajeswar
Abstract:Large Audio Language Models (LALMs) are rapidly advancing, but evaluating them remains challenging due to inefficient and nonstandardized toolkits that limit fair comparison and systematic assessment. Existing evaluation frameworks exhibit three critical limitations: (1) slow and inefficient processing pipeline that bottlenecks large-scale studies, (2) inadequate multi-turn dialogue support, leaving fundamental questions about cross-turn context integration and performance dynamics over extended conversations in LALMs unanswered; and (3) the absence of unified and scalable evaluation framework capable of keeping pace with the rapid growth of both LALMs and audio benchmarks. To address these issues, we introduce AU-Harness, an efficient and comprehensive evaluation framework for LALMs. Our empirical studies demonstrate that AU-Harness achieves 77% reduction in evaluation processing time over existing evaluation toolkits, enabling large-scale LALM evaluations previously considered impractical. We provide standardized evaluation protocols and flexible configurations for fair model comparison across diverse scenarios. AU-Harness unlocks a range of in-depth analyses difficult to conduct without a unified foundation, including multi-turn dialogue dynamics, enabling the study of true audio reasoning capabilities in existing LALMs. AU-Harness provides both practical evaluation tools and insights into model limitations, advancing systematic LALM development.
| Comments: | EMNLP 2026 Findings |
| Subjects: | Sound (cs.SD); Artificial Intelligence (cs.AI); Machine Learning (cs.LG); Audio and Speech Processing (eess.AS) |
| Cite as: | arXiv:2509.08031 [cs.SD] |
| (or arXiv:2509.08031v4 [cs.SD] for this version) | |
| https://doi.org/10.48550/arXiv.2509.08031 arXiv-issued DOI via DataCite |
Submission history
From: Aman Tiwari [view email]
[v1]
Tue, 9 Sep 2025 15:30:40 UTC (636 KB)
[v2]
Thu, 11 Sep 2025 16:27:59 UTC (634 KB)
[v3]
Mon, 11 May 2026 14:29:23 UTC (5,367 KB)
[v4]
Fri, 23 Oct 2026 04:45:20 UTC (6,188 KB)
来源：arXiv cs.AI · arxiv.org