同样的词，不同的判断：基于 ICC 的文本与语音偏好标注跨模态研究
Same Words, Different Judgments: How Preferences Vary Across Modalities
AI 导读
一项 NeurIPS 2026 接收的研究首次用组内相关系数（ICC）方法，在 100 个提示词上对比了人类与合成标注者对相同语义内容的文本和音频偏好评价。结果显示，单模态内达到较好一致性（音频 ICC(2,k) 约 .81，文本约 .74）约需 9 名评分者，且音频评分者决策阈值更窄、长度偏见更小，跨模态一致性接近随机；合成评分可有效预测评分者间一致性。
正文 · 原文
Abstract:Preference-based reinforcement learning (PbRL) is the dominant framework for aligning AI systems to human preferences. However, evaluation protocols for such data were designed for text and have not been validated for speech. We present the first intraclass correlation coefficient (ICC)-based, controlled cross-modal study of human and synthetic preference annotations, comparing text and audio evaluations of identical semantic content across 100 prompts. We show that achieving \textit{good} (audio) to \textit{near good} (text) agreement within either modality (ICC(2,$k$) $\approx$ .81 audio, .74 text) requires $\sim$9 raters and modalities show marked differences in how people report preferences: audio raters exhibit narrower decision thresholds, reduced length bias, and more user-oriented evaluation criteria, with near-chance cross-modality agreement. We demonstrate that synthetic ratings can be used to effectively predict inter-rater agreement, thus serving as an early signal for stimulus selection and proxy for human annotations. Together, these findings argue that evaluation protocols for audio preference data require modality-specific design rather than direct adaptation from text. Code, data, and the audio corpus are available at: this https URL
| Comments: | Accepted to NeurIPS 2026 Evaluations and Datasets Track |
| Subjects: | Sound (cs.SD); Artificial Intelligence (cs.AI); Human-Computer Interaction (cs.HC) |
| Cite as: | arXiv:2602.22710 [cs.SD] |
| (or arXiv:2602.22710v4 [cs.SD] for this version) | |
| https://doi.org/10.48550/arXiv.2602.22710 arXiv-issued DOI via DataCite |
Submission history
From: Aaron Broukhim [view email]
[v1]
Thu, 26 Feb 2026 07:34:15 UTC (148 KB)
[v2]
Wed, 6 May 2026 21:24:15 UTC (260 KB)
[v3]
Tue, 3 Nov 2026 17:54:51 UTC (275 KB)
[v4]
Wed, 4 Nov 2026 18:43:22 UTC (275 KB)
来源：arXiv cs.AI · arxiv.org