arXiv 论文：基于偏好的模型排行榜易受操纵，单项攻击可提升名次 15-20 位
Leaderboard Hacking: Preference-Based Model Evaluations are Vulnerable to Manipulation
AI 导读
arXiv 论文 arXiv:2611.14769 发现 Arena 式大语言模型排行榜存在三类操纵漏洞：策略性投票、策略性提示和策略性提名无关模型。研究在公开排行榜数据上验证这些攻击对几乎所有测试模型有效，单次攻击可将模型名次提升最多 15-20 位，组合使用最多提升 45 位，并针对每类攻击提出防御方法；漏洞适用于任何允许开放提示、投票或提交模型的 Bradley-Terry 偏好评估。
推荐理由
论文用公开排行榜数据演示了三类偏好评估操纵手法及其提升名次的幅度，并给出对应防御思路。
正文 · 原文
Abstract:Arena-style rankings of language models are a widely used evaluation framework in online leaderboards, research papers, and model evaluation in production. These rankings aim to realistically assess model performance using pairwise comparisons of anonymous models on user prompts, aggregating votes via the Bradley-Terry model to produce the final ranking. However, we find that these leaderboards are highly vulnerable to several manipulation strategies that unscrupulous providers could use to boost a model's rank: strategic voting (individual votes that boost a model's rank, including votes on irrelevant models), strategic prompting (choosing prompts that favor a model), and strategic nomination (boosting the rank of a model by inserting irrelevant models). We demonstrate these effects on public leaderboard data, analyze the circumstances that cause these vulnerabilities, and describe approaches to safeguard against each attack. Notably, these issues hold for any preference-based evaluation using the Bradley-Terry model that allows open prompting, voting, or model submission. These manipulations boost the rank of nearly every tested model; each attack can boost a model by up to 15-20 places in a ranking and, combined, they can boost a model by up to 45 places.
| Subjects: | Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2611.14769 [cs.AI] |
| (or arXiv:2611.14769v1 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2611.14769 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Eve Fleisig [view email]
[v1]
Tue, 10 Nov 2026 21:47:55 UTC (2,991 KB)
来源：arXiv cs.AI · arxiv.org