arXiv 论文：用户评价大语言模型的标准在上升，AI 并未变笨
AI Isn't Getting Dumber, We're Getting Pickier: Rising Standards in Human Evaluation of Large Language Models
AI 导读
研究者分析 Chatbot Arena 上 2023 年 5 月至 2024 年 8 月的 1,260,967 张盲投票，在版本固定的模型对上检验用户评价标准是否在变化。
推荐理由
论文用 Chatbot Arena 固定模型对的盲投数据分离模型退化与用户标准上升，对理解 LLM 排行榜和人类反馈训练数据的偏差有参考价值。
正文 · 原文
Abstract:Users often complain that AI assistants such as ChatGPT are getting worse. Some versions do change, but human evaluation leaves a second possibility untested: the people judging may change. We separate the two on Chatbot Arena, a public platform where people compare answers from two anonymous large language models (LLMs) and pick the better one or say both are bad. We study 1,260,967 blind votes cast from May 2023 to August 2024 on pairs of models whose versions are fixed: dated vendor snapshots and models with fixed open weights. For the same pair of unchanged models, the share of votes saying "both are bad" rises by 1.71 percentage points a month on a mean of 17.1%, and by 1.91 points a month in 2024. In fourteen pairs observed often in both 2023 and 2024, it rose in every one, from 15.5% to 23.3%. The ranking does not trend: in 2024, the stronger model's share of decisive votes is flat. The rise holds within the same voter and on the same pair, and across arrival cohorts from their first recorded vote. It holds with prompt controls, in every major language, for dated vendor snapshots, open-weight models and models with stable answer length, and between interface changes. The drift is shared across models: month effects estimated from other models' pairs remove it from each model's own pairs. Platform-wide, the average model would be expected to score 78% against the 2023 average, yet the share of "both are bad" votes ended near where it began. The pattern fits a standard that follows the category's progress, so a product that stands still is judged as declining. Absolute verdicts on AI drifted while relative comparisons held steady, an evaluation drift that matters for LLM leaderboards and for the human feedback used to train models.
| Comments: | 23 pages, 6 figures, 10 tables; working paper |
| Subjects: | Human-Computer Interaction (cs.HC); Artificial Intelligence (cs.AI); Computation and Language (cs.CL); Computers and Society (cs.CY) |
| ACM classes: | H.1.2; I.2.7; J.4 |
| Cite as: | arXiv:2611.14333 [cs.HC] |
| (or arXiv:2611.14333v1 [cs.HC] for this version) | |
| https://doi.org/10.48550/arXiv.2611.14333 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Di Zhu [view email]
[v1]
Tue, 10 Nov 2026 16:56:16 UTC (128 KB)
来源：arXiv cs.AI · arxiv.org