LLM 主观偏好中的规律：Bouba 击败 Kiki
Bouba Beats Kiki, and Other Regularities in the Subjective Preferences of LLMs
AI 导读
一项将在 NeurIPS 2026 发表的研究考察大语言模型在数字、婴儿名字、虚构城市等短文本比较任务中如何在多个可行选项间做主观选择。研究者识别出长度、字母顺序、流行度、元音搭配、读音等 29 个方面，发现模型对这些方面存在强且可复现的偏好，且提问方式的细微变化会使某些偏好被激活或关闭。此外，当模型同时负责生成候选列表和排序时，两个阶段之间的意外交互也会影响其表述的偏好。
正文 · 原文
Abstract:Users engage with large language models on tasks that are purely objective ("When was Netflix founded?"), purely subjective ("What should I name my child?") or anywhere in-between. We have rich tools for evaluating model accuracy with respect to objective criteria. However, when it comes to subjective aspects of a task, we do not have similarly developed methodology at hand. In this paper, we take some early steps towards framing the question of subjective evaluation. Rather than measure how often a model is right or wrong on a question, we instead ask, how do models make choices among multiple viable responses without a clear winner?
To study this question, we restrict our focus to short-form content, and consider comparison tasks among three such content types: numbers, baby names, and fictional cities. We identify 29 aspects of these short-form examples covering many areas of syntax and semantics (length, alphabetic order, popularity, vowel mix, spoken sound, etc.). We show first that models tend to have strong and reproducible preferences for or against some of these aspects. Second, we show that variations in how the model is asked to express a preference may cause some of these aspects to activate or deactivate in the model's selection. Third, we show that overall preference distributions among pairs may vary radically based on seemingly-unimportant variations in the nature of the question posed to the model, raising new questions about manipulation. And finally, we show that, if models are used both to create lists of alternatives and to rank them, there are unexpected interactions between the two stages that may also drive the models' stated preferences.
| Comments: | 10 pages, 5 figures in main text; 27 pages, 13 figures in supplement. To appear at NeurIPS 2026 |
| Subjects: | Computation and Language (cs.CL); Artificial Intelligence (cs.AI); Computers and Society (cs.CY); Machine Learning (cs.LG) |
| Cite as: | arXiv:2610.37692 [cs.CL] |
| (or arXiv:2610.37692v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2610.37692 arXiv-issued DOI via DataCite |
Submission history
From: Katy Blumer [view email]
[v1]
Wed, 21 Oct 2026 01:37:54 UTC (4,386 KB)
来源：arXiv cs.AI · arxiv.org