LLM 错误中的跨题目结构从何而来：共享随机种子、共享提示词与答案字母偏好
Where Cross-Question Structure in LLM Errors Comes From: Shared Seeds, Shared Prompts, and Answer-Letter Preference
AI 导读
一项 arXiv 研究用题目相关矩阵特征值检验 LLM 错误间的跨题目结构：单模型独立抽样的答案无可检测结构，只有当一次评测中所有题目共享随机种子或 few-shot 提示词时才出现。
正文 · 原文
Abstract:Disagreement between language models is often read as missing knowledge. Questions that need the same missing knowledge should then be right or wrong together. We test for this structure with the eigenvalues of the questions' correlation matrix. Within one model, independently drawn answers show no detectable structure; it appears when all questions in one pass over the benchmark share a random seed or few-shot prompt. Across 24 models, the largest structure beyond overall skill is a preference among answer letters: when the options are rotated, it follows the letter, not the question, and it depends on one model, Llama-3.2-1B; without it only a smaller excess remains. Counting a question correct only if right in every rotation removes the part that acts on guesses. What remains is a marginal excess at the 5% line, tied post hoc to two near-duplicate model pairs and to Llama-3.2-1B.
| Comments: | 12 pages, 3 figures, 3 tables. Accepted at EIML@ICML 2026 |
| Subjects: | Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2607.20464 [cs.AI] |
| (or arXiv:2607.20464v2 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2607.20464 arXiv-issued DOI via DataCite |
Submission history
From: Izhar Ali [view email]
[v1]
Mon, 18 May 2026 14:40:37 UTC (179 KB)
[v2]
Fri, 23 Oct 2026 02:03:05 UTC (762 KB)
来源：arXiv cs.AI · arxiv.org