超越准确率：六款大语言模型国际象棋评估的几何稳定性分析

View PDF HTML (experimental)

Abstract:Large language models (LLMs) are usually evaluated in chess by how closely their position evaluations agree with a strong engine, but such agreement does not show that a model evaluates equivalent positions consistently. We apply metamorphic relations to chess evaluation: a 180-degree rotation of pawnless positions, left-right mirroring, color swapping, position pairs one move apart, conversion between Forsyth-Edwards Notation and a natural-language description, and illegal positions. Six LLMs (Claude Sonnet 4.5, DeepSeek Chat, Gemini 2.5 Flash, GPT-5.1, Grok 4.1 Fast and Kimi K2 Turbo) evaluated seven test sets of 500 items each, with Stockfish 16.1 as the reference. All six models failed on rotation: at most 10.6% of the 500 rotated pairs received evaluations within 300 centipawns (cp) of each other, and the models often declared the legal rotated positions illegal. On the other four transformations, stability ranked the models as accuracy did except for the first two places: GPT-5.1 had the lowest mean stability error (186.17 cp) and Claude Sonnet 4.5 the lowest accuracy error (330.41 cp). Rejection rates for illegal positions ranged from 44.2% to 96.0%. Stability should be reported with the share of consistent pairs, because means over answered pairs concealed most rotation failures.

Comments:	30 pages, 8 figures, 5 tables. v2: substantially revised; methods described as implemented in the code, all results recomputed from the archived model outputs, related work extended. Data and code: this https URL
Subjects:	Artificial Intelligence (cs.AI)
Cite as:	arXiv:2512.15033 [cs.AI]
	(or arXiv:2512.15033v2 [cs.AI] for this version)
	https://doi.org/10.48550/arXiv.2512.15033 arXiv-issued DOI via DataCite

Submission history

From: Ruifeng Cao [view email]
[v1] Wed, 17 Dec 2025 02:49:10 UTC (580 KB)
[v2] Thu, 12 Nov 2026 13:31:32 UTC (998 KB)