LLM 评委评估实体对齐任务的可靠性研究
Reliability of LLM Judges for Evaluating Entity Alignment
AI 导读
一项 arXiv 研究首次系统评测了 LLM-as-judge 在实体对齐（EA）评估中的可靠性，覆盖三个前沿模型、三个数据集和四个 EA 系统。研究发现“锚点偏差”：当提示词暴露系统决策标签时，评委判别力崩溃（J-ROC-AUC 0.12–0.87），而无标签协议在特征名数据集上可达 0.98–1.00。
正文 · 原文
Abstract:Entity Alignment (EA) identifies equivalent entities across knowledge graphs and is critical for knowledge base integration and ontology merging. Evaluating EA systems at scale requires expensive expert annotation, making systematic assessment across diverse domains practically infeasible. LLM-as-judge evaluation offers a potentially scalable alternative, yet its reliability for structured prediction tasks like EA remains unstudied. We present the first systematic benchmarking study across three frontier models, three datasets, and four EA systems, using perturbation bias diagnostics, meta-evaluation across all dataset--judge--prompt combinations, and counterfactual label-flip tests. We identify anchor bias, a failure mode in which judges invert discrimination when the system's decision label is visible. Judge discrimination collapses under label-exposing prompts (J-ROC-AUC 0.12--0.87), while a label-free protocol recovers near-ceiling capability on distinctive-name datasets (0.98--1.00 for NeuSymEA) and significant recovery on biomedical pairs (0.93--0.95). Counterfactual label flips show that the visible label causally shifts scores by more than 2 points in 53--99% of pairs (FSR), with sensitivity growing with model capability, a pattern we call the frontier model paradox: stronger judges are more label-sensitive, not less, yet also better label-free predictors. The direction of the shifts is mixed, so we report this as an observed pattern. A blinded two-annotator human evaluation (102 pairs, Cohen's $\kappa$ = 0.902) provides a gold-label reference. We release the first biomedical EA benchmark (MeSH--SNOMED CT, 15K pairs) and a reproducible auditing framework for LLM judge reliability in EA. Code and data are available at this https URL.
| Subjects: | Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2610.09554 [cs.AI] |
| (or arXiv:2610.09554v2 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2610.09554 arXiv-issued DOI via DataCite |
Submission history
From: Vaibhava Lakshmi Ravideshik [view email]
[v1]
Wed, 7 Oct 2026 06:55:45 UTC (1,498 KB)
[v2]
Thu, 29 Oct 2026 09:17:14 UTC (1,499 KB)
来源：arXiv cs.AI · arxiv.org