arXiv cs.AI· William DeShong, Barath Kannan, Auberon Palma, Brenda Bradshaw, Ann Bruhn, James F. Leathrum, Jr., Yuzhong Shen·· 2 小时前AI 评分35
无真值参考下筛选 LLM 评判模型:AI 牙科卫生教练的初步评估
Screening LLM Judges Without Ground Truth: A Preliminary Evaluation of an AI Dental Hygiene Coach
AI 导读
一项初步研究提出分阶段流水线,在专家评审前筛选候选 LLM 评判和学生模型,并在 AI 牙科卫生教练上做了案例验证。实验用 Granite-4.1-8B 与 Qwen3-8B 两个微调小模型、各三种数据规模,由五个评判按七项指标打分,本地评判 Qwen3.8-27B 总体与托管评判最相似,且无评判表现出偏向自家模型家族。训练和参考数据均为合成且未经临床验证,专家评审尚未进行,结论仍是初步的。
正文 · 原文
Abstract:Evaluating a fine-tuned coaching model with expert graders is slow and expensive, and an LLM judge cannot be trusted without ground truth. We present a staged pipeline that screens candidate LLM judges and student models before expert review, using explicit decision rules at each stage: judge output audit, similarity with hosted judges, bias checks, and decision robustness. We demonstrate it on an AI dental hygiene coach. The case study uses synthetic, clinically unvalidated training and reference data to demonstrate the operation of the pipeline, rather than to establish the clinical performance of the resulting models. Two fine-tuned small language models (Granite-4.1-8B and Qwen3-8B), each trained at three data sizes, were scored on seven metrics by five judges (three local, two hosted). Two local judges, Qwen3.8-27B and Gemma-4-31B-it, were not distinguishable from the agreement between the hosted judges on six of seven metrics by rank correlation, and Qwen3.8-27B was the most similar to the hosted judges overall. No judge met our criterion for favoring its own model family or training-data source. The student comparison did not separate the two models on any metric under the pipeline's rule, so the choice of Granite-4.1-8B, which led in 29 of 35 judge-by-metric cells, is a lean that depends partly on truncated Qwen3-8B outputs. The pipeline also surfaced problems that raw scores hid, including the truncated outputs and hallucination flags from one hosted judge that no other judge corroborated. No dental hygiene expert has graded these responses, so the findings are preliminary and the pipeline's final stage, anchoring to expert judgement, is proposed but not performed.
| Comments: | 29 pages, 2 figures, 19 tables |
| Subjects: | Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2610.24335 [cs.AI] |
| (or arXiv:2610.24335v2 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2610.24335 arXiv-issued DOI via DataCite |
Submission history
From: William DeShong [view email]
[v1]
Sat, 17 Oct 2026 00:02:02 UTC (6,837 KB)
[v2]
Tue, 17 Nov 2026 15:19:46 UTC (3,102 KB)
来源:arXiv cs.AI · arxiv.org