幻觉检测器排名在小模型上是否成立?面向从业者的端侧量化 LLM 审计
Does the Ranking Hold? A Practitioner-Facing Audit of Hallucination Detectors on Small, Quantized, On-Device LLMs
AI 导读
一项 arXiv 研究在单张 Colab T4 GPU 上以 4-bit NF4 量化审计了四种幻觉检测器,测试对象为 Llama-3.2-3B-Instruct 和 Qwen2.5-3B-Instruct 两个端侧小模型。
正文 · 原文
Abstract:Hallucination detection for large language models is dominated by methods evaluated on 7B-70B research models with abundant sampling budgets and exact-match or ROUGE correctness labels. The small models practitioners actually ship on-device, Phi-3-mini, Llama-3.2-3B-Instruct, Qwen2.5-3B-Instruct, and Gemma-2-2B-Instruct, typically at 4-bit precision, sit outside this evidence base. We present a reproducible one-GPU audit of four detectors spanning three paradigms: length-normalized token-margin calibration, a lexical n-gram baseline, and two sampling-based consistency methods (semantic entropy and SelfCheckGPT-NLI) on two small open instruction-tuned models (Llama-3.2-3B-Instruct and Qwen2.5-3B-Instruct) loaded at 4-bit NF4 quantization on a single Colab T4 GPU, evaluated on closed-book TriviaQA. Cross-model ranking transfer is poor: Spearman rho is -0.80 and Kendall tau-b is -0.667, meaning the detector ranking actively inverts between the two models, and the cheap lexical baseline is statistically indistinguishable from the sampling-based consistency detectors on both models under bootstrap 95 percent confidence intervals. As a second, methodological contribution, we show that the standard token-overlap-to-multiple-choice correctness labeler for TruthfulQA-Generation produces near-chance AUROC (0.46-0.59) for all four detectors on both models, a labeler failure, not a detector failure, since the same detectors reach 0.69-0.89 AUROC on TriviaQA under the identical pipeline. We release the harness, prompts, and per-item scores and frame the detector choice explicitly as a fairness-of-error-rates allocation problem for practitioners deploying on resource-constrained, on-device hardware.
| Subjects: | Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2611.17339 [cs.AI] |
| (or arXiv:2611.17339v1 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2611.17339 arXiv-issued DOI via DataCite |
Submission history
From: Shubhra Mittal [view email]
[v1]
Mon, 31 Aug 2026 19:11:30 UTC (217 KB)
来源:arXiv cs.AI · arxiv.org