跳到正文
arXiv cs.AI· Shubhra Mittal, Gaurav Naresh Mittal·· 4 小时前AI 评分47

幻觉检测器排名在小模型上是否成立?面向从业者的端侧量化 LLM 审计

Does the Ranking Hold? A Practitioner-Facing Audit of Hallucination Detectors on Small, Quantized, On-Device LLMs

AI 导读

一项 arXiv 研究在单张 Colab T4 GPU 上以 4-bit NF4 量化审计了四种幻觉检测器,测试对象为 Llama-3.2-3B-Instruct 和 Qwen2.5-3B-Instruct 两个端侧小模型。

正文 · 原文

View PDF

Abstract:Hallucination detection for large language models is dominated by methods evaluated on 7B-70B research models with abundant sampling budgets and exact-match or ROUGE correctness labels. The small models practitioners actually ship on-device, Phi-3-mini, Llama-3.2-3B-Instruct, Qwen2.5-3B-Instruct, and Gemma-2-2B-Instruct, typically at 4-bit precision, sit outside this evidence base. We present a reproducible one-GPU audit of four detectors spanning three paradigms: length-normalized token-margin calibration, a lexical n-gram baseline, and two sampling-based consistency methods (semantic entropy and SelfCheckGPT-NLI) on two small open instruction-tuned models (Llama-3.2-3B-Instruct and Qwen2.5-3B-Instruct) loaded at 4-bit NF4 quantization on a single Colab T4 GPU, evaluated on closed-book TriviaQA. Cross-model ranking transfer is poor: Spearman rho is -0.80 and Kendall tau-b is -0.667, meaning the detector ranking actively inverts between the two models, and the cheap lexical baseline is statistically indistinguishable from the sampling-based consistency detectors on both models under bootstrap 95 percent confidence intervals. As a second, methodological contribution, we show that the standard token-overlap-to-multiple-choice correctness labeler for TruthfulQA-Generation produces near-chance AUROC (0.46-0.59) for all four detectors on both models, a labeler failure, not a detector failure, since the same detectors reach 0.69-0.89 AUROC on TriviaQA under the identical pipeline. We release the harness, prompts, and per-item scores and frame the detector choice explicitly as a fairness-of-error-rates allocation problem for practitioners deploying on resource-constrained, on-device hardware.
Subjects: Artificial Intelligence (cs.AI)
Cite as: arXiv:2611.17339 [cs.AI]
  (or arXiv:2611.17339v1 [cs.AI] for this version)
  https://doi.org/10.48550/arXiv.2611.17339

arXiv-issued DOI via DataCite

Submission history

From: Shubhra Mittal [view email]
[v1] Mon, 31 Aug 2026 19:11:30 UTC (217 KB)

来源:arXiv cs.AI · arxiv.org