系统幻觉量表（SHS）：评估大语言模型感知可靠性的以人为中心工具
The System Hallucination Scale (SHS): A Human-Centered Instrument for Assessing Perceived Reliability of Large Language Models
AI 导读
研究者提出系统幻觉量表（SHS），一种仿照 SUS 和 SCS 设计的轻量工具，用 10 个平衡的 Likert 量表条目从事实准确性、来源可靠性、逻辑连贯性、呈现欺骗性和对用户引导的响应性五个维度衡量用户感知的大语言模型可靠性。
正文 · 原文
Abstract:Large language models routinely produce factually incorrect or misleading outputs, yet conventional automated benchmarks overlook how users perceive reliability during interaction. We introduce the System Hallucination Scale (SHS), a lightweight instrument modeling the famous System Usability Scale (SUS) and System Causability Scales (SCS). SHS measures perceived reliability across five dimensions: factual accuracy, source reliability, logical coherence, deceptiveness of presentation, and responsiveness to user guidance, using ten balanced Likert-scale items that provide a directional score and consistency diagnostic. In a supervised validation study with 193 participants (386 evaluations across 47 trained experimenters), SHS demonstrated strong internal consistency and high clarity, completing in an average of 4.2 minutes. Participants rated source verifiability and presentation deceptiveness worse than factual accuracy, indicating perceived vulnerabilities lie primarily in traceability and framing. As a proof of concept, SHS assesses human-perceived reliability rather than ground-truth correctness, providing open-source multilingual implementations and an interactive calculator.
| Subjects: | Computation and Language (cs.CL); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2603.09989 [cs.CL] |
| (or arXiv:2603.09989v2 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2603.09989 arXiv-issued DOI via DataCite |
|
| Related DOI: | https://doi.org/10.1080/10447318.2026.2739758
DOI(s) linking to related resources |
Submission history
From: Heimo Müller [view email]
[v1]
Fri, 13 Feb 2026 10:13:09 UTC (400 KB)
[v2]
Thu, 29 Oct 2026 13:02:53 UTC (85 KB)
来源：arXiv cs.AI · arxiv.org