跳到正文
arXiv cs.AI· Xingyu Zhao·· 1 小时前AI 评分44

大语言模型的可靠性是因用户而异的属性

Reliability is a user-specific property of large language models

AI 导读

arXiv 论文提出,大语言模型(LLM)服务的可靠性应针对特定用户和使用条件评估,仅靠公开 benchmark 的能力分数不足以支撑未来运行中故障的判断。作者区分发现缺陷的测试与评估可靠性的测试,归纳出十项建模考量,并提出涵盖声明规范、动态运行画像、分级结果与噪声判断、局部概率鲁棒性、交互依赖性及统计推断与再评估的六条研究路线。

正文 · 原文

View PDF HTML (experimental)

Abstract:Public benchmarks support comparisons of large language model (LLM) capabilities, yet users can experience the same highly ranked model differently. Drawing on traditional software reliability, we argue that the reliability of an LLM service should be assessed for specified users and conditions of use. This requires more than a capability score: evaluation evidence must support a claim about failure during future operation. We distinguish testing to discover faults from testing to assess reliability and identify ten modelling considerations behind such claims. We propose a six-workstream research roadmap linking claim specification, dynamic operational profiles, graded outcomes and noisy judgements, local probabilistic robustness, interaction dependence, and statistical inference with reassessment. Local probabilistic robustness addresses the behavioural variation hidden when different concrete inputs are treated as the same task. Published findings and an illustrative customer-support example motivate the roadmap; its integrated validation remains an open research task. Reliability claims complement, but do not replace, separate safety and security arguments. Capability scores should therefore be complemented by versioned operational claims that state whose requirements are covered, under which conditions, and with what uncertainty.
Subjects: Software Engineering (cs.SE); Artificial Intelligence (cs.AI)
Cite as: arXiv:2611.23732 [cs.SE]
  (or arXiv:2611.23732v1 [cs.SE] for this version)
  https://doi.org/10.48550/arXiv.2611.23732

arXiv-issued DOI via DataCite (pending registration)

Submission history

From: Xingyu Zhao [view email]
[v1] Wed, 18 Nov 2026 02:44:28 UTC (1,704 KB)

来源:arXiv cs.AI · arxiv.org