跳到正文
arXiv cs.AI· Pavel Kireyev·· 56 分钟前AI 评分50

RAW 方法:因果测量部署中 LLM 拒答的用户流失代价

RAW: Refusal Aversion in the Wild, A Causal Measurement Method for Deployed LLMs

AI 导读

论文提出 RAW,一种利用 LLM 随机性作为自然实验、从既有日志中因果测量拒答对用户影响的方法。在 LMSYS-Chat-1M 和 WildChat 的 15 个部署模型上,约半数可精确测量的模型每次拒答损失 3 至 17 个百分点的后续再互动,其余模型无可区分效应;方法可在任何已部署模型上仅凭日志运行,成本约 $0.30/千条对话,并已开源标签、审核分数、提示词嵌入与代码。

正文 · 原文

View PDF HTML (experimental)

Abstract:Deployed LLMs field hundreds of millions of conversations a day and refuse a non-trivial fraction of user requests, yet the user-side cost of refusal, whether users disengage and by how much, is unmeasured on any production model: a blind spot for the consumer-facing platforms built on them. A/B-testing refusal is ethically and operationally constrained on frontier-lab products and unavailable to third-party platforms deploying these models through a fixed API. We introduce RAW, a method that uses LLM stochasticity as a natural experiment: at near-identical prompts in any existing log, the same model sometimes refuses and sometimes does not, and that variation identifies the causal user-side effect of refusal. Applying the method to 15 deployed LLMs in LMSYS-Chat-1M and WildChat, with user-clustered inference, we find that for about half of the models we can measure precisely, a refusal costs 3 to 17 pp of subsequent re-engagement; the rest show no distinguishable effect, among them one deployment whose positive point estimate we trace to persistent users retrying the refused request. Our contribution is the measurement method, which runs on any deployed model from logs alone for about \$0.30 per 1,000 conversations. We release labels, moderation scores, prompt embeddings, and pipeline code.
Comments: Accepted to the EMNLP 2026 Industry Track. 26 pages incl. appendices, 8 figures, 11 tables. Code: this https URL Data: this https URL
Subjects: Computation and Language (cs.CL); Artificial Intelligence (cs.AI); Human-Computer Interaction (cs.HC); General Economics (econ.GN)
Cite as: arXiv:2611.21641 [cs.CL]
  (or arXiv:2611.21641v1 [cs.CL] for this version)
  https://doi.org/10.48550/arXiv.2611.21641

arXiv-issued DOI via DataCite (pending registration)

Submission history

From: Pavel Kireyev [view email]
[v1] Fri, 25 Sep 2026 18:55:32 UTC (1,899 KB)

来源:arXiv cs.AI · arxiv.org