跳到正文
arXiv cs.AI· Yezhou Cheng, Zehua Yang, Bojun Lin·· 2 小时前AI 评分34

基于符号化词汇证据的风险校准对话 AI 意图路由方法

Risk-Calibrated Intent Routing for Conversational AI via Signed Lexical Evidence

AI 导读

一项 arXiv 研究提出结合句子分类器最高两档分数差与符号化词汇证据的双特征置信度模型,用于对话 AI 的意图路由。在 BANKING77、CLINC150 和 HWU64 上,该方法将风险-覆盖率曲线下面积分别降低 15.8%、15.1% 和 11.8%;在 5% 错误率目标下,接受请求比例在 BANKING77 提升 1.83 个百分点、HWU64 提升 5.14 个百分点。

正文 · 原文

View PDF HTML (experimental)

Abstract:Conversational AI has become widely used as advances in language models let people find information, get writing help, and ask for practical guidance in their own words. When an assistant handles specific tasks, it must infer what the user wants and decide whether to act on that prediction. A sentence classifier's confidence can miss clues in the wording. We combine the gap between the classifier's two highest scores with signed lexical evidence, a score based on words and short character patterns. The score is positive when those patterns favor the predicted intent and negative when they favor another. We use separate calibration data and a statistical upper bound on the error rate to set the minimum confidence required to accept a request. Across ten runs per dataset, our method reduces the area under the risk-coverage curve, which summarizes error as more requests are accepted, by 15.8%, 15.1%, and 11.8% on BANKING77, CLINC150, and HWU64, respectively, compared with a confidence model that uses only sentence classifier scores. At a 5% error target, the share of accepted requests rises by 1.83 percentage points on BANKING77 and 5.14 points on HWU64. CLINC150 already has near-full coverage at that target. At a 2% target, calibration finds a rule that accepts some requests in all 30 runs. Our contribution is a two-feature confidence model that improves routing by measuring how strongly the wording supports or challenges the original intent prediction.
Subjects: Computation and Language (cs.CL); Artificial Intelligence (cs.AI)
Cite as: arXiv:2610.00262 [cs.CL]
  (or arXiv:2610.00262v2 [cs.CL] for this version)
  https://doi.org/10.48550/arXiv.2610.00262

arXiv-issued DOI via DataCite

Submission history

From: Yezhou Cheng [view email]
[v1] Thu, 24 Sep 2026 07:06:22 UTC (142 KB)
[v2] Wed, 18 Nov 2026 03:17:05 UTC (145 KB)

来源:arXiv cs.AI · arxiv.org