Laya 类型化决策模型置信度校准实操指南：一个下午完成重校准
Recalibrating a Typed-Decision Model in an Afternoon: A Hands-On Recipe for Laya's Confidence Scores
AI 导读
一篇教程给出五步配方（测量、诊断、拟合、验证、部署），用 130 行 numpy 重校准 Laya 类型化决策模型的置信度。发布版 checkpoint 在 1,065 条合成 LLM 标注样本上 pooled ECE 为 0.073，但按桶计算的 macro-ECE 达 0.143，12 选项桶过自信 +0.22，单一全局温度反而使 pooled ECE 升至 0.173。
正文 · 原文
Abstract:A typed-decision model such as Laya reads a text and a list of options that the caller defines, for example ticket categories, and returns a probability for each option. Code around it compares the top probability with a threshold to answer, abstain or escalate to a person. Earlier work reports, for a Turkish model, that temperature scaling can raise held-out calibration error (ECE). This tutorial gives a five-step recipe (measure, diagnose, fit, validate, deploy) in 130 lines of numpy with standard methods, none new. On 1,065 synthetic, LLM-labelled items the released Laya checkpoint has a pooled ECE of 0.073 but a bucket-macro ECE of 0.143 (0.127-0.184), the mean over buckets (questions grouped by type and option count): the 12-option bucket is over-confident by +0.22, the three-to-five-option bucket under-confident by 0.05, and they cancel. One global temperature makes pooled ECE worse (0.173); a cross-fitted temperature per bucket brings macro-ECE to 0.073 (0.078 over 10 fold assignments) and reaches 0.089 with 10 labelled items per bucket, but holding out one of seven evaluation sets gives 0.099. After the refit, a gate aimed at 10% error answers 46.3% of items (10.5% wrong) against 6.9% on the published confidences, but one cut-off per bucket on the published confidences answers 45.4%, so most of the gain comes from separating buckets. Confidence-only maps (Platt, histogram, isotonic) need 20 or more labelled items per bucket to approach the temperature; with 320 per bucket the lowest macro-ECE belongs to Platt. In these files the retrained checkpoints store no per-bucket table and the stored 12-option value 0.1006 is clamped to 0.5 at runtime; the refit helps 31 of 33 checkpoints on random folds and 21 with a set held out. Fit and test share one generator and LLM labels, so these are findings about these checkpoints and this data.
| Comments: | 25 pages, 21 figures, 16 tables |
| Subjects: | Artificial Intelligence (cs.AI); Software Engineering (cs.SE) |
| Cite as: | arXiv:2611.16499 [cs.AI] |
| (or arXiv:2611.16499v1 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2611.16499 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Aliyas Ahmad Misri [view email]
[v1]
Thu, 12 Nov 2026 09:02:57 UTC (2,015 KB)
来源：arXiv cs.AI · arxiv.org