决策编码器需要多少 LLM 标注数据，标签错到什么程度仍可用？
How many LLM-labelled rows does a decision encoder need, and how wrong can the labels be?
AI 导读
一项 arXiv 研究考察决策编码器 Laya（ModernBERT-large）需要多少 LLM 标注训练数据以及标签噪声的容忍度。对 2,818 行数据池的 19 次微调显示，准确率从微调前的 69.4% 升至全量数据的 86.9%，且 10% 对称标签噪声未造成可检测变化（-0.2 点），20% 噪声损失 3.4 点、40% 损失 8.7 点。
正文 · 原文
Abstract:A decision encoder such as Laya reads a question with named options (for example, which department should handle a ticket) and a text, and picks an option. It reads both in one forward pass and returns a probability for every option, without writing any text. We ask how many LLM-written training rows such an encoder needs and how wrong those labels may be. We ran 19 fine-tunes of Laya (ModernBERT-large) on nested subsets of a 2,818-row pool and scored them on 1,065 held-out items from the same generator, so every test item is in-distribution and no out-of-family test was run. Accuracy rises from 69.4% before fine-tuning to 74.3% with 5% of the rows, 84.8% with 50% and 86.9% with all rows. The curve flattens, and a saturating fit with bootstrap intervals cannot pin down its plateau. Seed spread at 20% of the rows (s.d. 1.6 points) exceeds some of the steps between sizes. Giving small subsets as many updates as the full pool gains 1.9 points at 5% and 1.9 points at 20% of the rows (one seed; p=0.14 and 0.13), and at 20% that is below the 2.3-point difference between two ordinary seeds. A single run with 10% symmetric label noise shows no detectable change (-0.2 points, 95% interval -1.7 to +1.4); 20% costs 3.4 points and 40% costs 8.7 points. Confusable-pair noise at 20% costs 7.4 points, and under noise the model moves from over-confident to under-confident. An independent labeller disagrees with the gold labels on 3.5% of held-out items, and about half of its choice disagreements (16 of 33, leaving each item out of the partner map) name the partner option that the confusable-pair noise uses. Laya is +0.9 points ahead of a TF-IDF classifier trained on the same rows at full size (the interval includes zero) and +21.1 points ahead at 5% of the rows. Everything is synthetic, LLM-labelled and from one backbone, one trainer and mostly one seed.
| Comments: | 24 pages, 20 figures, 15 tables |
| Subjects: | Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2611.16498 [cs.AI] |
| (or arXiv:2611.16498v1 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2611.16498 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Aliyas Ahmad Misri [view email]
[v1]
Thu, 12 Nov 2026 09:02:55 UTC (1,695 KB)
来源：arXiv cs.AI · arxiv.org