计数中的分歧：行政文本 AI 标注的美元加权测量、选择性机器标注与预算内人工复核研究
Counting Under Disagreement: Dollar-Weighted Measurement, Selective Machine Labelling and Budgeted Human Review from Disagreeing AI Readers of Administrative Text
AI 导读
一项 arXiv 研究提出在 AI 标注存在分歧时估计行政文本货币总量的方法，测试集为美国税表中的 121,070 条资助用途文本（799,675 条记录，$126.4B），由规则分类器和三个语言模型标注，651 条文本由三人独立阅读。
正文 · 原文
Abstract:We study AI-assisted measurement when one text label applies to many administrative records, the target is a monetary total, and the human readings that anchor the machines disagree. The testbed is 121,070 grant-purpose texts from US tax filings (799,675 records, \$126.4B), labelled by a rule-based classifier and three language models, with a designed sample of 651 texts read independently by three people under one codebook. We state exact conditions for a fixed-predictor difference estimator and for a cross-fitted predictor whose folds are assigned to the whole frame before sampling, under Poisson sampling, and assess the implemented fixed-size procedures by simulation: 8 predictors whose own totals span \$11.2-15.0B agree to within \$0.4B once rectified. Two rules that accept machine labels at a stated dollar-weighted error, weighted conformal p-values under a dollar measure and Learn-then-Test on the design estimate of the error, are evaluated on nested held-out folds and on re-draws of the design with synthetic truths; their error control in deployment is not established, and the count-based rule accepts no text under a dollar-drawn sample. In a replay that charges every human label, a two-phase review policy whose calibrator is fitted on its own first phase reduces the RMSE of the total by 28% at a budget of 100 texts relative to dollar-proportional sampling, while pure Neyman allocation fails with an uncalibrated predictor. Independent readers shift the estimated total by \$3.93B, 2.2 times the within-reader sampling interval, and institution-level intervals under-cover at every size. Every headline result is re-run under each reader. The results identify both the benefits and the limits of using learned predictions to decide what a person should verify.
| Comments: | 61 pages, 8 figures, 46 tables. Replication record (calibration sample, population aggregates, prompts and analysis code): this https URL |
| Subjects: | Artificial Intelligence (cs.AI); Machine Learning (cs.LG); Methodology (stat.ME) |
| Cite as: | arXiv:2611.13215 [cs.AI] |
| (or arXiv:2611.13215v1 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2611.13215 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: George Koomullil [view email]
[v1]
Mon, 9 Nov 2026 20:37:48 UTC (266 KB)
来源：arXiv cs.AI · arxiv.org