项目级误差限制小语言模型在 Fermi 估算中的多智能体聚合收益
Shared Item-Level Error Limits the Aggregation of Small Language Models in Fermi Estimation
AI 导读
一项 arXiv 研究分析了 15 个开源模型（3B–72B）在 realFP 等基准上给出的 251 万条 Fermi 估算答案，发现条目级误差分量占误差方差的中位数 0.906。
正文 · 原文
Abstract:Multi-agent systems of language models are often motivated by an averaging argument: if the agents' errors are unbiased and independent, the error of an aggregate of N answers falls as 1/sqrt(N). Fermi estimation, whose continuous answers are aggregated on the logarithmic scale, permits a direct test. We decompose the log10 error of each answer into the mean error of its model on the item, common to all its draws on the item, and a residual, and estimate both from 2.51M answers given before any communication by 15 open-weight models (3B-72B) on realFP, with replication under a single prompt pair on realFP, Science Olympiad Fermi questions and a guesstimation set. Given the item, teammates' errors are uncorrelated and teams behave as independent draws, but the item-level component accounts for a median 0.906 of the error variance over items. Teams of thirty therefore reduce a single agent's error by a median 1.7% (at most 5.4%), whereas draws that were also unbiased on each item would reduce it by 74-79%. All models compress their estimates toward about 10^6; this trend explains a median 17% of the item-level component, and a cross-validated linear recalibration on labelled items removes the overall bias but not that component. Three rounds of debate reduce within-team dispersion by 93% and leave the item-level component unchanged; revision without peers attains the same small gain. Mixing five models of similar accuracy reduces error by 6.6% relative to their own homogeneous teams, but no mixture improves on the most accurate model. Higher sampling temperature changes team error by about 1% at most, and contrastive activation steering not detectably; per-agent scaffold prompts (not significant after multiplicity correction) and skipping one layer in a single-model screen improve it. The results are conditional on the models, benchmarks, prompts and settings tested.
| Comments: | 21 pages, 2 figures, 17 tables |
| Subjects: | Multiagent Systems (cs.MA); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2611.16315 [cs.MA] |
| (or arXiv:2611.16315v1 [cs.MA] for this version) | |
| https://doi.org/10.48550/arXiv.2611.16315 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Carolina Fortuna PhD [view email]
[v1]
Thu, 12 Nov 2026 05:23:22 UTC (579 KB)
来源：arXiv cs.AI · arxiv.org