跳到正文
arXiv cs.AI· Abhash Kumar Jha, Diana Alexandra Onu\c{t}u, Neeratyoy Mallik, Swagatam Haldar, Sam Laing, Niccol\`o Ajroldi, Shiwei Liu, Joaquin Vanschoren, Aaron Klein·· 2 小时前AI 评分44

论文提出摊销缩放定律构建成本的新框架,计算节省可达 10–100 倍

Amortizing Scaling Law Construction Costs

AI 导读

一项 arXiv 研究提出高效构建缩放定律的框架,将数据收集表述为贝叶斯优化问题,并引入受限计算预算下比较拟合方法的指标。通过逐步扩大获取阶段的计算预算,再用代理幻想评估增强已观测配置,可恢复更广的实验网格。该方法能近似匹配完整密集网格的缩放定律拟合,计算节省最高达 10–100 倍。

正文 · 原文

View PDF HTML (experimental)

Abstract:Scaling laws guide the design choices for training large foundation models, but deriving them involves training an exhaustive grid over hyperparameters, token budgets, and parameter counts, which is computationally expensive. Fitting a scaling law, however, only requires the best-loss frontier across compute scales, discarding most of the trained configurations. We propose a framework for efficient scaling law construction that formulates data collection as a Bayesian optimization problem, and introduce metrics for comparing scaling law fitting methods under constrained compute budgets. We find that progressively expanding the compute budget during acquisition, mirroring the compute-ordered evaluation of configurations in practice, substantially improves recovery efficiency. Augmenting the observed configurations with surrogate-fantasized evaluations then recovers the broader experimental grid, allowing accurate scaling law fitting without training every configuration. Together, these can closely match scaling law fits over a full dense grid at computational savings of up to $10\text{--}100\times$.
Comments: 5 pages, 2 figures, workshop
Subjects: Machine Learning (cs.LG); Artificial Intelligence (cs.AI)
Cite as: arXiv:2609.05016 [cs.LG]
  (or arXiv:2609.05016v2 [cs.LG] for this version)
  https://doi.org/10.48550/arXiv.2609.05016

arXiv-issued DOI via DataCite

Submission history

From: Abhash Jha [view email]
[v1] Fri, 4 Sep 2026 11:25:41 UTC (2,670 KB)
[v2] Tue, 17 Nov 2026 13:09:58 UTC (1,484 KB)

来源:arXiv cs.AI · arxiv.org