arXiv cs.AI· Johannes Hog, Neeratyoy Mallik, Steven Adriaensen, Jan Hendrik Metzen, Tom Viering, Danny Stoll, Frank Hutter·· 2 小时前AI 评分36
面向表格基础模型的可靠 Scaling Laws 研究
Towards Reliable Scaling Laws for Tabular Foundation Models
AI 导读
一项 arXiv 研究检验 TabPFNv2 类表格基础模型的 scaling laws 能否可靠外推到更大算力并泛化到真实数据集。分析发现合成与真实数据的 scaling 行为存在差异,仅用过少评测数据集选型会带来系统性高估。研究表明算力最优扩展需随模型规模同步扩大合成预训练数据集的行数与特征数范围,以借助开源的 nanoTabPFN 训练管线训练更高效的模型。
正文 · 原文
Abstract:Scaling laws are widely used to predict foundation model performance and guide compute allocation. However, tabular foundation models (TFMs) differ from conventional foundation models in that they are pretrained on virtually unlimited synthetic data from configurable distributions, while ultimately being evaluated on real-world datasets. We investigate whether scaling laws for TabPFNv2-style models reliably extrapolate to larger compute budgets and generalize across evaluation settings. Our analysis reveals differences between synthetic and real-world scaling behavior and shows that accurately fitting a scaling law does not necessarily yield reliable prescriptions on unseen datasets. In particular, selecting configurations using too few evaluation datasets leads to systematically over-optimistic performance estimates. We further demonstrate that compute-optimal scaling requires expanding the range of synthetic pretraining dataset sizes alongside model size and training compute. Increasing the maximum numbers of rows and features sampled during pretraining improves performance and produces more predictable scaling trends. These insights guide the training of more efficient TabPFNv2-like models, using the open-source training pipeline of nanoTabPFN.
| Subjects: | Machine Learning (cs.LG); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2611.24214 [cs.LG] |
| (or arXiv:2611.24214v1 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2611.24214 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Johannes Hog [view email]
[v1]
Wed, 18 Nov 2026 10:52:48 UTC (499 KB)
来源:arXiv cs.AI · arxiv.org