树集成模型的最小极大收敛率与谱蒸馏方法
Minimax Rates and Spectral Distillation for Tree Ensembles
AI 导读
一篇 NeurIPS 论文从谱视角研究随机森林（RF）与梯度提升机（GBM）两类树集成模型。论文推导出 RF 回归的最小极大最优收敛率，证明核算子的特征值衰减决定统计速率；并利用 RF 核算子的主特征函数与 GBM 平滑矩阵的主奇异向量学习非线性映射，得到比原模型小几个数量级、性能仍有竞争力的蒸馏模型，效果优于最先进的森林剪枝与规则提取算法。
正文 · 原文
Abstract:Tree ensembles such as random forests (RFs) and gradient boosting machines (GBMs) are among the most widely used supervised learners, yet their theoretical properties remain incompletely understood. We adopt a spectral perspective on these algorithms, with two main contributions. First, we derive minimax-optimal convergence for RF regression, showing that, under mild regularity conditions on tree growth, the eigenvalue decay of the induced kernel operator governs the statistical rate. Second, we exploit this spectral viewpoint to develop compression schemes for tree ensembles. For RFs, leading eigenfunctions of the kernel operator capture the dominant predictive directions; for GBMs, leading singular vectors of the smoother matrix play an analogous role. Learning nonlinear maps for these spectral representations yields distilled models that are orders of magnitude smaller than the originals while maintaining competitive predictive performance. Our methods compare favorably to state of the art algorithms for forest pruning and rule extraction, with applications to resource constrained computing.
| Comments: | NeurIPS camera ready version. 10 pages main text, 48 pages total (excluding checklist), with 12 figures and 10 tables total |
| Subjects: | Machine Learning (stat.ML); Artificial Intelligence (cs.AI); Machine Learning (cs.LG) |
| Cite as: | arXiv:2605.11841 [stat.ML] |
| (or arXiv:2605.11841v3 [stat.ML] for this version) | |
| https://doi.org/10.48550/arXiv.2605.11841 arXiv-issued DOI via DataCite |
Submission history
From: Binh Vu [view email]
[v1]
Tue, 12 May 2026 09:26:00 UTC (2,627 KB)
[v2]
Sun, 8 Nov 2026 20:04:41 UTC (2,634 KB)
[v3]
Tue, 10 Nov 2026 11:02:27 UTC (2,634 KB)
来源：arXiv cs.AI · arxiv.org