通过单位置回归理解时间序列基础模型（TSFM）中的轴向注意力
Understanding axial attention in TSFMs through single location regression
AI 导读
一项 arXiv 研究将 Marion 等人（2025）提出的单位置回归框架扩展到二维结构化数据，从理论上分析时间序列基础模型中广泛使用的轴向注意力机制。论文构造了一个可解析的轴向注意力型预测器，证明其在渐近机制下可达到 Bayes 最优，而一类半线性简化则严格无法做到；同时证明投影梯度下降在特定温度缩放下全局收敛到最优参数。数值实验验证了理论结果，并凸显逆温度调度对经验收敛的关键作用。
正文 · 原文
Abstract:Recent advances in Time Series Foundation Models (TSFMs) increasingly rely on axial attention mechanisms, but despite their empirical success, the theoretical properties of axial attention remain largely unexplored. To bridge this gap, we extend the Single Location Regression framework introduced in Marion et al. [2025] to 2D structured data, in order to study sparse information settings where a target depends on a small number of tokens. We formulate an analytically tractable axial attention-type predictor and rigorously analyze its statistical properties and training dynamics. We prove that under a specific asymptotic regime and with knowledge of the underlying oracle parameters, our predictor achieves Bayes optimality, whereas a class of semi-linear simplifications strictly fails to do so, underlining the role of the inner non-linearity. We prove that for a specific temperature scaling, Projected Gradient Descent converges globally to the optimal oracle parameters. Extensive numerical experiments validate our theoretical findings and highlight the critical role of inverse temperature scheduling in empirical convergence.
| Subjects: | Machine Learning (stat.ML); Artificial Intelligence (cs.AI); Machine Learning (cs.LG) |
| Cite as: | arXiv:2610.40188 [stat.ML] |
| (or arXiv:2610.40188v1 [stat.ML] for this version) | |
| https://doi.org/10.48550/arXiv.2610.40188 arXiv-issued DOI via DataCite |
Submission history
From: Andrei-Tiberiu Pantea [view email]
[v1]
Thu, 29 Oct 2026 14:46:50 UTC (19,724 KB)
来源：arXiv cs.AI · arxiv.org