ELinFormer：通过熵校准实现面向多变量时间序列的高效线性注意力
Efficient Linear Attention for Multivariate Time Series Modeling via Entropy Calibration
AI 导读
研究者提出熵校准线性注意力模块，并构建为 ELinFormer 预测器，将信息聚合分解为均匀分量与查询相关修正，用查询特定温度校准二阶 KL 代理。在五个 PEMS 基准上取得有竞争力的预测精度，与 vanilla 注意力和 Linformer 相比在精度与内存间取得更优权衡，PEMS08 上其有理温度规则和查询特定校准的 MAE 低于线性规则或共享温度方案。
正文 · 原文
Abstract:Attention mechanisms are effective for multivariate time series modeling because they adaptively aggregate information across timestamps and variables. Their quadratic cost, however, limits their use in multivariate and spatio-temporal systems. Linear attention reduces this cost by replacing nonlinear attention mappings with forms that support associative computation. Replacing exponential weighting may weaken the amplification of high-score positions and further bias weakly selective distributions toward uniform aggregation. We propose an entropy-calibrated linear attention module and instantiate it as the ELinFormer forecaster. Our operator decomposes aggregation into a uniform component and a query-dependent correction, whose magnitude is calibrated by a query-specific temperature. We derive a second-order KL proxy from centered query-key scores. The Taylor expansion reduces the proxy to the squared-score energy divided by the squared temperature. This score energy can be computed from shared products of the centered keys without forming the full pairwise score matrix, and the same products support the linear attention output. We design a smooth temperature rule to calibrate the KL proxy: it keeps low-energy residuals vanishing, approximately calibrates moderate-energy queries to a shared KL level, and allows the proxy to continue growing for high-energy queries. Experiments on five PEMS benchmarks show competitive forecasting accuracy. Controlled comparisons with vanilla attention and Linformer demonstrate a favorable accuracy--memory trade-off, while analysis on PEMS08 shows that the proposed rational temperature rule and query-specific calibration achieve lower MAE than a linear rule or shared-temperature alternatives. These results support entropy calibration as an effective mechanism for preserving selective deviations in linear attention.
| Comments: | 10-page main paper with a 6-page supplement, 4 figures in total. Version 2 revises the theoretical formulation and proofs, adds a high-energy interpretation and controlled NaLaFormer comparison, and expands reproducibility material, and the core method is unchanged. Code on this https URL |
| Subjects: | Machine Learning (cs.LG); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2511.03190 [cs.LG] |
| (or arXiv:2511.03190v2 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2511.03190 arXiv-issued DOI via DataCite |
Submission history
From: Mingtao Zhang [view email]
[v1]
Wed, 5 Nov 2025 05:07:55 UTC (54 KB)
[v2]
Sat, 24 Oct 2026 16:22:36 UTC (377 KB)
来源：arXiv cs.AI · arxiv.org