循环神经网络中的时间秩瓶颈研究
Temporal Rank in Recurrent Neural Networks
AI 导读
一项 arXiv 研究指出,RNN 除梯度消失/爆炸外还存在一个未被充分认识的限制:秩约束的隐藏状态动态造成信息瓶颈,限制可随时间保留的独立输入方向数量。研究者提出结构化高维状态动态方法,保持时间可分性并防止 input-to-state Jacobian 随时间饱和,在受控序列任务上实现快速学习。结论表明仅缓解梯度病理不足以提升 RNN 性能,还须显式维持时间容量。
正文 · 原文
Abstract:Recurrent Neural Networks (RNNs) are a natural framework for temporal modelling, yet are often dismissed due to their sequential, non-parallelisable computation and difficulty in capturing long-range dependencies, typically attributed to vanishing and exploding gradients. We identify a distinct and under-explored limitation: an information bottleneck induced by rank-constrained hidden state dynamics, which limits the number of independent input directions that can be preserved over time, as reflected in the rank of the input-to-state mapping. We show that this constraint restricts temporal capacity independently of gradient stability. To address this, we introduce structured high-dimensional state dynamics that preserve temporal separability and prevent saturation of the input-to-state Jacobian over time. On controlled sequence tasks, the resulting model exhibits rapid learning despite remaining strictly within a standard RNN formulation. Our results demonstrate that RNN performance is governed by the evolution and preservation of temporal rank, and suggest that mitigating gradient pathologies alone is insufficient without explicitly maintaining temporal capacity.
| Subjects: | Machine Learning (cs.LG); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2611.24394 [cs.LG] |
| (or arXiv:2611.24394v1 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2611.24394 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Anton Ragni [view email]
[v1]
Wed, 18 Nov 2026 13:28:07 UTC (3,301 KB)
来源:arXiv cs.AI · arxiv.org