谱神经网络（SNN）与非线性矩阵感知下的隐式正则化研究

View PDF HTML (experimental)

Abstract:The phenomenon of implicit bias has attracted interest in recent years as a fundamental aspect of the remarkable generalizing ability of neural networks. In a nutshell, it entails that gradient descent dynamics in many neural nets, even without any explicit regularizer in the loss function, converges to the solution of a regularized learning problem.
In this work, we show that this phenomenon extends to non-linear spectral parameterizations in matrix sensing. More precisely, we propose a matrix network architecture called Spectral Neural Networks (abbrv. SNN) where each neuron applies nonlinear activation to the singular values of the input matrix. Conceptually, this entails coordinatizing the space of matrices by their singular values and singular vectors, as opposed to by their entries, a potentially fruitful perspective for matrix learning. We prove the exponentially fast decay of squared loss during the training dynamics of an SNN parameterized matrix sensing model under gradient flow (which is a continuous approximation of the discrete gradient descent), and also that the training implicitly selects an approximate solution that minimizes nuclear norm. We also demonstrate the effectiveness of the SNN architecture in the context of matrix sensing by empirical investigations. We believe that the SNN architecture has the potential to be of wide applicability in a broad class of matrix learning scenarios.

Subjects:	Machine Learning (cs.LG); Artificial Intelligence (cs.AI); Neural and Evolutionary Computing (cs.NE); Machine Learning (stat.ML)
Cite as:	arXiv:2402.17595 [cs.LG]
	(or arXiv:2402.17595v2 [cs.LG] for this version)
	https://doi.org/10.48550/arXiv.2402.17595 arXiv-issued DOI via DataCite

Submission history

From: Xiaoyu Dong [view email]
[v1] Tue, 27 Feb 2024 15:28:01 UTC (3,491 KB)
[v2] Thu, 5 Nov 2026 18:23:13 UTC (2,888 KB)