正交循环神经网络量化研究：提出 QORNN

View PDF

Abstract:In recent years, Orthogonal Recurrent Neural Networks (ORNNs) have gained popularity due to their ability to manage tasks involving long-term dependencies, such as the Copy-task, and their linear complexity. However, existing ORNNs utilize full precision weights and activations, which prevents their deployment on compact devices. In this paper, we explore the quantization of the weight matrices in ORNNs, leading to Quantized approximately Orthogonal RNNs (QORNNs). The construction of such networks remained an open problem, acknowledged for its inherent instability. We propose and investigate two strategies to learn QORNN by combining quantization-aware training (QAT) and orthogonal projections. We also study post-training quantization of the activations for pure integer computation of the recurrent loop. The most efficient fully-quantized models are currently the only alternatives for solving the Copy-task with 1000 timesteps. They also surpass existing quantized models on tasks requiring long-term dependencies, such as permuted MNIST as well as some Internet of Things (IoT) tasks. However, they are less effective on tasks where long short-term memory (LSTMs) and other gated RNNs outperform ORNNs.

Comments:	Progress in Artificial Intelligence, In press
Subjects:	Neural and Evolutionary Computing (cs.NE); Artificial Intelligence (cs.AI); Machine Learning (cs.LG); Signal Processing (eess.SP); Statistics Theory (math.ST)
Cite as:	arXiv:2402.04012 [cs.NE]
	(or arXiv:2402.04012v3 [cs.NE] for this version)
	https://doi.org/10.48550/arXiv.2402.04012 arXiv-issued DOI via DataCite

Submission history

From: Francois Malgouyres [view email] [via CCSD proxy]
[v1] Mon, 5 Feb 2024 09:59:57 UTC (3,509 KB)
[v2] Mon, 10 Jun 2024 11:40:40 UTC (2,492 KB)
[v3] Fri, 23 Oct 2026 16:19:05 UTC (420 KB)