NoiseFormer：噪声扩散对称注意力 Transformer
NoiseFormer -- Noise Diffused Symmetric Attention Transformer
AI 导读
论文提出 Noise Diffused Symmetric Attention Transformer（NoiseFormer），在 Symmetric Dot-Product Attention 基础上引入噪声扩散注意力，以极小的参数和计算开销，弥补对称注意力相比常规注意力带来的性能损失。
正文 · 原文
Abstract:Transformer architecture has been very successful long runner in the field of Deep Learning (DL) and Large Language Models (LLM) because of its powerful attention-based learning and parallel-natured architecture. As the models grow gigantic in terms of memory footprint, difficulties in fitting the model on a device like a GPU or an AI accelerator give rise to the need for multiple computing devices thereby escalating the computing cost. This increased training/inference cost paved the way for optimizing the memory footprint of the models in terms of activation memory as well as model size reduction. In this paper, we start analyzing one of the Attention architectures aimed at model-size reduction called Symmetric Dot-Product Attention (referred to as Symmetric Attention) and propose a novel unified model architecture called Noise Diffused Symmetric Attention Transformer to enhance the model's performance. While maintaining the memory gains of Symmetric Attention, with minute overhead in terms of model parameters and compute in the form of noise diffused attention, the proposed model recovers the loss in performance due to Symmetric Attention compared to conventional attention as depicted through the inference-time sampling and GLUE scores on downstream tasks.
| Subjects: | Machine Learning (cs.LG); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2601.11619 [cs.LG] |
| (or arXiv:2601.11619v2 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2601.11619 arXiv-issued DOI via DataCite |
Submission history
From: Phani Kumar Nyshadham [view email]
[v1]
Sat, 10 Jan 2026 14:10:48 UTC (537 KB)
[v2]
Mon, 26 Oct 2026 02:26:42 UTC (506 KB)
来源：arXiv cs.AI · arxiv.org