RAWATTA:无需训练的多视角 3D Transformer 注意力加速方法
Training-Free Acceleration of Multi-View 3D Transformers via Regime-Aware Attention Approximation
AI 导读
研究者提出 RAWATTA,一种无需训练的多视角 3D Transformer 全局注意力加速方法。它发现各注意力层分为场景平均、跨图像匹配和首图锚定三种角色,仅用 8-bit 精度运行注意力,并对必需的层保留 16-bit 精度。
正文 · 原文
Abstract:Multi-view 3D transformers such as VGGT reconstruct a scene from a set of unposed images in a single forward pass, predicting camera poses, depth and 3D points, but the cost of their global attention grows quadratically with the num- ber of images. Accelerators borrowed from large language models assume that each token attends to only a few oth- ers; on these models they lose accuracy or gain little speed. We study the global attention of three multi-view transform- ers on seven datasets and find that every layer plays one of three roles: it averages over the whole scene, matches points across many images, or anchors on the first image, whose coordinates the outputs use. Building on this anal- ysis, we propose Regime-AWare ATTention Approximation (RAWATTA), a training-free method that runs attention in 8- bit precision and keeps 16-bit precision only in the layers where a worst-case calibration check requires it. Its fast set- ting, RAWATTA-Fast, also approximates attention in a few averaging and anchoring layers, while matching layers and the first image keep all their keys. On a 12 GB GPU at 50 im- ages, RAWATTA runs VGGT faster than the released code, mostly through our CUDA engine, and loses at most about one point of pose accuracy (AUC@3) on any dataset; RAWATTA-Fast is slightly faster and less accurate. Both fit 200 images, where the released code runs out of memory at 100.
| Subjects: | Computer Vision and Pattern Recognition (cs.CV); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2611.17629 [cs.CV] |
| (or arXiv:2611.17629v1 [cs.CV] for this version) | |
| https://doi.org/10.48550/arXiv.2611.17629 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Muhammed Sezer [view email]
[v1]
Fri, 13 Nov 2026 00:59:07 UTC (1,382 KB)
来源:arXiv cs.AI · arxiv.org