EcoVGGT:基于固定键集推理的长视频高效流式前馈 3D 重建
EcoVGGT: Pinned-Key Inference for Efficient and Streamable Feed-Forward 3D Reconstruction from Long Videos
AI 导读
EcoVGGT 是一种免训练推理框架,用 31 帧均匀分布画面组成固定键集(Pinned Key Set),将长视频的逐帧成本降至常数,解决 VGGT 等前馈多视角 Transformer 跨帧注意力随帧数平方增长的问题。
正文 · 原文
Abstract:Feed-forward multi-view transformers such as VGGT recover the cameras and geometry of a video in one pass, but the cost of their cross-frame attention grows quadratically with the frame count, and their accuracy drops on long videos. Most remedies approximate attention over all frames within a budget or select a few keys by their content. We hypothesize that the keys of a long video can be chosen without looking at its content, and our experiments on VGGT and $\pi^3$ support it: a few dozen evenly spread frames can serve as the keys of every frame. We present EcoVGGT, a training-free inference framework whose keys form a Pinned Key Set of 31 such frames. A pin pass stores their tokens in a Scene Cache, and the other frames read it in independent batches that reproduce joint inference with these keys exactly, at a constant cost per frame. An Online Stream updates the pinned frames during recording. Six middle global blocks align the frames and form a reusable VGGT module: the Alignment Score reads their attention to flag failures, and Alignment Refocusing reduces them by fine-tuning only these blocks (3.6 GPU hours). EcoVGGT ($\pi^3$) matches the success rate of $\pi^3$ on 1,000-frame ScanNet videos at 13.8 times its speed. With refocusing, EcoVGGT (VGGT) raises the success rate of VGGT from 70% to 82% at 8.5 times its speed on 500 frames. EcoVGGT reconstructs 4,000 frames on one GPU, and EcoVGGT ($\pi^3$) streams 1,000 frames at 18.6 frames per second with the lowest online absolute trajectory error.
| Subjects: | Computer Vision and Pattern Recognition (cs.CV); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2611.18099 [cs.CV] |
| (or arXiv:2611.18099v1 [cs.CV] for this version) | |
| https://doi.org/10.48550/arXiv.2611.18099 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Hongsen Cao [view email]
[v1]
Fri, 13 Nov 2026 12:26:54 UTC (17,659 KB)
来源:arXiv cs.AI · arxiv.org