arXiv cs.AI· Xinyi Zheng, Yunze Liu, Chi-Hao Wu, Fan Zhang, Hao Zheng, Wenqi Zhou, Walterio W. Mayol-Cuevas, Junxiao Shen·· 3 小时前AI 评分47
SpatialMem：面向语言落地与问答的度量对齐长时程视频记忆系统
SpatialMem: Metric-Aligned Long-Horizon Video Memory for Language Grounding and QA
AI 导读
SpatialMem 是一个将 3D 几何、语义与语言整合为可查询统一表示的记忆系统，仅用日常第一视角 RGB 视频即可重建度量尺度的室内场景，无需专用传感器。
正文 · 原文
Abstract:SpatialMem is a memory-based system which integrates 3D geometry, semantics, and language with one another in one combined, queryable representation. Starting with casually recorded egocentric RGB video, SpatialMem recreates metrically scaled interior scenes, recognizes structural 3D anchors (walls, doors, windows) as the first-layer scaffold, and associates open-vocabulary object nodes with evidence patches, visual embeddings, and two-layer descriptions to 3D coordinates, enabling spatial-relation reasoning, language-directed retrieval, and QA without special sensors. We test at scale on public benchmarks. On VSI-Bench ($288$ videos, $5{,}130$ questions) SpatialMem scores $37.5$ overall with RGB alone, and its margin is precisely where a metric memory should earn it (room size $41.4$ vs.\ $24.4$ with a same-size frame model). On the episodic-memory task of OpenEQA ($953$ questions) the memory serialised to text is lower than the blind control, but as evidence-frame retrieval it recovers frame-level accuracy, matching uniform sampling at an equal eight-frame budget and pulling ahead as the budget tightens, with every answer grounded in linked 3D evidence. On $12$ ScanNet scenes evaluated using instance-level official annotations, the anchor-first hierarchy assigns the retrieved object to the proper structural parent where open-vocabulary 3D scene graphs essentially cannot ($0.24$ compared to $0.01$ with matched input), a capability difference instead of an overall win. The queryable index has a cost of $14$\,KB per scene without the linked evidence frames, and we make available the suites, memories and code.
| Subjects: | Computer Vision and Pattern Recognition (cs.CV); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2601.14895 [cs.CV] |
| (or arXiv:2601.14895v4 [cs.CV] for this version) | |
| https://doi.org/10.48550/arXiv.2601.14895 arXiv-issued DOI via DataCite |
Submission history
From: Xinyi Zheng [view email]
[v1]
Wed, 21 Jan 2026 11:32:24 UTC (6,450 KB)
[v2]
Thu, 5 Mar 2026 22:39:31 UTC (7,055 KB)
[v3]
Thu, 22 Oct 2026 10:53:26 UTC (1,239 KB)
[v4]
Fri, 23 Oct 2026 14:47:39 UTC (4,434 KB)
来源：arXiv cs.AI · arxiv.org