arXiv cs.AI· Daeun Lee, Subhojyoti Mukherjee, Branislav Kveton, Ryan A. Rossi, Viet Dac Lai, Seunghyun Yoon, Trung Bui, Franck Dernoncourt, Mohit Bansal·· 4 小时前AI 评分41
StreamGaze：首个面向流式视频注视引导时序推理与主动理解的基准
StreamGaze: Gaze-Guided Temporal Reasoning and Proactive Understanding in Streaming Videos
AI 导读
首个基准 StreamGaze 发布，用于评估多模态大语言模型（MLLM）在流式视频中利用人类注视信号进行时序推理与主动预测的能力。该基准包含注视引导的过去、现在与主动任务，通过注视轨迹对齐自我中心视频构建问答数据。实验显示当前 SOTA MLLM 与人类表现存在明显差距，数据与代码已公开，论文被 NeurIPS 2026 接收。
正文 · 原文
Abstract:Streaming video understanding requires models not only to process temporally incoming frames, but also to anticipate user intent for realistic applications such as Augmented Reality (AR) glasses. While prior streaming benchmarks evaluate temporal reasoning, none measure whether Multimodal Large Language Models (MLLMs) can interpret or leverage human gaze signals within a streaming setting. To fill this gap, we introduce StreamGaze, the first benchmark designed to evaluate how effectively MLLMs utilize gaze for temporal and proactive reasoning in streaming videos. StreamGaze introduces gaze-guided past, present, and proactive tasks that comprehensively assess streaming video understanding. Past and present tasks require immediate responses based on observations available at query time, whereas proactive tasks require models to continuously process incoming frames and respond at the appropriate future moment. To build StreamGaze, we develop a gaze-video Question Answering (QA) generation pipeline that aligns egocentric videos with raw gaze trajectories through fixation extraction, region-specific visual prompting, and scanpath construction. This pipeline produces spatio-temporally grounded QA pairs that reflect human perceptual dynamics. Across all StreamGaze tasks, we observe substantial performance gaps between state-of-the-art MLLMs and human performance, highlighting key limitations in gaze-based temporal reasoning, intent modeling, and proactive prediction. We further provide detailed analyses of gaze prompting strategies, reasoning behaviors, and task-specific failure modes, offering insights into current limitations and directions for future research. All data and code are publicly available to support research in gaze-guided streaming video understanding.
| Comments: | Accepted to the 40th Annual Conference on Neural Information Processing Systems (NeurIPS 2026) |
| Subjects: | Computer Vision and Pattern Recognition (cs.CV); Artificial Intelligence (cs.AI); Computation and Language (cs.CL) |
| Cite as: | arXiv:2512.01707 [cs.CV] |
| (or arXiv:2512.01707v4 [cs.CV] for this version) | |
| https://doi.org/10.48550/arXiv.2512.01707 arXiv-issued DOI via DataCite |
Submission history
From: Daeun Lee [view email]
[v1]
Mon, 1 Dec 2025 14:15:44 UTC (4,693 KB)
[v2]
Fri, 27 Mar 2026 17:30:08 UTC (4,748 KB)
[v3]
Wed, 13 May 2026 04:12:19 UTC (4,748 KB)
[v4]
Thu, 5 Nov 2026 15:52:54 UTC (5,323 KB)
来源：arXiv cs.AI · arxiv.org