V-JEPA-LM:让视频语言模型读出编码器中被浪费的动态信息
V-JEPA-LM: Video-Language Models That Read Dynamics Beyond Static, Text-Aligned Representations
AI 导读
研究者提出 V-JEPA-LM,指出 V-JEPA 等世界模型编码器学到的动态信息未被视频语言模型利用:线性探针可从冻结 V-JEPA 特征以 87% 准确率读出速度,VLM 却接近随机。
正文 · 原文
Abstract:World-model video encoders such as V-JEPA learn how scenes change (direction, speed, order, state) from video alone. To answer questions, they are joined to a language model trained on captions and question-answer pairs. But each caption or question names only a few attributes of a clip, so the model learns to retrieve those attributes rather than to read the clip, and the dynamics the encoder carries go unused: on the same items, a linear probe reads speed from frozen V-JEPA features at 87% while the VLM answers at chance. We show where dynamics lives in a frozen encoder: in a subspace shared across videos for predictive encoders such as V-JEPA, but not as a shared component of frame-wise, text-aligned encoders such as SigLIP. We then build counterfactual supervision that makes the language model read it: pixel transformations turn public videos into answer-balanced variants whose answers are set by the transformation and differ only in the change. Trained in a single stage on 4.53M public samples with 32 visual tokens per frame, where open video LLMs use several stages, 9M to 172M samples and 64 to 256 tokens per frame on average, the resulting V-JEPA-LM models (1B to 8B, Qwen3 and Llama) lead every size class in average score across eleven video benchmarks; our 4B model scores 72.1, against 61.1 for the best open 4B and 64.8 for the best open 8B model. A controlled encoder x supervision study shows that the encoder decides what can be read and the supervision what is read: at a matched budget, counterfactual supervision adds 4.5 points on average and 38 on the speed questions of natural benchmarks. Code and models: this https URL.
| Subjects: | Computer Vision and Pattern Recognition (cs.CV); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2611.17831 [cs.CV] |
| (or arXiv:2611.17831v1 [cs.CV] for this version) | |
| https://doi.org/10.48550/arXiv.2611.17831 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Dongjun Kim [view email]
[v1]
Fri, 13 Nov 2026 07:05:22 UTC (2,103 KB)
来源:arXiv cs.AI · arxiv.org