arXiv cs.AI· Haofeng Wang, Pengcheng Wen, Ruiqi Li, Yuhao Zhang, Tiange Zhang, Boren Zheng, Xuanyi Liu, Siyu Tao, Danni Yu, Gan Lin, Haocheng Tang, Tianzizhe Guo, Mingyuan Su, Boyuan Chen, Zhongyuan Wang·· 3 小时前AI 评分35
PhysCapBench：面向物理性视频描述的跨领域基准
PhysCapBench: A Cross-Domain Benchmark for Physically Grounded Video Captioning
AI 导读
研究者推出 PhysCapBench，据称为首个覆盖第三人称场景、第一人称操作和手持夹爪演示三类领域的物理描述基准，提供 1,500 条视频的人工校验标注，从七个物理维度评估准确性与覆盖度。物理描述使第三人称视频重建 LPIPS 降低 8.6-9.5%、第一人称视频降低 26.9-30.9%，训练指令增强使 pi0.5 在 RoboTwin 2.0 子任务上提升 7.5 个百分点。
正文 · 原文
Authors:Haofeng Wang, Pengcheng Wen, Ruiqi Li, Yuhao Zhang, Tiange Zhang, Boren Zheng, Xuanyi Liu, Siyu Tao, Danni Yu, Gan Lin, Haocheng Tang, Tianzizhe Guo, Mingyuan Su, Boyuan Chen, Zhongyuan Wang
Abstract:Physical AI aims to learn transferable physical knowledge from real-world experience to understand, predict, and act in the physical world. Internet videos and embodied demonstrations provide complementary observations of motion, interaction, and state evolution. A foundational data challenge is to measure and preserve this physical information as training supervision. In video-language pretraining, captions express observed processes in language, yet a semantically correct description can omit the interactions and transitions that define those processes. Rich physical experience in video does not automatically translate into sufficient language supervision. This gap calls for a yardstick to assess annotation quality, compare captioning strategies, and guide training-data refinement. Motivated by our data construction for world-model pretraining, we introduce PhysCapBench, to our knowledge the first unified physical-caption benchmark spanning all three domains: third-person scenes, egocentric manipulation, and handheld-gripper demonstrations. It provides human-verified annotations for 1,500 video sequences and evaluates accuracy and coverage through seven physical dimensions. We examine downstream utility through video reconstruction and manipulation-policy fine-tuning. Physical captions reduce reconstruction LPIPS by 8.6-9.5% on third-person videos and 26.9-30.9% on egocentric videos relative to metadata captions. Training-instruction enrichment improves both PhysCapBench scores and policy success, including a 7.5-percentage-point gain for pi0.5 on a RoboTwin 2.0 subtask. PhysCapBench makes the physical content of language supervision inspectable and comparable, providing a cross-domain basis for its evaluation and improvement.
| Comments: | Project page: this https URL |
| Subjects: | Computer Vision and Pattern Recognition (cs.CV); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2611.16424 [cs.CV] |
| (or arXiv:2611.16424v1 [cs.CV] for this version) | |
| https://doi.org/10.48550/arXiv.2611.16424 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Haofeng Wang [view email]
[v1]
Thu, 12 Nov 2026 07:33:09 UTC (22,509 KB)
来源：arXiv cs.AI · arxiv.org