MARBLE:面向空间推理与视觉抽象的智能体基准发布,GPT-5.4 最难设定仅 7.3% 成功率
MARBLE: An Agentic Benchmark for Spatial Reasoning and Visual Abstraction
AI 导读
研究团队发布 MARBLE 基准,通过 CUBE、MAZE、BLOX 三个智能体环境评估多模态模型的空间推理与规划能力,覆盖 2D 与 3D 空间谜题,并沿观察格式和规划视野两个维度控制难度。
正文 · 原文
Abstract:A central challenge for multimodal language models (MLLMs) in real-world tasks is to abstract visual observations into structured spatial states and reason over how those states change under actions over long horizons. Existing multimodal benchmarks often focus on static question answering, where answers can be directly extracted from images, or assume symbolic states are available and bypass the visual-to-spatial abstraction process. We present MARBLE, a diagnostic benchmark for spatially grounded multimodal reasoning and planning across three agent environments: CUBE, MAZE, and BLOX. MARBLE spans 2D and 3D spatial puzzles under geometric constraints, dynamic layouts, and spatial transformations, with difficulty systematically controlled along two axes: observation format and planning horizon. Extensive evaluations show that frontier MLLMs struggle on MARBLE, with the strongest model GPT-5.4 achieving only a 7.3% average success rate across the three environments in the hardest setting. Pairwise ablations on the open-weight models identify visual-to-spatial abstraction as the major bottleneck: when models must infer spatial states from images instead of receiving human-annotated states, success drops from 50.7% to 3.7% for Gemma-4-31B and from 87.9% to 6.5% for Kimi-K2.6. The results indicate that models often fail to construct precise spatial states from visual observations before planning, a task that is straightforward for humans. Overall, MARBLE reveals the gap in reliable visual-to-spatial abstraction for frontier MLLMs and provides a controlled testbed for measuring progress in spatially grounded multimodal reasoning and planning.
| Comments: | Accepted at NeurIPS 2026, E&D Track(in press) |
| Subjects: | Artificial Intelligence (cs.AI); Computation and Language (cs.CL); Computer Vision and Pattern Recognition (cs.CV) |
| Cite as: | arXiv:2506.22992 [cs.AI] |
| (or arXiv:2506.22992v2 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2506.22992 arXiv-issued DOI via DataCite |
Submission history
From: Michael Moor [view email]
[v1]
Sat, 28 Jun 2025 19:44:32 UTC (5,879 KB)
[v2]
Wed, 18 Nov 2026 17:42:16 UTC (7,646 KB)
来源:arXiv cs.AI · arxiv.org