arXiv cs.AI· Ruimeng Liu, Denan Liang, Ruofei Bai, Fuming Zhang, Kuan Xu, Shenghai Yuan, Lihua Xie·· 3 小时前AI 评分40
PhysMoE：基于单 token 坐标表示的模态解耦 VLM 快速空间定位方法
PhysMoE: Fast Spatial Grounding with Single-Token Coordinate Representation in Modality-Decoupled VLMs
AI 导读
PhysMoE 是一种面向具身 VLM 的空间定位架构，通过底层共享、上层按模态路由到空间与语言专家，并用视觉编码器位置编码将每个坐标编码为单 token，同时引入可微 SDF 监督。相比基座模型，PhysMoE 在 9 个坐标定位基准上平均提升 15.7%，吞吐加速 8.31 倍，并在 11 个文本具身基准上保持多模态理解与推理能力。
正文 · 原文
Abstract:Embodied Vision-Language Models (VLMs) are backbones for robotic planning and manipulation. However, for coordinate prediction in grounding tasks, existing architectures suffer from two distinct bottlenecks: discrete tokenization disrupts spatial continuity with high decoding latency, while standard point regression fails on multi-contact affordances and irregularly shaped targets. To resolve these challenges, we propose PhysMoE, introducing three core contributions: (1) sharing lower layers for multimodal context while routing upper layers to specialized spatial (pixel) and linguistic (text) experts by physical modality, eliminating gradient interference; (2) encoding each coordinate as a single token by reusing the visual encoder's learned positional embeddings, injecting 2D spatial priors to natively align coordinates with visual features and ease training; and (3) differentiable Signed Distance Field (SDF) supervision on pointing, accommodating affordances and irregular geometries without costly mask decoders. Compared to the base model, PhysMoE achieves a 15.7% average gain and an 8.31x throughput speedup across 9 coordinate-based grounding benchmarks (trajectory, affordance, and localization), while preserving multimodal understanding and reasoning across 11 text-based embodied benchmarks.
| Subjects: | Computer Vision and Pattern Recognition (cs.CV); Artificial Intelligence (cs.AI); Robotics (cs.RO) |
| Cite as: | arXiv:2611.13000 [cs.CV] |
| (or arXiv:2611.13000v1 [cs.CV] for this version) | |
| https://doi.org/10.48550/arXiv.2611.13000 arXiv-issued DOI via DataCite |
Submission history
From: Ruimeng Liu [view email]
[v1]
Sun, 8 Nov 2026 16:00:02 UTC (969 KB)
来源：arXiv cs.AI · arxiv.org