arXiv cs.AI· Zhaoyi Liu, Haichuan Zhang, Xiaochuan Gong, Yancheng Zhu, Bowei Tian, Wanghao Ye, Yufan Zhang, Ang Li·· 2 小时前AI 评分41
解耦视觉语言模型中的多模态旋转位置编码 MRoPE
Disentangling Multimodal Rotary Positional Embedding in Vision-Language Models
AI 导读
一项 arXiv 研究解耦了 MRoPE 中视觉内部空间编码与跨模态空间效应，通过多模型规模、多种 MRoPE 变体和多个 benchmark 上的受控实验发现：MRoPE 给文本 token 赋予的高度和宽度坐标带来的跨模态空间效应几乎没有额外益处，甚至损害模型性能；而视觉内部空间编码仍对细粒度 grounding 有效。研究还指出了共享坐标布局的约束是难以消除跨模态空间效应的原因。
正文 · 原文
Abstract:Vision-language models widely adopt Multimodal Rotary Position Embedding (MRoPE), an extension of Rotary Position Embedding (RoPE) that incorporates temporal, height, and width axes to encode the spatial structure of visual tokens. However, MRoPE also assigns height and width coordinates to text tokens that lack inherent spatial structure, introducing spatial positional effects into text-vision attention. Existing MRoPE variants explore different frequency allocations and coordinate layouts, with their designs suggesting different intuitions about these cross-modal effects: they may provide useful spatial references for associating language with visual regions, or introduce unnecessary positional preferences that hinder text-vision interactions. In this work, we disentangle intra-visual spatial structure from cross-modal spatial effects to investigate this question. We identify a constraint of shared coordinate layouts that prevents eliminating cross-modal spatial effects while preserving intra-visual spatial encoding, and develop controlled experiments to isolate their respective contributions. Our results across multiple model scales, MRoPE variants, and benchmarks show that cross-modal spatial effects provide little additional benefit as spatial references and can even impair model performance, while intra-visual spatial encoding remains useful for fine-grained grounding.
| Subjects: | Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2611.13334 [cs.AI] |
| (or arXiv:2611.13334v1 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2611.13334 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Zhaoyi Liu [view email]
[v1]
Mon, 9 Nov 2026 23:00:25 UTC (13,237 KB)
来源：arXiv cs.AI · arxiv.org