LBS：用 GUI grounding 训练超越屏幕的可迁移视觉观察

View PDF HTML (experimental)

Abstract:Visual perception is evolving from passive recognition toward active, task-oriented observation. We investigate GUI grounding as a scalable source of spatial supervision for learning observation behaviors that transfer beyond interfaces. We introduce LBS (Look Beyond Screens), a vision-language model trained through supervised initialization on teacher-generated GUI grounding trajectories, followed by reinforcement learning for multi-turn crop-and-inspect decisions. Under fixed inference settings, LBS models based on Qwen3.5-35B-A3B and Qwen3.8-27B consistently improve over their corresponding baselines. Gains beyond supervised fine-tuning transfer to PerceptionBench for both backbones and to HR-Bench and ChartQAPro for Qwen3.5-35B-A3B, supporting observation strategies that generalize beyond GUI-specific imitation. Further analysis suggests complementary roles for the two stages: SFT establishes visual inspection primitives, while RL improves their downstream effectiveness. These findings support GUI grounding as a practical supervision paradigm for transferable active visual perception and broader multimodal reasoning. Our code and model are publicly available at this https URL and this https URL, respectively.

Comments:	Technical Report
Subjects:	Computer Vision and Pattern Recognition (cs.CV); Artificial Intelligence (cs.AI)
Cite as:	arXiv:2610.37093 [cs.CV]
	(or arXiv:2610.37093v1 [cs.CV] for this version)
	https://doi.org/10.48550/arXiv.2610.37093 arXiv-issued DOI via DataCite (pending registration)

Submission history

From: Ruilin Yao [view email]
[v1] Tue, 27 Oct 2026 11:56:00 UTC (6,751 KB)