arXiv cs.AI· Qingyun Liu, Yuankun Liu, Jiaye Feng, Jie Huang, Zonghong Dai, Haochen Li, Jingkun Wang, Weiping Li, Tong Mo·· 4 小时前AI 评分35
PSGAgent：通过智能体推理增强 MLLM 动态视觉感知以实现全场景图生成
PSGAgent: Enhancing MLLMs' Dynamic Visual Perception for Panoptic Scene Graph Generation via Agentic Reasoning
AI 导读
PSGAgent 是一个智能体框架，通过 think-act-observe 循环增强多模态大语言模型在全场景图生成（PSG）任务中的动态视觉感知，解决固定分辨率视角下忽略小目标关系的问题。
正文 · 原文
Abstract:Panoptic Scene Graph generation (PSG) requires holistic yet fine-grained reasoning over complex relations among objects. However, current Multimodal Large Language Models (MLLMs) applied to PSG suffer from a perception bottleneck: they function as static observers limited by fixed-resolution views. As a result, models often overlook critical relations involving small or peripheral objects and struggle to differentiate ambiguous predicates in complex scenes. To address these limitations, we propose PSGAgent, an agentic framework that enhances visual perception for PSG through a think-act-observe loop. PSGAgent allows MLLMs to dynamically refine their visual perception using two tools: a Region Zoom-In Tool for local exploration of small or boundary relations and a Visual Highlight Tool for relation-focused segmentation that filters out irrelevant contextual noise. To optimize decision-making, we construct a two-stage training pipeline. First, we create a high-quality, error-aware Multimodal Chain-of-Thought (M-CoT) dataset that provides robust supervision for cold-start learning. Then, we design a novel hierarchical reward mechanism integrated with Group Sequence Policy Optimization (GSPO) to enable strategic and adaptive tool usage. Extensive experiments on the PSG benchmark show that PSGAgent achieves SOTA performance, demonstrating strong capability in both holistic and fine-grained scene understanding.
| Comments: | Accepted by ACM Multimedia 2026. 10 pages, 8 figures |
| Subjects: | Computer Vision and Pattern Recognition (cs.CV); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2611.07408 [cs.CV] |
| (or arXiv:2611.07408v1 [cs.CV] for this version) | |
| https://doi.org/10.48550/arXiv.2611.07408 arXiv-issued DOI via DataCite (pending registration) |
|
| Related DOI: | https://doi.org/10.1145/3767308.3835827
DOI(s) linking to related resources |
Submission history
From: QingYun Liu [view email]
[v1]
Thu, 5 Nov 2026 04:02:16 UTC (2,730 KB)
来源：arXiv cs.AI · arxiv.org