个性化图像分割新方法"Where is My Mug?"：免训练框架突破单实例评测局限
Where is My Mug? Training-Free Framework for Personalized Image Segmentation
AI 导读
一篇 arXiv 论文提出个性化图像分割（PIS）的免训练框架，指出既有方法在含多个同类相似物体的真实场景中会失效，瓶颈在于召回而非掩码质量。该系统用现成类别无关检测器生成候选框，以 BLIP-2 特征匹配参考图并提出 top-K Sinkiron 部分最优传输距离，再用 SAM 精修，成本降低超过 100 倍。
正文 · 原文
Abstract:\textbf{Personalized Image Segmentation (PIS)} aims to segment the specific object in a target image that matches a user-provided reference. Unlike category-based segmentation or few-shot segmentation, PIS must discriminate between multiple visually similar instances, a requirement central to deployed applications such as augmented reality, personalized content editing, robotics manipulation, inventory tracking, and fine-grained visual retrieval. We show that the strong numbers reported by existing PIS methods are largely an artifact of single-instance benchmarks: when evaluated on realistic scenes with several same-category look-alikes, prior point-prompt methods collapse, because the correct instance is frequently never surfaced as a prompt. We trace this to a recall bottleneck rather than a mask-quality one, and address it with a training-free, modular system. The pipeline first generates high-recall object proposals using an off-the-shelf class-agnostic detector, then matches each proposal to the reference using foundation-model features (e.g., BLIP-2), and finally refines the selected box with SAM. For robust matching we propose \textbf{top-$K$ Sinkhorn}, a patch-level distance that scores each candidate only on its most confidently matched patches; trimming the Sinkhorn dual potentials recovers the robustness of partial optimal transport at more than $100\times$ lower cost, keeping it inside the per-proposal candidate loop. To evaluate this regime, we introduce \textbf{\datasetname}, a realistic multi-instance benchmark. Across \datasetname, PerSeg, PerMIS, and DAVIS, our system surpasses PerSAM, $\mathrm{P}^2\mathrm{SAM}$, and PDM by up to $\sim\!48$ mIoU while matching their runtime and using fewer parameters, establishing a strong, deployable training-free baseline for real-world personalized segmentation.
| Subjects: | Computer Vision and Pattern Recognition (cs.CV); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2611.15216 [cs.CV] |
| (or arXiv:2611.15216v1 [cs.CV] for this version) | |
| https://doi.org/10.48550/arXiv.2611.15216 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Quang Qui-Vinh Nguyen [view email]
[v1]
Wed, 11 Nov 2026 09:09:54 UTC (20,243 KB)
来源：arXiv cs.AI · arxiv.org