SPY-Bench 评估大视觉语言模型的推理时个性化隐私保护能力
Specify Privacy Yourself: Assessing Inference-Time Personalized Privacy Preservation Ability of Large Vision-Language Models
AI 导读
研究者提出推理时个性化隐私保护（ITP³）任务，并发布首个系统性评测基准 SPY-Bench，包含 32,700 个样本、67 个类别和 24 个真实场景。对 21 个 SOTA LVLM 的评测显示，即使表现最好的 o4-mini 合规准确率也仅约 24%，上下文隐私理解能力有限。
正文 · 原文
Abstract:Large Vision-Language Models (LVLMs) have demonstrated remarkable capabilities but raise significant privacy concerns due to their abilities to infer sensitive personal information from images with high precision. While current LVLMs are relatively well aligned to protect universal privacy, e.g., credit card data, we argue that privacy is inherently personalized and context-dependent. This work pivots towards a novel task: can LVLMs achieve Inference-Time Personalized Privacy Protection (ITP$^3$), allowing users to dynamically specify privacy boundaries through language specifications? To this end, we present SPY-Bench, the first systematic assessment of ITP$^3$ ability, which comprises (1) 32,700 unique samples with image-question pairs and personalized privacy instructions across 67 categories and 24 real-world scenarios, and (2) novel metrics grounded in user specifications and context awareness. Benchmarking the ITP$^3$ ability of 21 SOTA LVLMs, we reveal that: (i) most models, even the top-performing o4-mini, perform poorly, with only $\sim$24% compliance accuracy; (ii) they show quite limited contextual privacy understanding capability. Therefore, we implemented initial ITP$^3$ alignment methods, including a novel Noise Contrastive Alignment variant which achieves 96.88% accuracy while maintaining reasonable general performance. These results mark an initial step towards the ethical deployment of more controllable LVLMs. Code and data are available at this https URL
| Comments: | Accepted by ACM Multimedia 2025 (Brave New Ideas track) |
| Subjects: | Computer Vision and Pattern Recognition (cs.CV); Artificial Intelligence (cs.AI); Computation and Language (cs.CL); Computers and Society (cs.CY); Machine Learning (cs.LG) |
| Cite as: | arXiv:2611.13997 [cs.CV] |
| (or arXiv:2611.13997v1 [cs.CV] for this version) | |
| https://doi.org/10.48550/arXiv.2611.13997 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Xingqi Wang [view email]
[v1]
Tue, 10 Nov 2026 12:41:02 UTC (5,201 KB)
来源：arXiv cs.AI · arxiv.org