TRAG：面向视觉-语言模型零样本对抗鲁棒性的文本响应感知引导
Textual Response-Aware Guidance for Zero-Shot Adversarial Robustness of Vision-Language Models
AI 导读
论文提出 TRAG（Textual Response-Aware Guidance），利用预训练文本编码器对输入嵌入局部变化的响应方向与幅度构建锚点级约束，并按图文相关性聚合，约束视觉表示漂移。分析指出匹配视觉特征和匹配预测分布两种现有对抗微调引导各有局限。实验显示 TRAG 在多架构、多任务上取得 SOTA 的干净与鲁棒性能，并有助于保持预训练图文关系。
正文 · 原文
Abstract:Vision-language models (VLMs) exhibit strong zero-shot generalization but remain vulnerable to adversarial attacks. Existing adversarial fine-tuning methods often use pre-trained model guidance to preserve image--text alignment while improving robustness. Our analysis of visual representation drift reveals complementary limitations of two representative forms of guidance. Matching visual features constrains all directions equally, providing broad coverage without semantic emphasis. Matching prediction distributions prioritizes semantically relevant directions, but remains limited by the candidate texts and prediction probabilities. These limitations can hinder preservation of pre-trained image--text relationships and degrade zero-shot performance. We therefore propose Textual Response-Aware Guidance (TRAG), which constrains visual representation drift using the pre-trained text encoder's responses to local variations in input embeddings. TRAG constructs anchor-level constraints from the response directions and magnitudes around semantic anchors, then aggregates them using pre-trained image--anchor relevance, yielding an image-adaptive constraint. Extensive experiments across architectures and tasks demonstrate state-of-the-art clean and robust performance. Further analysis shows that TRAG helps preserve pre-trained image--text relationships.
| Subjects: | Computer Vision and Pattern Recognition (cs.CV); Artificial Intelligence (cs.AI); Machine Learning (cs.LG); Multimedia (cs.MM) |
| Cite as: | arXiv:2610.40309 [cs.CV] |
| (or arXiv:2610.40309v2 [cs.CV] for this version) | |
| https://doi.org/10.48550/arXiv.2610.40309 arXiv-issued DOI via DataCite |
Submission history
From: Yubo Cui [view email]
[v1]
Thu, 29 Oct 2026 15:48:43 UTC (459 KB)
[v2]
Mon, 9 Nov 2026 17:35:03 UTC (1,394 KB)
来源：arXiv cs.AI · arxiv.org