arXiv cs.AI· Hunar Batra, Lachin Naghashyar, Ashkan Khakzar, Philip Torr, Christian Schroeder de Witt, Constantin Venhoff, Ronald Clark·· 4 小时前AI 评分48
MMDiff：面向多模态模型特征发现与控制的多模态模型差分框架
Multimodal Model Diffing for Feature Discovery and Control
AI 导读
一项 NeurIPS 2026 接收的 arXiv 研究提出 MMDiff，一个通过训练多模态 SAE 实现特征级发现与控制多模态行为的模型差分框架。该框架在 LLaVA-MORE、PaliGemma 2 和 InternVL3.5 三个多模态模型家族上训练多模态 SAE，支持特征隔离、任务特定特征检测和特征级控制。
正文 · 原文
Abstract:Multimodal Large Language Models (MLLMs) exhibit strong visual understanding, yet the internal features that cause these behaviors remain difficult to identify, audit, or control. While applicable to post-hoc inspection, hidden states that are decomposed into interpretable feature directions using sparse autoencoders (SAEs) neither readily isolate which features are changed by multimodal training, nor are they directly useful for targeted control. We introduce MMDiff, a multimodal model-diffing framework that trains multimodal SAEs and turns them into feature-level interfaces for discovering and controlling multimodal behavior. MMDiff supports three uses: (i) feature isolation, by diffing a base-LM SAE against its multimodal-adapted counterpart to identify features altered by multimodal training; (ii) task-specific feature detection, via per-token contrastive firing analysis that isolates causal features; and (iii) feature-level control, by causally removing or steering the discovered feature directions. We train multimodal SAEs for three MLLM families, LLaVA-MORE, PaliGemma 2, and InternVL3.5, and evaluate on visual-spatial understanding, multimodal safety, and OCR. MMDiff discovers sparse, causally specific features whose removal selectively degrades target behaviors by an average of 12% on spatial tasks and 17% on OCR, and reduces attack success rate by 24% on multimodal safety attacks, with no impact on VQA performance. Steering these features improves spatial and OCR accuracy by +3.6% and +1.8% on average over a standard single-layer steering baseline. These results show that multimodal SAEs can serve not only as interpretability tools, but as mechanisms for auditing, steering, and controlling MLLMs behavior toward safer and more capable generations.
| Comments: | Accepted at the 40th Conference on Neural Information Processing Systems (NeurIPS 2026) |
| Subjects: | Computer Vision and Pattern Recognition (cs.CV); Artificial Intelligence (cs.AI); Computation and Language (cs.CL); Machine Learning (cs.LG) |
| Cite as: | arXiv:2608.09928 [cs.CV] |
| (or arXiv:2608.09928v2 [cs.CV] for this version) | |
| https://doi.org/10.48550/arXiv.2608.09928 arXiv-issued DOI via DataCite |
Submission history
From: Hunar Batra [view email]
[v1]
Mon, 10 Aug 2026 17:59:30 UTC (43,197 KB)
[v2]
Tue, 10 Nov 2026 18:59:47 UTC (47,583 KB)
来源：arXiv cs.AI · arxiv.org