arXiv cs.AI· Ziniu Zhang, Zhenshuo Zhang, Jianglin Lu, Ruoxuan Xiong, Yun Fu, Hongyang R. Zhang·· 3 小时前AI 评分33
结构化核代理模型（SKS）用于多模态归因
Structured Kernel Surrogate Models for Multimodal Attribution
AI 导读
arXiv 论文提出结构化核代理模型 SKS，用结构化组合核的核岭回归捕捉跨样本、跨模态的归因交互。在涵盖卫星图像、路网、文本、音频与表格数据的两个多模态基准上，SKS 与真实预测结果的相关性比现有方法高 57.8%，下游数据选择平均准确率提升 1.93%，且所选数据训练的模型 Hessian 迹更低、损失面更平坦。
正文 · 原文
Abstract:We study the problem of cross-sample modality attribution, which involves measuring how one modality in one sample affects the learning outcome in other multimodal samples. This problem is crucial in a multimodal system because a data modality (such as a text description) may be redundant for one sample (a self-explanatory image), but the same data modality would otherwise be uniquely informative for another sample, or informative when combined with other modalities. Prior work focuses on influence estimation at the level of one sample, and it remains an open question how one should capture the interactions across both samples and modalities. To capture scenarios like the above, we propose structured kernel surrogate models, which are surrogate mappings from both samples and modalities to the prediction outcome when they are trained together. We design an algorithm, SKS, which applies kernel ridge regression with a structured, compositional kernel on the selection states of individual samples and modalities. Specifically, the kernel function multiplies kernel-based similarity scores between every pair of data modalities for one sample and a pair of two samples. Additionally, we design a gradient-based estimation algorithm to efficiently estimate the structured kernel surrogate model. We evaluate our approach on two multimodal benchmarks, spanning a range of data modalities including satellite images, road networks, text, audio, and tabular data. We find that SKS achieves $57.8$% higher correlation with the true prediction outcomes than existing methods. In downstream data selection, SKS improves the average accuracy by $1.93$% over baselines. Models with data selected by SKS have lower Hessian traces, indicating a flatter loss surface. Using SKS to select satellite images identifies road segments, where accident scenes commonly occur, as opposed to background images.
| Comments: | 20 pages. To appear at NeurIPS 2026 |
| Subjects: | Machine Learning (cs.LG); Artificial Intelligence (cs.AI); Computer Vision and Pattern Recognition (cs.CV) |
| Cite as: | arXiv:2611.13198 [cs.LG] |
| (or arXiv:2611.13198v1 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2611.13198 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Ziniu Zhang [view email]
[v1]
Mon, 9 Nov 2026 20:21:39 UTC (3,771 KB)
来源：arXiv cs.AI · arxiv.org