炎症性肠病全切片图像与临床数据的多模态融合方法比较研究
Multimodal Fusion Methods for Whole-Slide Images and Clinical Data in Inflammatory Bowel Disease
挪威斯塔万格大学团队提出模块化多模态融合框架，将冻结基础模型的切片级嵌入与每次就诊的临床数据嵌入，通过 15 种可互换融合方法结合统一预测头，预测新确诊炎症性肠病患者的 STRIDE-II 对齐缓解结局。最佳方法 macro-F1 达 0.57-0.75，在六种场景中四种优于单模态模型，且性能相近的方法对切片图像的依赖程度差异很大。
Authors:Ankana Banerjee (1), Mark van der Giezen (1, 2, 3), Ole Gunnar Aasprong (2), Emiel A. M. Janssen (1, 2), Øyvind Steinsbø (2, 4), Tore Grimstad (2, 4), Kjersti Engan (1) ((1) University of Stavanger, Norway, (2) Stavanger University Hospital, Norway, (3) University of Greenwich, UK, (4) University of Bergen, Norway)
Abstract:Multimodal AI fusion of histology and clinical data is an active research area in computational pathology, but fusion methods are rarely compared under identical conditions and across different clinical endpoints. We present a modular framework in which slide-level embeddings, obtained by training an attention-based aggregator over patch embeddings from a frozen foundation model, and per-visit clinical embeddings are fed into 15 interchangeable fusion methods with a shared output dimension and prediction head (except decision-level methods). Using a prospective cohort of newly diagnosed inflammatory bowel disease patients, we predicted STRIDE-II-aligned remission outcomes from baseline or follow-up visits. The best fusion method reached macro-F1 of 0.57-0.75, outperforming both single-modality models in four of six scenarios, and differed by outcome and input. Image reliance, the drop in balanced accuracy when the slide embedding is removed, tended to follow the remission type - highest for histological and lowest for endoscopic remission; methods with similar accuracy relied on the slide to very different degrees.
| Comments: | 5 pages, 4 figures, Submitted to the 2027 IEEE 24th International Symposium on Biomedical Imaging (ISBI) |
| Subjects: | Computer Vision and Pattern Recognition (cs.CV); Artificial Intelligence (cs.AI); Machine Learning (cs.LG) |
| Cite as: | arXiv:2611.05579 [cs.CV] |
| (or arXiv:2611.05579v1 [cs.CV] for this version) | |
| https://doi.org/10.48550/arXiv.2611.05579 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Ankana Banerjee [view email]
[v1]
Tue, 3 Nov 2026 20:11:07 UTC (4,655 KB)
来源：arXiv cs.AI · arxiv.org