跳到正文
arXiv cs.AI· Huu Binh Ta, Alex Morehead, N. Benjamin Erichson, Ferdinando Fioretto, Sandeep Madireddy·· 2 小时前AI 评分29

任务感知融合 TCAF:为材料科学多模态语言模型选择更优表征

Choosing the Right Representations: Task-Aware Fusion for Multimodal Materials Language Models

AI 导读

TCAF(Task-Conditioned Attention Fusion)是一种轻量级注意力桥接模块,可根据目标属性自适应组合多个结构编码器的表征,用于材料科学多模态语言模型。它直接在下游任务上训练,免去对比对齐阶段,相比 MatterChat 最多降低 47.8% 的 RMSE,可训练参数更少且训练时间减半。加入 UMA-S 的三编码器配置取得 89.61% 的最高宏观平均分类准确率。

正文 · 原文

View PDF HTML (experimental)

Abstract:Multimodal language models for materials science rely on effective mechanisms for incorporating three-dimensional structural representations. MatterChat addresses this challenge using a large bridge module between a pretrained structural encoder and the LLM, together with a separate contrastive alignment stage. We instead ask whether this bridge can adaptively select and combine representations from multiple structural encoders according to the target property. We find that supervised and generative material encoders provide complementary information for property prediction. We found that, for Zatom-1, this information is not concentrated in the final layer. Instead, the most informative representations occur at different depths depending on the target property. This motivates Task-Conditioned Attention Fusion (TCAF), a lightweight attention-based bridge that uses task-conditioned latent queries to combine CHGNet features with intermediate Zatom-1 representations. The bridge is trained directly on downstream prediction tasks, allowing representation selection and material-language alignment to be learned jointly without separate contrastive pretraining. Across classification and regression tasks, TCAF outperforms MatterChat, with up to 47.8% lower RMSE, while using substantially fewer trainable parameters and halving the overall training time. Further experiments incorporating UMA-S demonstrate TCAF's flexibility across encoder combinations, with the three-encoder configuration achieving the highest macro-average classification accuracy of 89.61%.
Comments: Accepted at the AI4Mat Workshop at NeurIPS 2026
Subjects: Machine Learning (cs.LG); Artificial Intelligence (cs.AI)
Cite as: arXiv:2611.24641 [cs.LG]
  (or arXiv:2611.24641v1 [cs.LG] for this version)
  https://doi.org/10.48550/arXiv.2611.24641

arXiv-issued DOI via DataCite (pending registration)

Submission history

From: Huu Binh Ta [view email]
[v1] Wed, 18 Nov 2026 15:48:25 UTC (262 KB)

来源:arXiv cs.AI · arxiv.org