SMIC：跨 Transformer 层共享多尺度输入-参考条件化方法用于自然语言推理
SMIC: Shared Multiscale Input-Reference Conditioning across Transformer Layers for Natural Language Inference
AI 导读
arXiv 论文提出共享多尺度输入-参考条件化方法 SMIC，构建单一依赖输入的多尺度参考并在各 Transformer 层间复用，每层对共享参考施加自己的多尺度变换后在注意力更新前注入条件特征。
正文 · 原文
Abstract:Natural Language Inference models can be sensitive to localized lexical changes. We introduce Shared Multiscale Input-Reference Conditioning (SMIC), which constructs a single input-dependent multiscale reference and reuses it across Transformer layers. Each layer applies its own multiscale transformation to the shared reference and injects the resulting conditioning feature before the attention update. We instantiate SMIC in a token--context Transformer (TCT) and conduct from-scratch comparisons against decoder-based and Multi-Layer Perceptron (MLP) baselines across all three ANLI rounds. TCT-SMIC attains the highest observed ten-seed mean accuracy on all three ANLI rounds. The ablations further support the contribution of SMIC. We evaluate a real-valued Llama-style plug-in and extend deletion/insertion diagnostics to pretrained ModernBERT encoder and Llama decoder models with and without SMIC. Across the evaluated perturbation protocols, TCT-SMIC exhibits comparatively low pooled-attention redistribution and smaller signed accuracy changes. We further establish a finite-sample bound on average subset-mass redistribution in terms of the mean Jensen--Shannon divergence between the restricted and renormalized attention distributions.
| Subjects: | Computation and Language (cs.CL); Artificial Intelligence (cs.AI); Machine Learning (cs.LG) |
| Cite as: | arXiv:2611.16270 [cs.CL] |
| (or arXiv:2611.16270v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2611.16270 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Xin Zhang [view email]
[v1]
Thu, 12 Nov 2026 04:15:47 UTC (861 KB)
来源：arXiv cs.AI · arxiv.org