arXiv cs.AI· Giwon Hong, Emile van Krieken, Edoardo Ponti, Esmeralda S. Whitammer, Pasquale Minervini·· 2 小时前AI 评分24
Mixtures of In-Context Learners(MoICL):将演示子集作为专家混合的上下文学习方法
Mixtures of In-Context Learners
AI 导读
一篇 ACL 2025 论文提出 Mixtures of In-Context Learners(MoICL),把上下文学习的演示子集当作专家,基于训练集学习加权函数来合并其输出分布,避免演示消耗上下文窗口与内存。
正文 · 原文
Abstract:In-context learning (ICL) adapts LLMs by providing demonstrations without fine-tuning the model parameters; however, it does not differentiate between demonstrations and quadratically increases the complexity of Transformer LLMs, exhausting the memory. As a solution, we propose Mixtures of In-Context Learners (MoICL), a novel approach to treat subsets of demonstrations as experts and learn a weighting function to merge their output distributions based on a training set. In our experiments, we show performance improvements on 5 out of 7 classification datasets compared to a set of strong baselines (up to +13\% compared to ICL and LENS). Moreover, we enhance the Pareto frontier of ICL by reducing the inference time needed to achieve the same performance with fewer demonstrations. Finally, MoICL is more robust to out-of-domain (up to +11\%), imbalanced (up to +49\%), or noisy demonstrations (up to +38\%) or can filter these out from datasets. Overall, MoICL is a more expressive approach to learning from demonstrations without exhausting the context window or memory.
| Comments: | ACL 2025; code: this https URL |
| Subjects: | Computation and Language (cs.CL); Artificial Intelligence (cs.AI); Machine Learning (cs.LG) |
| Cite as: | arXiv:2411.02830 [cs.CL] |
| (or arXiv:2411.02830v2 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2411.02830 arXiv-issued DOI via DataCite |
|
| Related DOI: | https://doi.org/10.18653/v1/2025.acl-long.1277
DOI(s) linking to related resources |
Submission history
From: Esmeralda Whitammer [view email]
[v1]
Tue, 5 Nov 2024 06:02:41 UTC (213 KB)
[v2]
Wed, 18 Nov 2026 16:51:08 UTC (196 KB)
来源:arXiv cs.AI · arxiv.org