跳到正文
arXiv cs.AI· Jihoon Kwon, Jiwon Choi, Jy-yong Sohn·· 2 小时前AI 评分42

分布对齐作为 In-Context Learning 任务向量设计准则:LTV 方法提出

Distributional Alignment as a Criterion for Designing Task Vectors in In-Context Learning

AI 导读

一篇 NeurIPS 2026 论文提出 Linear Task Vector(LTV)方法,用 d_NTP 指标衡量任务向量推理与 ICL 推理的下一词概率差异,该指标与下游准确率呈强负相关。在 8 个分类基准和 5 个 LLM 上,LTV 比现有基线平均提升准确率 9.2%,并降低推理延迟;大模型提取的任务向量可使小模型性能提升 6.4%。

正文 · 原文

View PDF HTML (experimental)

Abstract:In-context learning (ICL) allows large language models (LLMs) to adapt to new tasks through demonstrations, yet it suffers from escalating inference costs as context length increases. While task vectors offer a promising alternative by compressing demonstrations into compact hidden-state representations, their quality has been evaluated only through downstream task accuracy. This indirect criterion provides limited insight into how to design more effective task vector extraction methods. In this paper, we posit that inference using task vectors should align their predictive distribution with that of ICL. To quantify this, we introduce $d_{\text{NTP}}$, a metric that measures the discrepancy in next-token probabilities between task vector-based and ICL-based inference. Our empirical analysis reveals that $d_{\text{NTP}}$ serves as a performance proxy, exhibiting a strong negative correlation with downstream accuracy. Motivated by this, we develop Linear Task Vector (LTV), a method designed to minimize $d_{\text{NTP}}$ via a closed-form linear mapping that estimates demonstration effects through regression. Across eight classification benchmarks and five LLMs, LTV consistently outperforms existing task vector baselines, improving average accuracy by 9.2\% while reducing inference latency. We further show that LTV outperforms the baselines on regression tasks. Moreover, we investigate the transferability of LTV across different model scales; an aspect that has remained nascent in task vector research. Specifically, we empirically show that task vectors from a larger model can enhance a smaller model's performance by 6.4\%, suggesting a new utility for extracted task representations.
Comments: Accepted at NeurIPS 2026. 29 pages
Subjects: Computation and Language (cs.CL); Artificial Intelligence (cs.AI)
Cite as: arXiv:2605.20730 [cs.CL]
  (or arXiv:2605.20730v2 [cs.CL] for this version)
  https://doi.org/10.48550/arXiv.2605.20730

arXiv-issued DOI via DataCite

Submission history

From: Jihoon Kwon [view email]
[v1] Wed, 20 May 2026 05:26:38 UTC (2,391 KB)
[v2] Tue, 17 Nov 2026 05:53:04 UTC (2,385 KB)

来源:arXiv cs.AI · arxiv.org