任务越难，LLM 表征越稀疏：大语言模型 OOD 机制分析

View PDF HTML (experimental)

Abstract:In this work, we investigate how the internal representations of Large Language Models (LLMs) change with inputs of increasing difficulty. Although sparsity changes may not be obvious for individual samples, dataset-level results reveal a consistent statistical trend: as task difficulty increases (e.g., harder questions, longer contexts, or more answer choices), the last hidden states of LLMs become systematically sparser. In short, \textbf{the harder the task, the sparser the representation}. This sparsity--difficulty relation is observable across diverse models and domains, suggesting that harder inputs drive more concentrated activation patterns in the last hidden state. Through a series of controlled analyses and learning-dynamics experiments, we show that representational density is a learned property of data familiarity: models develop rich, distributed representations for mastered patterns, while unfamiliar inputs default to sparser activations. Our finding is also actionable. We illustrate this with Sparsity-Guided Curriculum In-Context Learning (SG-ICL), which uses sparsity to select few-shot demonstrations matched to the query's difficulty, outperforming standard CoT and Auto-CoT baselines on MATH-500. Our study provides new insights into how LLM representations reflect task difficulty and how this signal can be leveraged for inference. The source code is available at the URL: this https URL.

Subjects:	Computation and Language (cs.CL); Artificial Intelligence (cs.AI)
Cite as:	arXiv:2603.03415 [cs.CL]
	(or arXiv:2603.03415v4 [cs.CL] for this version)
	https://doi.org/10.48550/arXiv.2603.03415 arXiv-issued DOI via DataCite

Submission history

From: Yutong Yin [view email]
[v1] Tue, 3 Mar 2026 18:48:15 UTC (427 KB)
[v2] Thu, 19 Mar 2026 17:36:27 UTC (429 KB)
[v3] Sat, 26 Sep 2026 05:28:48 UTC (1,266 KB)
[v4] Wed, 4 Nov 2026 05:06:30 UTC (1,258 KB)