Pick the Mixture, Not the Samples：面向 LLM 的组合式数据选择方法
Pick the Mixture, Not the Samples: Combinatorial Data Selection for LLMs
AI 导读
一项 arXiv 研究提出组合式数据选择（CDS）框架，将打分归因与子集选择分离，并刻画训练样本间的非可加交互。其两阶段算法 Mixture-then-Select 先决定各领域取多少样本，再按归因分数在领域内排序，理论上与最优子集的差距与选择规模呈线性且与数据池规模无关。
正文 · 原文
Abstract:Data selection for LLM fine-tuning faces two facts about real training: data pools draw from many distinct domains, and evaluation spans diverse tasks. As a result, the same sample helps different tasks differently, and samples interact during training so that a subset's value is not the sum of its parts. Current data-selection methods leave these realities largely unaddressed. We formalize them as Combinatorial Data Selection (CDS), a framework that separates attribution (how samples are scored) from selection (which subset to pick) and models the non-additive interactions among samples. On a real instruction-tuning data pool, we find substantial cross-domain complementarity (joint utility exceeds the sum of singletons), alongside redundancy within domains. Inspired by this, we propose Mixture-then-Select, a two-stage algorithm that first decides how many samples to take from each domain, then ranks samples within each domain by attribution score. Theoretically, the gap to the optimal subset is linear in the selection size and independent of data pool size. Empirically, Mixture-then-Select outperforms strong baselines by 20 to 54%, is robust across attribution methods, and works across different models and data pool sizes. The framework also extends to the data purchasing problem under a budget constraint when sample prices vary. Our findings suggest that allocating selection across domains offers a promising path toward more efficient and effective data selection for modern LLMs.
| Subjects: | Machine Learning (cs.LG); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2610.14028 [cs.LG] |
| (or arXiv:2610.14028v2 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2610.14028 arXiv-issued DOI via DataCite |
Submission history
From: Luyang Zhang [view email]
[v1]
Wed, 30 Sep 2026 18:01:37 UTC (214 KB)
[v2]
Sat, 24 Oct 2026 09:10:44 UTC (214 KB)
来源：arXiv cs.AI · arxiv.org