Conceptizer：下一个词预测并非一切
Conceptizer: Next-Token Prediction Is Not All You Need
AI 导读
Conceptizer 是一个在下一个词预测（NTP）之外为"语义"提供奖励的训练框架，使用约 9K 个概念（由上下文同义词构建）作为含义单元，不增加模型参数。在 3 个 1B-8B 参数规模的模型家族上，它在概念目标位置取得最低的 held-out NTP 困惑度，优于全部 3 个基线，并在知识与情感任务上提升下游表现。
正文 · 原文
Abstract:Next-token prediction (NTP) rewards only the token that occurs next, penalizing every alternative. But natural language often offers many ways to express the same meaning. We introduce Conceptizer, a training framework that extends next-token prediction with a reward for meaning. Conceptizer uses a fixed, model-agnostic vocabulary of ~9K concepts, built from contextual synonyms, as units of meaning alongside tokens as units of text. It maps each next-token distribution to a concept distribution and shares the reward for the target meaning across its possible wordings. This provides a concept-level training signal without adding model parameters. Applied to 3 model families at 1B-8B parameters, Conceptizer achieves the lowest held-out NTP perplexity at concept-target positions and assigns the highest probability to contextual synonyms, against all 3 baselines. It also improves downstream performance on knowledge and sentiment tasks while remaining comparable on commonsense. These results show that language models can be directly optimized to predict meaning, not just the next token.
| Comments: | 35 pages, 10 figures, 16 tables |
| Subjects: | Computation and Language (cs.CL); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2610.33027 [cs.CL] |
| (or arXiv:2610.33027v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2610.33027 arXiv-issued DOI via DataCite |
Submission history
From: Haoran Zheng [view email]
[v1]
Sun, 11 Oct 2026 08:59:07 UTC (408 KB)
来源：arXiv cs.AI · arxiv.org