CORE:用概念导向强化弥合大语言模型数学推理中的定义与应用鸿沟
CORE: Concept-Oriented Reinforcement for Bridging the Definition-Application Gap in Mathematical Reasoning
AI 导读
研究者提出 CORE(Concept-Oriented REinforcement),一种把显式概念转化为可控监督信号的 RL 训练框架,以解决 LLM 会做题却难以真正应用概念的问题。
正文 · 原文
Abstract:Large language models (LLMs) often solve challenging math exercises yet fail to apply the concept right when the problem requires genuine understanding. Popular Reinforcement Learning with Verifiable Rewards (RLVR) pipelines reinforce final answers but provide little fine-grained conceptual signal, so models improve at pattern reuse rather than conceptual applications. We introduce CORE (Concept-Oriented REinforcement), an RL training framework that turns explicit concepts into a controllable supervision signal. Starting from a high-quality, low-contamination textbook resource that links verifiable exercises to concise concept descriptions, we run a sanity probe showing LLMs can restate definitions but fail concept-linked quizzes, quantifying the conceptual reasoning gap. CORE then (i) synthesizes concept-aligned quizzes, (ii) injects brief concept snippets during rollouts to elicit concept-primed trajectories, and (iii) reinforces conceptual reasoning via trajectory replacement after group failures, a lightweight forward-KL constraint that aligns unguided with concept-primed policies, or standard GRPO directly on concept-aligned quizzes. Across several models, CORE delivers consistent gains over vanilla and SFT baselines on both in-domain concept-exercise suites and diverse out-of-domain math benchmarks. CORE unifies direct training on concept-aligned quizzes and concept-injected rollouts under outcome regularization. It provides fine-grained conceptual supervision that bridges problem-solving competence and genuine conceptual reasoning, while remaining algorithm- and verifier-agnostic.
| Comments: | Accepted at ICLR 2026. Code: this https URL |
| Subjects: | Artificial Intelligence (cs.AI); Machine Learning (cs.LG) |
| Cite as: | arXiv:2512.18857 [cs.AI] |
| (or arXiv:2512.18857v4 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2512.18857 arXiv-issued DOI via DataCite |
Submission history
From: Zijun Gao [view email]
[v1]
Sun, 21 Dec 2025 19:01:35 UTC (157 KB)
[v2]
Mon, 2 Mar 2026 21:29:46 UTC (924 KB)
[v3]
Thu, 7 May 2026 11:50:40 UTC (1,137 KB)
[v4]
Tue, 17 Nov 2026 06:48:39 UTC (297 KB)
来源:arXiv cs.AI · arxiv.org