跳到正文
arXiv cs.AI· Zijun Gao, Zhikun Xu, Xiao Ye, Ben Zhou·· 2 小时前AI 评分42

CORE:用概念导向强化弥合大语言模型数学推理中的定义与应用鸿沟

CORE: Concept-Oriented Reinforcement for Bridging the Definition-Application Gap in Mathematical Reasoning

AI 导读

研究者提出 CORE(Concept-Oriented REinforcement),一种把显式概念转化为可控监督信号的 RL 训练框架,以解决 LLM 会做题却难以真正应用概念的问题。

正文 · 原文

View PDF HTML (experimental)

Abstract:Large language models (LLMs) often solve challenging math exercises yet fail to apply the concept right when the problem requires genuine understanding. Popular Reinforcement Learning with Verifiable Rewards (RLVR) pipelines reinforce final answers but provide little fine-grained conceptual signal, so models improve at pattern reuse rather than conceptual applications. We introduce CORE (Concept-Oriented REinforcement), an RL training framework that turns explicit concepts into a controllable supervision signal. Starting from a high-quality, low-contamination textbook resource that links verifiable exercises to concise concept descriptions, we run a sanity probe showing LLMs can restate definitions but fail concept-linked quizzes, quantifying the conceptual reasoning gap. CORE then (i) synthesizes concept-aligned quizzes, (ii) injects brief concept snippets during rollouts to elicit concept-primed trajectories, and (iii) reinforces conceptual reasoning via trajectory replacement after group failures, a lightweight forward-KL constraint that aligns unguided with concept-primed policies, or standard GRPO directly on concept-aligned quizzes. Across several models, CORE delivers consistent gains over vanilla and SFT baselines on both in-domain concept-exercise suites and diverse out-of-domain math benchmarks. CORE unifies direct training on concept-aligned quizzes and concept-injected rollouts under outcome regularization. It provides fine-grained conceptual supervision that bridges problem-solving competence and genuine conceptual reasoning, while remaining algorithm- and verifier-agnostic.
Comments: Accepted at ICLR 2026. Code: this https URL
Subjects: Artificial Intelligence (cs.AI); Machine Learning (cs.LG)
Cite as: arXiv:2512.18857 [cs.AI]
  (or arXiv:2512.18857v4 [cs.AI] for this version)
  https://doi.org/10.48550/arXiv.2512.18857

arXiv-issued DOI via DataCite

Submission history

From: Zijun Gao [view email]
[v1] Sun, 21 Dec 2025 19:01:35 UTC (157 KB)
[v2] Mon, 2 Mar 2026 21:29:46 UTC (924 KB)
[v3] Thu, 7 May 2026 11:50:40 UTC (1,137 KB)
[v4] Tue, 17 Nov 2026 06:48:39 UTC (297 KB)

来源:arXiv cs.AI · arxiv.org