跳到正文
arXiv cs.AI· Prakhar Mishra, Bhavya Yajush Awasthi, Tanuja, Nishant·· 1 小时前AI 评分34

自奖励强化学习改进小语言模型医学解释的结构化研究

Self-Rewarded Reinforcement Learning for Better-Structured Medical Explanations in Small Language Models

AI 导读

一项研究在 Mistral-7B 骨干上用 PPO 加 LoRA 对 2000 条 Meerkat 衍生医学提示做轻量强化学习,奖励更长、词汇多样、标点良好的回答,无需人工偏好标注。在单张 NVIDIA A100 上训练约 170 个 PPO 步后奖励稳定提升,模型生成更长、多点组织化的解释,且保留底层医学内容。

正文 · 原文

View PDF HTML (experimental)

Abstract:Small language models (SLMs) can run on commodity hardware and support privacy-preserving, on-premises clinical deployment, but their medical explanations are often short and poorly structured compared with large language models. Recent supervised approaches such as Meerkat transfer chain-of-thought reasoning into 7B-8B backbones, yet their training objective does not directly target the style or organization of the generated rationales. Building on the instruction-tuned Mistral-7B backbone, we investigate whether a lightweight reinforcement learning (RL) step can improve explanation structure without human preference labels. We apply Proximal Policy Optimization (PPO) with LoRA adaptation to 2000 Meerkat-derived medical prompts, guided by a simple self-supervised reward that favors longer, lexically diverse, and well-punctuated responses while penalizing very short ones. Training for about 170 PPO steps on a single NVIDIA A100 GPU yields a stable increase in the average reward. Quantitative and qualitative comparisons show that the RL-tuned model produces longer, more organized, multi-point explanations while preserving the underlying medical content. We discuss the promise and the limitations of self-rewarded RL for shaping the style of medical explanations.
Comments: 14 Pages, 10 Figures
Subjects: Artificial Intelligence (cs.AI); Computation and Language (cs.CL)
Cite as: arXiv:2611.24650 [cs.AI]
  (or arXiv:2611.24650v2 [cs.AI] for this version)
  https://doi.org/10.48550/arXiv.2611.24650

arXiv-issued DOI via DataCite

Submission history

From: Bhavya Yajush Awasthi [view email]
[v1] Wed, 18 Nov 2026 15:52:24 UTC (48,594 KB)
[v2] Thu, 19 Nov 2026 05:26:05 UTC (48,590 KB)

来源:arXiv cs.AI · arxiv.org