arXiv cs.AI· Sutanshu Saksena Raj, Purusharth Malik, Pasunuti Prasanjith, Daksh Maan, Parijat Chatterjee, Avinash Benki·· 3 小时前AI 评分46
奖励范围之外：多语言后训练中的能力漂移研究
Outside the Purview of the Reward: Capability Drift in Multilingual Post-Training
AI 导读
一项 arXiv 研究考察多语言 mixture-of-experts 模型的两个 GRPO 分支，发现后训练在提升被奖励行为的同时会削弱其他能力：发布分支在 MATH 上提升 36.07 分，却在 XQuAD-Indic 上损失 19.32 F1。
正文 · 原文
Abstract:Post-training can improve rewarded behavior while weakening capabilities users still need. We study two GRPO branches from a shared supervised checkpoint in a multilingual mixture-of-experts model. The released branch gains 36.07 points on MATH but loses 19.32 F1 points on XQuAD-Indic, while XQuAD-English improves under the same scoring rule. An independently trained sibling adds Indic instruction following and language-aware shaping. It recovers the regressing Indic task scores and preserves MATH, but does not restore script compliance and trades off competition mathematics and code. In a later round, judge training is followed by a sharp extraction decline from the Indic branch; reward-free distillation from its earlier checkpoint restores most of the lost F1. A MATH score decomposition associates most of the measured gain with increased boxed-answer emission; accuracy among boxed answers changes little. These results show why benchmark scores should be read alongside the response forms they reward.
| Comments: | Accepted at the Foundations of LLM Post-Training in Changing Environments, NeurIPS 2026 |
| Subjects: | Computation and Language (cs.CL); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2611.15717 [cs.CL] |
| (or arXiv:2611.15717v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2611.15717 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Sutanshu Saksena Raj [view email]
[v1]
Wed, 11 Nov 2026 16:55:22 UTC (38 KB)
来源：arXiv cs.AI · arxiv.org