arXiv 论文发现 SWE-Bench Pro 更高的 Claude Fable 5 并未显著提升真实开发产出
Higher-Scoring Coding Models Did Not Get More Work Done
AI 导读
一篇 arXiv 论文利用 2026 年 6 月 12 日美国出口管制指令导致 Claude Fable 5 停用 19 天的自然实验,追踪 297 名用户切换到 SWE-Bench Pro 低 11 分的 Opus 4.8 前后的 GitHub pull requests,并以 8,908 名一直使用 Opus 4.8 的开发者作对照。
推荐理由
研究用出口管制造成的模型切换作自然实验,发现基准分数更高的编码模型并未带来更高的 PR 合并率,为基准与真实生产力的差距提供了实证。
正文 · 原文
Abstract:New coding models advertise themselves on score improvements of coding benchmarks. Yet no study has tested if a model that scores higher on these benchmarks gets more real work done. On June 12, 2026, a U.S. export control directive banned access to Claude Fable 5 for 19 days, which forced users to switch to other models. During this time, most former Fable 5 users switched to Opus 4.8 (11 points lower on SWE-Bench Pro compared to Fable 5). We treat this suspension as a natural experiment, where the people and projects stayed the same and only the model changed, so we could isolate the effect of Fable 5 on user productivity. We tracked GitHub pull requests of 297 Fable 5 users before, during, and after the suspension. We used 8,908 developers who stayed on Opus 4.8 all the time as the control group. We also repeated the comparison across five model upgrades before and after the suspension in 2026 (1,208 developers, 15,442 pull requests). Fitting a difference-in-differences model with developer and date fixed effects, we found that having access to Fable 5 increased the 7-day merge rate of pull requests by only 1.4 percentage points [-1.4; +4.3], which was not statistically significant. Similarly, no significant increases of merge rates were found with the upgrades (weighted mean: -0.9 percentage points [-1.9; +0.2]); all five estimates were negative (95% intervals included 0, did not differ from each other). We also did not observe a significant increase in the number of pull requests merged per week (+2.4% [-4.9%; +10.2%]).
| Comments: | 20 pages, 4 figures |
| Subjects: | Software Engineering (cs.SE); Artificial Intelligence (cs.AI); Machine Learning (cs.LG) |
| Cite as: | arXiv:2611.23216 [cs.SE] |
| (or arXiv:2611.23216v1 [cs.SE] for this version) | |
| https://doi.org/10.48550/arXiv.2611.23216 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Yining Hua [view email]
[v1]
Tue, 17 Nov 2026 18:56:42 UTC (170 KB)
来源:arXiv cs.AI · arxiv.org