arXiv 论文评估语言模型迭代完成数据科学项目的能力
Iterated creation, grading, and revision of data science projects by language models
AI 导读
arXiv 论文（arXiv:2611.05548）评估智能体模型能否端到端完成数据科学项目，涵盖下载、清洗、分析数据集并撰写短论文。7 个模型来自 5 家开发方，组成 49 个 writer-grader 配对各跑 3 次，147 份初始提交中 107 份可运行渲染，平均分经三轮反馈从 59% 升至 76%。
正文 · 原文
Abstract:We evaluate the extent to which agentic AI models can complete an end-to-end data science project, and how this improves with feedback. A 'writer' model with specific tools creates a complete data science project including downloading, cleaning, and analyzing a dataset, and writing a short paper about it. The project is executed and a 'grader' model evaluates it. The writer can then revise its submission, and this repeats for a total of three rounds. We consider seven models from five developers: Anthropic, OpenAI, Google, DeepSeek, and this http URL. Each model is both a writer and a grader, so there are 49 writer-grader pairs and we run each pair three times. Of the 147 initial submissions, 107 run and render, and the average score increases from 59 per cent to 76 per cent after the final round of feedback. By writer the extent of the increase ranges from 3 percentage points for Anthropic's Opus 5.5 to 46 percentage points for Anthropic's Haiku 4.5, and by grader it ranges from 13 percentage points for Google's Gemini 3.8 Flash to 32 percentage points for Anthropic's Haiku 4.5. When a model grades its own writing the score tends to be about 9 percentage points higher than the average of what the other models give it. About half of the difference that we see from round to round is within the grader's own noise. Feedback generally results in the highest marks, but even just having the chance to revise, or having the additional budget, in terms of time or tool calls, without any feedback, results in considerable increases for some of the models but not for others. We find that how much such a loop helps depends on which model plays which role, but weaker models generally do not improve stronger models.
| Subjects: | Artificial Intelligence (cs.AI); Computation and Language (cs.CL) |
| Cite as: | arXiv:2611.05548 [cs.AI] |
| (or arXiv:2611.05548v1 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2611.05548 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Rohan Alexander [view email]
[v1]
Tue, 3 Nov 2026 19:39:03 UTC (233 KB)
来源：arXiv cs.AI · arxiv.org