Astro2Method：评测 LLM 在天文学机器学习方法选择上的能力
Astro2Method: Benchmarking LLMs on Machine Learning Method Selection in Astronomy
AI 导读
研究者推出 Astro2Method 基准，用来自 346 篇天文学论文的 500 道专家审核二选题评测 LLM 选择机器学习方法的能力。五个 LLM 在 Easy 题上直接提示准确率为 87.50%-92.55%，Hard 题仅 58.87%-64.52%；ReAct、Self-Ask、CRITIC 等智能体方法和 RAG 均无一致提升。
正文 · 原文
Abstract:Machine learning (ML) is increasingly used in astronomy, but choosing a suitable method for a scientific problem often requires expertise in both astronomy and ML. Large language models (LLMs) could help astronomers make such choices, yet this ability remains underexplored in existing benchmarks. We introduce Astro2Method, a benchmark of 500 expert-reviewed binary questions from 346 astronomy papers (2024/01-2026/06), based on clear performance differences and complete (Easy) or partial (Hard) explanations from the authors. Across five LLMs, direct prompting achieves 87.50-92.55% accuracy on Easy questions but only 58.87-64.52% on Hard. Using historical papers, neither retrieval-augmented generation nor most agent methods with additional reasoning or retrieval (including ReAct, Self-Ask, and CRITIC) bring consistent improvement. Agents required to search the web improve accuracy mainly when they find the papers the questions are based on, with no consistent improvement otherwise. Providing the source paper's pipeline description with method names masked improves Hard accuracy, matching or exceeding web search. Together, these results suggest that relevant experimental information helps distinguish ML methods, but inferring it from related studies and domain knowledge remains challenging for LLMs in astronomy.
| Comments: | 15 pages, 4 figures, 4 tables |
| Subjects: | Instrumentation and Methods for Astrophysics (astro-ph.IM); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2610.30282 [astro-ph.IM] |
| (or arXiv:2610.30282v1 [astro-ph.IM] for this version) | |
| https://doi.org/10.48550/arXiv.2610.30282 arXiv-issued DOI via DataCite |
Submission history
From: Xingzhong Fan [view email]
[v1]
Wed, 21 Oct 2026 18:01:34 UTC (2,557 KB)
来源：arXiv cs.AI · arxiv.org