AstroVLBench：视觉语言模型在观测天文推理任务上的系统性评测
A systematic evaluation of vision-language models for observational astronomical reasoning tasks
AI 导读
研究者发布 AstroVLBench 基准，包含 4,100 多个专家核验实例，覆盖光学成像、射电干涉测量、多波段测光、时域光变曲线和光学光谱五类任务，并系统评测了六个前沿 VLM。
正文 · 原文
Abstract:Vision-language models (VLMs) are increasingly proposed as general-purpose tools for scientific data interpretation, yet their reliability on real astronomical observations across diverse modalities remains untested. We present AstroVLBench, a benchmark of over 4,100 expert-verified instances across five tasks spanning optical imaging, radio interferometry, multi-wavelength photometry, time-domain light curves, and optical spectroscopy. We systematically evaluate six state-of-the-art frontier models and find that performance is strongly modality-dependent and uncorrelated with general benchmark rankings. Mechanistic cross-model ablations reveal three concrete bottlenecks. First, prompt grounding exhibits marked model and task dependence: physical guidance yields more balanced classifications on optical imaging, whereas phenomenological prompts better navigate undersampled time-domain data. Second, presenting one-dimensional measurements as numerical tables rather than rendered plots consistently improves light-curve classification across all models by up to 23 percentage points, demonstrating that plot rendering can obscure physically relevant signal. Third, an expert audit of optical spectroscopy reveals a notable "right-answer-wrong-reason" phenomenon: across evaluated models, an average of 26% of correct predictions are accompanied by physically inconsistent or invalid reasoning chains, showing that accuracy alone is insufficient for trustworthy scientific deployment. AstroVLBench provides the first systematic, multi-modal baselines for VLMs in observational astronomy and identifies the representation, grounding, and reasoning bottlenecks that must be addressed before VLMs can be trusted as autonomous scientific interpreters.
| Comments: | 42 pages, 15 figures, 11 tables. Accepted at NeurIPS 2026, Track on Evaluations and Datasets |
| Subjects: | Artificial Intelligence (cs.AI); Astrophysics of Galaxies (astro-ph.GA); Instrumentation and Methods for Astrophysics (astro-ph.IM) |
| Cite as: | arXiv:2604.24589 [cs.AI] |
| (or arXiv:2604.24589v2 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2604.24589 arXiv-issued DOI via DataCite |
Submission history
From: Wenke Ren [view email]
[v1]
Mon, 27 Apr 2026 15:11:31 UTC (4,415 KB)
[v2]
Mon, 26 Oct 2026 07:52:20 UTC (5,114 KB)
来源：arXiv cs.AI · arxiv.org