LongReasonArena：评估大语言模型长推理能力的基准
LongReasonArena: A Long Reasoning Benchmark for Large Language Models
AI 导读
LongReasonArena 是一个专门评估大语言模型长推理能力的基准，要求模型执行检索、回溯等多步算法来解题，推理长度可通过控制输入任意扩展，最难题目可达 100 万 token 推理。开源与闭源模型均面临显著挑战，GPT-5 在 Level 3 上准确率仅 27.8%。分析显示准确率随预期推理步数的对数呈线性下降，代码与数据已公开。
正文 · 原文
Abstract:Existing long-context benchmarks for Large Language Models (LLMs) focus on evaluating comprehension of long inputs, while overlooking the evaluation of long reasoning abilities. To address this gap, we introduce LongReasonArena, a benchmark specifically designed to assess the long reasoning capabilities of LLMs. Our tasks require models to solve problems by executing multi-step algorithms that reflect key aspects of long reasoning, such as retrieval and backtracking. By controlling the inputs, the required reasoning length can be arbitrarily scaled, reaching up to 1 million tokens of reasoning for the most challenging tasks. Extensive evaluation results demonstrate that LongReasonArena presents a significant challenge for both open-source and proprietary LLMs. For instance, even GPT-5 achieves only 27.8% accuracy on Level 3. Further analysis also reveals that the accuracy exhibits a linear decline with respect to the logarithm of the expected number of reasoning steps. Our code and data are available at this https URL.
| Subjects: | Computation and Language (cs.CL); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2508.19363 [cs.CL] |
| (or arXiv:2508.19363v2 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2508.19363 arXiv-issued DOI via DataCite |
|
| Journal reference: | Findings of the Association for Computational Linguistics: EMNLP 2026, pp. 2470-2485, 2026 |
| Related DOI: | https://doi.org/10.18653/2026.findings-emnlp.120
DOI(s) linking to related resources |
Submission history
From: Jiayu Ding [view email]
[v1]
Tue, 26 Aug 2025 18:41:53 UTC (364 KB)
[v2]
Wed, 11 Nov 2026 09:42:33 UTC (376 KB)
来源：arXiv cs.AI · arxiv.org