arXiv cs.AI· Yucheng Wu, Jundong Xu, Mingzhen Ju, Yue Yu, Chenpeng Wang, Haoxuan Li, Liangming Pan·· 4 小时前AI 评分38
HOLogicBench：评测 LLM 高阶逻辑推理能力的基准
HOLogicBench: Evaluating Higher-Order Logical Reasoning in LLMs
AI 导读
研究者推出 HOLogicBench，一个包含 1,379 个实例的基准，系统评测五种高阶推理操作，并配有 Isabelle/HOL 形式化与验证过的答案和参考推理路径。在 16 个模型配置上平均准确率为 53.86%，最佳为 65.99%。结果显示 LLM 与有效推导的对齐有限，且易受最小反事实扰动影响。
正文 · 原文
Abstract:Existing LLM logical-reasoning benchmarks largely focus on object-level inference under fixed predicates and rules, leaving higher-order reasoning over rules and policies underexplored. We introduce HOLogicBench, a 1,379-instance benchmark that systematically evaluates five representative higher-order reasoning operations, with Isabelle/HOL formalizations and proof-verified answers. Beyond answer accuracy, HOLogicBench provides solver-verified reference reasoning paths for each instance, enabling fine-grained evaluation of model-generated reasoning against valid derivations. Across 16 model configurations, average accuracy is 53.86\%, with the best configuration reaching 65.99\%. Further analyses reveal that LLMs exhibit limited alignment with valid derivations, remain vulnerable to minimal counterfactual perturbations, and display distinct failure patterns across higher-order reasoning types.
| Subjects: | Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2606.23238 [cs.AI] |
| (or arXiv:2606.23238v3 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2606.23238 arXiv-issued DOI via DataCite |
Submission history
From: Yucheng Wu [view email]
[v1]
Mon, 22 Jun 2026 12:23:54 UTC (1,154 KB)
[v2]
Tue, 23 Jun 2026 08:36:37 UTC (1,154 KB)
[v3]
Mon, 26 Oct 2026 15:08:45 UTC (899 KB)
来源：arXiv cs.AI · arxiv.org