arXiv cs.AI· Pranava Sai Maganti, Meghasyam Peddireddy, Muzakker Hossain, Kaining Gao, Farhan Ahmed, Ali Jannesari·· 1 小时前AI 评分37
MLIRBench：面向编译器与硬件感知推理的大语言模型基准与数据集
MLIRBench - A Benchmark and Dataset for Compiler and Hardware-Aware Reasoning
AI 导读
MLIRBench 是一个评估大语言模型对 MLIR 程序推理能力的基准，通过多智能体管线生成程序变异并经执行验证，构建了 1,768 对已验证样本（888 对 linalg、880 对 torch），标注为 EQUIV、DIVERGE 或 ERROR。
正文 · 原文
Abstract:In recent years, large language models (LLMs) have become increasingly popular in software engineering and have performed well on different programming tasks. However, it is still difficult to know whether these models truly understand program behavior or whether they mainly rely on patterns learned from their training data. This distinction is particularly important for intermediate representations (IRs), where operations, dataflow, and execution semantics are explicitly represented. In this paper, we introduce MLIRBench, a benchmark for evaluating how LLMs reason about MLIR programs. MLIRBench uses a multi-agent pipeline to generate program mutations, validate their behavior through execution, and construct reasoning tasks from torch and linalg program pairs. The dataset contains 1,768 validated pairs, including 888 linalg pairs and 880 torch pairs, labeled as EQUIV, DIVERGE, or ERROR. We evaluated nine models, including five open-weight and four proprietary models. Gemini 3.1 Pro achieved the highest hard-label accuracy of 92.65% across all 1,768 pairs, counting unsuccessful predictions as incorrect. Across its 1,768 scored predictions, it achieved an aggregate score of 74.18 and a judge-assigned result-behavior score of 5.04/10. These results show that strong semantic classification does not necessarily imply accurate prediction of detailed execution behavior, highlighting a gap in current LLM reasoning over compiler IRs. By exposing this gap through execution-validated program pairs, MLIRBench provides a reproducible foundation for studying semantic reasoning over compiler IRs and related tasks such as program repair and semantic classification.
| Comments: | Accepted at the 2027 IEEE/ACM International Symposium on Code Generation and Optimization (CGO 2027) |
| Subjects: | Programming Languages (cs.PL); Artificial Intelligence (cs.AI); Machine Learning (cs.LG) |
| Cite as: | arXiv:2611.14881 [cs.PL] |
| (or arXiv:2611.14881v1 [cs.PL] for this version) | |
| https://doi.org/10.48550/arXiv.2611.14881 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Pranava Sai Maganti [view email]
[v1]
Wed, 11 Nov 2026 01:12:50 UTC (2,061 KB)
来源：arXiv cs.AI · arxiv.org