REFORGE:面向 LLM 反编译二进制函数命名逆向工程能力的基准测试方法
REFORGE: A Method for Benchmarking LLMs' Reverse Engineering Capabilities in Decompiled Binary Function Naming
AI 导读
一篇发表于 AC 2026 并获最佳论文奖的研究提出 Reforge,一个通过编译、DWARF 与语法抽取、对齐和反编译构建函数级真值、并带溯源追踪的基准测试流水线,将二进制-源码对齐的不确定性量化为八道闸门的三层置信漏斗。
正文 · 原文
Abstract:Large language models (LLMs) are increasingly applied to reverse-engineering tasks, and recent threat-intelligence reporting shows them operating inside live offensive-security workflows. Claims about their capability, however, outpace our ability to measure it. Existing benchmarks for LLM-assisted binary analysis treat the construction of function-level ground truth as a solved pre-processing step and report accuracy without disclosing how many functions were reliably evaluable. We argue that the principal obstacle to fair evaluation is not model capability but the reliability of binary-to-source alignment under compiler optimization. This paper presents Reforge, a provenance-tracked pipeline that constructs function-level ground truth from C source through compilation, DWARF and syntactic extraction, alignment, and decompilation, and that operationalizes alignment uncertainty as an eight-gate confidence funnel with three-tier stratification. On a controlled micro-benchmark, high-confidence yield falls from 87.2% to 65.9% across optimization levels, and unpaired comparisons overstate optimization-induced performance decay through survivorship bias. A proof-of-concept evaluation of seven contemporary LLMs on function naming demonstrates the validity of the concept and generally motivates an uncertainty-aware benchmarking practice.
| Comments: | 10 pages, 5 figures. Published in the proceedings of the 23rd International Conference on Applied Computing 2026 (AC 2026), Lisbon, 24-26 October 2026. Best Paper Award |
| Subjects: | Software Engineering (cs.SE); Artificial Intelligence (cs.AI); Cryptography and Security (cs.CR); Programming Languages (cs.PL) |
| Cite as: | arXiv:2607.07738 [cs.SE] |
| (or arXiv:2607.07738v3 [cs.SE] for this version) | |
| https://doi.org/10.48550/arXiv.2607.07738 arXiv-issued DOI via DataCite |
|
| Journal reference: | Proceedings of the 23rd International Conference on Applied Computing 2026 (AC 2026), Lisbon, Portugal, 24-26 October 2026, IADIS Press, pp. 202-211, ISBN 978-989-8704-82-5 |
Submission history
From: Andreas U. Schmidt [view email]
[v1]
Tue, 7 Jul 2026 23:17:03 UTC (351 KB)
[v2]
Fri, 24 Jul 2026 13:29:41 UTC (428 KB)
[v3]
Fri, 13 Nov 2026 14:57:18 UTC (428 KB)
来源:arXiv cs.AI · arxiv.org