D ecompile B ench: A Comprehensive Benchmark for Evaluating Decompilers in Real-World Scenarios (2025)

Abstract: Decompilers are fundamental tools for critical security tasks, from vulnerability discovery to malware analysis, yet their evaluation remains fragmented. Existing approaches primarily focus on syntactic correctness through synthetic micro-benchmarks or subjective human ratings, failing to address real-world requirements for semantic fidelity and analyst usability. We present **DecompileBench**, the first comprehensive framework that enables effective evaluation of decompilers in reverse engineering workflows through three key components: real-world function extraction (comprising 23,400 functions from 130 real-world programs), runtime-aware validation, and automated human-centric assessment using LLM-as-Judge to quantify the effectiveness of decompilers in reverse engineering workflows. Through a systematic comparison between six industrial-strength decompilers and six recent LLM-powered approaches, we demonstrate that LLM-based methods surpass commercial tools in code understandability despite 52.2% lower functionality correctness. These findings highlight the potential of LLM-based approaches to transform human-centric reverse engineering. We open source **DecompileBench** to provide a framework to advance research on decompilers and assist security experts in making informed tool selections based on their specific requirements.

## introduction modern software security fundamentally depends on understanding binary code. from identifying critical vulnerabilities in the network infrastructure to analyzing sophisticated malware, security analysts rely on decompilers to bridge the semantic gap between low-level machine instructions and human-comprehensible program logic. as a crucial line of defense against evolving cyber threats, these tools must not only faithfully recover program semantics, but also generate output that facilitates rapid analysis under real-world time constraints. two transformative forces have reshaped the decompilation landscape. first, modern software has grown increasingly complex and aggressive compiler optimizations systematically erase highlevel semantics, making decompilation increasingly harder. second, large language models have emerged as a disruptive force in decompilation. reverse engineers now experiment with generalpurpose models like gpt-4 for reverse engineering tasks (kwiatkowski; reveng.ai; ninja), while specialized models such as llm4decompile @xcite and mlm (ascendgrace) focus on neural decompilation. these llm-powered approaches show promise in generating more readable code by learning high-level programming patterns. however, these advances bring new challenges. while llms-powered approaches produce visually coherent code, reverse engineers question whether these decompilations preserve the semantic behaviors crucial for security analysis. this uncertainty reveals a critical gap in how we evaluate decompilers for real-world security tasks. crisis of functionality validation in real-world scenarios. current decompiler evaluation approaches suffer from fundamental limitations. most rely on artificially constructed programs @xcite (e.g., csmith-generated code @xcite ) that lack the complexity of production software. when validating these decompilers, symbolic execution methods fail due to path explosion, while unit tests @xcite only check input-output equivalence, thus ignoring critical behaviors like global state changes, heap manipulation, and exception handling that security analysts must trace when hunting vulnerabilities. deficits of automated understandability assessment. current methods for evaluating decom- ## methodology to evaluate decompilation in real-world settings, we use the oss-fuzz to construct dataset. we further develop an evaluation framework that comprehensively evaluates the decompiler's performance from three aspects cared for by the end users. ## evaluation our evaluation encompasses six traditional decompilers, representing both commercial and open-source solutions for reverse engineering, the description and decompilation technical details are presented in section a. for the emerging paradigm of llm-based decompilation, we evaluate both domain-specific and general-purpose models: llm4decompile is tested using its official prompt template, and mlm is assessed through its public api service. the general models (gpt-4o-mini, gpt-4o, qwen2.5-coder-32b-instruct, claude-3.5-sonnet, and deepseek-v3) category subclass explanation example readability typecast correctness redundant/incorrect type casts obscure intent exit((long long)"invalid size") literal representation non-idiomatic literals hinder understanding numeric 2685 instead of string literal "\n" control flow clarity complex pointer dereferencing for (@xmath0; a1 != (*((_qword *)(i+64))); @xmath1; to evaluate whether llms can effectively capture human judgments on decompiled code quality, we employed cohen's kappa (κ) to measure the agreement between llm and human ratings. we describe the detail in section e. ## conclusion we present decompilebench, a comprehensive decompiler evaluation framework that addresses real-world assessment challenges through three key innovations: production-grade datasets from oss-fuzz, runtime behavior validation, and llmpowered code quality analysis. our evaluation of 12 decompilers highlights critical trade-offs: traditional tools such as hex-rays achieve a 58.3% average recompilation success rate, whereas llmbased approaches like mlm excel in code quality, offering superior control flow clarity and more meaningful identifier naming. this work establishes new standards for decompiler evaluation in security-critical scenarios and highlights the need for hybrid approaches combining neural-based decompilers with rule-based approaches. our framework enables informed tool selection for reverse engineers while guiding future research toward balancing reliability with humancentric readability for reverse engineering.