C hat GPT as a J ava Decompiler (2023)

Abstract: We propose a novel approach using instruction-tuned large language models (LLMs), such as ChatGPT, to automatically decompile entire Java classes. Our method relies only on a textual representation of the Java bytecode and corresponding unit tests generated from the bytecode. While no additional domain knowledge or fine-tuning is performed, we provide a single training example of this decompilation process in the model’s prompt. To overcome both compilation errors and test failures, we use an iterative prompting approach. We find that ChatGPT-4 is able to generate more human-readable output than existing software-based decompilers while achieving slightly lower pass rates on unit tests. Source code and datasets are available at https://github.com/BradMcDanel/gpt-java-decompiler .

## introduction decompilation is the process of converting a binary machine language into a corresponding high-level language source code. this technique has numerous applications in fields such as rewriting legacy code, malware analysis, and software vulnerability repair. unfortunately, existing software-based decompilers are time-consuming to develop and can generate source code that is hard for humans to understand @xcite . neural machine translation (nmt) methods have been recently proposed as an alternative to conventional software solutions to translate between programming languages (e.g., c# to java) @xcite . nmt approaches have also been applied to program decompilation, where the source language is a compiled assembly/bytecode representation generated by a compiler and the target language is the original programming language. the majority of nmt approaches focus on translating a single function with no side effects. we speculate this constraint is due in large part to the limited source and target lengths for transformer-based translation models. for instance, codet5 @xcite and codet5+ @xcite typically use source and target sequence lengths 1024 or fewer tokens for a variety of code tasks such as writing and translation. this problem is exacerbated when the source sequence is an assembly/bytecode representation that can require 2-8x more tokens than their programming language counterpart. in this work, we focus on the task of translating the java bytecode of an entire class file to java source code. this problem is significantly more challenging than translating a single function for multiple reasons. first, a class can contain tens of methods that, when tokenized, often exceed smaller token limits. second, fields/methods defined earlier in a class are used in the implementation of other methods, making correct decompilation challenging due to long-term dependencies. similarly, imported packages, generally defined at the top of the class, are also used throughout the file. finally, there are many language-specific features that generate more rarely occurring patterns of bytecode (e.g., exceptions, static/final variables, multiple constructors). with the recent addition of longer context windows for commercial instruction-tuned models foot_0 , we believe it is possible to take on more challenging programming translation tasks, such as entire program decompilation. to achieve accurate decompilation, we propose an iterative generation loop to guide the model around two types of error conditions: compilation errors and unit test errors. for compilation errors, we use chatgpt to determine if the source of the error is due (1) early stopping (i.e., emitting a stop token in the middle of a large class) or (2) invalid java code. in the case of early stopping, we simply instruct the model to continue generation, while invalid code leads to a retry. once compiled, we apply unit tests to the code. if any unit test fails, we start over with a new generation pass up to a maximum attempt limit. our iterative approach achieves a slightly lower test pass rate but leads to higher quality code evaluated by several similarity metrics compared to state-of-the-art software-based decompilers. ## constructing a java bytecode dataset we extract java classes from github repositories indexed by google bigquery foot_1 . in order to generate bytecode, we must be able to compile these files with minimal configuration overhead. therefore, we discard files with 3rd party imports (anything not starting with import java.*). additionally, we discard files containing multiple classes. after these preprocessing steps, we split the java classes into a training and testing set with 150k classes and 20k classes, respectively. when building the training and testing set, we first divide at the java project level, setting a project to be either in the training or testing set. additionally, we discard any class that appears multiple times. for each class, we used the java 8 compiler to generate bytecode. this bytecode was then disassembled using krakatau (storyyeller, 2022) to achieve a human-readable bytecode representation. we use this disassembled bytecode representation as input to our nmt model. figure 1 shows the sequence length of java classes and disassembled bytecode representations after being tokenized with the codet5 tokenizer @xcite . for any given java class, the bytecode is often 3-4x longer. following the same approach as (roziere et al., while we use only a small subset of the test dataset in this work, we will release the entire dataset for future research into nmt-based java decompilers. ## conclusions we describe an iterative prompting approach using instruction-tuned llms such as chatgpt to perform decompilation of entire java classes. compared to existing software-based decompilers, our approach achieves a slightly lower pass rate but more human readable code. we hope the iterative prompting approach can be useful in other domains where an automatic feedback mechanism can be used to attempt additional generations.