Code Execution with Pre-trained Language Models (2023)

Abstract: Code execution is a fundamental aspect of programming language semantics that reflects the exact behavior of the code. However, most pre-trained models for code intelligence ignore the execution trace and only rely on source code and syntactic structures. In this paper, we investigate how well pre-trained models can understand and perform code execution. We develop a mutation-based data augmentation technique to create a large-scale and realistic Python dataset and task for code execution, which challenges existing models such as Codex. We then present CodeExecutor, a Transformer model that leverages code execution pre-training and curriculum learning to enhance its semantic comprehension. We evaluate CodeExecutor on code execution and show its promising performance and limitations. We also demonstrate its potential benefits for code intelligence tasks such as zero-shot code-to-code search and text-to-code generation. Our analysis provides insights into the learning and generalization abilities of pre-trained models for code execution.

## introduction pre-trained models have achieved remarkable results in natural language (nl) tasks @xcite @xcite , inspiring the development of pre-trained models for programming language (pl) tasks @xcite @xcite @xcite . these models leverage source code and code structures, such as abstract syntax tree (ast) @xcite and data flow @xcite , to learn code-related tasks. these structures, while useful, are not sufficient to represent the dynamic behavior of code during execution, which is reflected in the execution trace. using figure 1 as an example, the execution trace shows how code behaves during execution, reflecting the control flow and the state changes of variables. on the other hand, as stated by @xcite , source code contains two channels of information: natural & formal. the natural channel @xcite , such as identifiers and comments, enables language models to be leveraged to understand code-related tasks. the formal channel is used by interpreters and compilers to specify execution and has precise semantics. the formal channel is unique to code and is what makes it executable. execution trace falls into the second category since it reveals the formal channel of information that distinguishes code from natural language, as well as enabling code execution precisely @xcite . in this work, we aim to teach pre-trained models the real-world code execution process. we propose codeexecutor, a transformer-based model that learns to execute arbitrary programs and predict their execution traces. to support pre-training on large-scale data, we construct the python co-denetmut dataset by producing mutations based on submissions to competitive programming problems from codenet @xcite , along with single-line python transformations and programs adapted from python official tutorial. we design a pre-training task that predicts both the line order and the intermediate states of the execution trace, and apply curriculum learning to gradually increase the difficulty of the programs. we evaluate codeexecutor on code execution tasks and show that it outperforms existing models and demonstrates promising capabilities. we also conduct an in-depth analysis of the model's performance and reveal its strengths and weaknesses. furthermore, we show that codeexecutor can improve downstream tasks like zero-shot codeto-code search and text-to-code generation, indicating the potential of leveraging execution trace to @xmath0, w)): @xmath1, w) if @xmath2: print(i + 1) @xmath3: 3 <@xmath4: 3 ; w : 7 <@xmath5: 3 ; w : 7 ; n : 10 <@xmath6: 3 ; w : 7 ; n : 10 ; i : 0 <@xmath7: 3 ; w : 7 ; n : 3 ; i : 0 <@xmath8: 3 ; w : 7 ; n : 3 ; i : 0 <@xmath9: 3 ; w : 7 ; n : 3 ; i : 1 <@xmath10: 3 ; w : 7 ; n : -4 ; i : 1 <@xmath11: 3 ; w : 7 ; n : -4 ; i : 1 <@xmath12: 3 ; w : 7 ; n : -4 ; i : 1 <@xmath13: 3 ; w : 7 ; n : -4 ; i : 1 enhance code intelligence. our models and datasets are publicly available 1 . in summary, the contributions of this paper are: ## mutation-based data augmentation the goal of code execution task is to learn to emulate the execution without running a program by an interpreter. we treat the task as a generation task: given a source code c, the execution trace t is required to be generated. execution trace consists of two components: one is the order in which the computer executes statements, and the other is how the states of the variables change when jumping from one statement to another. normally, the statements inside a program are not executed sequentially, especially in a real-world scenario where programs embody complex logic and rich semantics. more-over, variables relate to various types of data structures with diverse characteristics and operations. given the complexity and difficulty of this task, it is of great importance to build a large-scale dataset and explore the capabilities and boundaries of large language models for code execution. ## codeexecutor our codeexecutor utilizes a transformer-based framework to learn code execution through pretraining. we will first describe the model architecture ( §4.1), then the pre-training task ( §4.2), and finally, the curriculum learning strategy ( §4.3). ## results and analysis in this section, we evaluate codeexecutor on code execution task( §6.1), conduct an in-depth analysis to understand model behavior and error mode ( §6.2), followed by two downstream tasks ( §6.3).