Code Representation Pre-training with Complements from Program Executions (2024)

Abstract: Language models for natural language processing have been grafted onto programming language modeling for advancing code intelligence. Although it can be represented in the text format, code is syntactically more rigorous, as it is designed to be properly compiled or interpreted to perform a set of behaviors given any inputs. In this case, existing works benefit from syntactic representations to learn from code less ambiguously in forms of abstract syntax tree, control-flow graph, etc. However, programs with the same purpose can be implemented in various ways showing different syntactic representations, while the ones with similar implementations can have distinct behaviors. Though trivially demonstrated during executions, such semantics about functionality are challenging to be learned directly from code, especially in an unsupervised manner. Hence, in this paper, we propose FuzzPretrain to explore the dynamic information of programs revealed by their test cases and embed it into the feature representations of code as complements. The test cases are obtained with the assistance of a customized fuzzer and are only required during pre-training. FuzzPretrain yielded more than 6%/19% mAP improvements on code search over its masked language modeling counterparts trained with only source code and source code coupled with abstract syntax trees (ASTs), respectively. Our experiments show the benefits of learning discriminative code representations from FuzzPretrain.

## introduction code representation learning is drawing growing attention across the community of artificial intelligence (ai) and software engineering (se) @xcite @xcite @xcite . the pre-training recipes (devlin et al., 2019; liu * corresponding author et al., 2019) for natural languages have been shown effective in code representation learning @xcite . these methods leverage source code and code structures, such as abstract syntax tree (ast) @xcite and control-flow graph (cfg) @xcite , to learn code representation. however, these structures are not sufficient for code representation, as they neglect the dynamic behavior of code, which is reflected in program execution @xcite . therefore, some works @xcite @xcite proposed to learn program embedding from the combination of symbolic and concrete execution behaviors. specially, @xcite proposed fuzztuning that utilized fuzzing @xcite to generate input-output pairs of programs for code-related downstream tasks through fine-tuning with these test cases. the underlying motivation is that the relationship between inputs and their corresponding outputs essentially represents the functions or subroutines, and ultimately, the entire program. although effective, these methods necessitate the use of input-output pairs during the inference process. yet, obtaining high-quality input-output pairs during inference can be timeconsuming and requires intricate engineering, thus posing challenges for practical implementation. in this work, we aim to embed the input-output relationships (represented by test cases) of code into its feature representations pre-training instead of fine-tuning, to address the dependency on realtime fuzzing during inference. to accomplish this goal, we follow @xcite to take advantage of fuzzing to produce test cases that cover the logic paths of code as comprehensively as possible. however, in this paper, these test cases are used in pre-training instead of fine-tuning. we propose a novel method called fuzzpretrain for joint static and dynamic information modeling. particularly, in addition to exploring code structure by masked language modeling @xcite , it formulates a dynamic information matching (dim) pretext task to tell test cases of different programs apart according to their correspondence to code. by doing so, the model learns holistic feature representations of code and its test cases, encoding both the structure and functionality. fuzzpretrain also involves a self-distillation objective to accomplish a dynamic information distillation (did) objective. thereby, the dynamic information is not only properly modelled but distilled from the holistic representations to code features, so to benefit in practice where the test cases are not available. we make three contributions in this paper: (1) we propose to leverage the test cases of programs obtained with the help of fuzzing as explicit indications of functionality to complement their code and syntactic representations. to the best of our knowledge, this is the first attempt to unveil the benefits of fuzzing to code representation pre-training. ## related work code representation learning. language models @xcite @xcite have achieved unprecedented breakthroughs in natural language processing in recent years @xcite @xcite . such successes of language models have been consistently transferred to code representation learning and advance code intelligence. these works leverage plain code @xcite @xcite and code structures, such as abstract syntax tree (ast) @xcite and control-flow graph (cfg) @xcite for code representation learning. however, as illustrated by @xcite , due to the inherent gap between program syntax and runtime semantics, models learned from source code and code structure (i.e., the static models) can be imprecise and not deep at capturing semantic properties. more recent approaches @xcite @xcite attempted to use dynamic execution traces to learn program representation. by considering dynamic execution paths, symbolic traces provide precise information about dynamic program behavior and reduce false-positive rates in code related tasks. ## experiments we adopted the benchmark datasets introduced by @xcite to be fuzzed as the training data of our experiments, which are composed of 1.2m code snippets implemented in c++/python/java. we want to emphasize that our fuzzpretrain is a generic method that can be integrated into many other static-based models more than codebert. to verify this, we perform experiments with one more base model called unixcoder @xcite , which was trained with code and ast. correspondingly, we denote the variant of fuzzpretrain built upon unixcoder as fuzzunixcoder. we followed the base models, i.e., codebert @xcite and unixcoder @xcite to take a 12-layer transformer with 125m learnable parameters for sequence encoding. we trained fuzzpretrain for 10k steps by the adam optimizer @xcite , which took around 12/20 hours on 8 nvidia v100 gpus for code and ast, respectively. for hyperparameter selections, we carefully aligned with our base models as well as @xcite regarding l did (eq. ( 3 )). we evaluated fuzzpretrain on four standard code understanding benchmarks adopted by @xcite including code-to-code search (abbreviated as code search) on codenet, clone detection on poj-104 @xcite , defect detection on devign @xcite ) and text-to-code search (abbreviated as text search) on cosqa @xcite . we adopted mean average precision (map) as the evaluation metric for code search and clone detection, accuracy for defect detection and mean reciprocal rank (mrr) for text search. more details about our implementation and evaluation protocols can be found in appendix a. note that test cases are only used in our unsupervised pre-training phase and never used in any downstream tasks in experiments. ## conclusion in this paper, we have made the first attempt to use (fuzzing) test cases to facilitate effective code representation pre-training. to benefit from such a "new modality" of data that is often not available in downstream tasks, we have proposed fuzzpretrain for joint static and dynamic information modeling. specifically, fuzzpretrain is trained not only to accomplish the conventional masked tokens predictions objective but also to learn the input-output relationships from test cases encoding the programspecific runtime behaviors, as well as enforcing the model to infer such dynamic knowledge from code structures solely. we have shown how fuzzpretrain can be used to enhance codebert and unixocder. extensive experiments on various code understanding downstream tasks demonstrate the benefits of our fuzzpretrain.