CODE - MVP : Learning to Represent Source Code from Multiple Views with Contrastive Pre-Training (2022)

Abstract: Recent years have witnessed increasing interest in code representation learning, which aims to represent the semantics of source code into distributed vectors. Currently, various works have been proposed to represent the complex semantics of source code from different views, including plain text, Abstract Syntax Tree (AST), and several kinds of code graphs (e.g., Control/Data Flow Graph). However, most of them only consider a single view of source code independently, ignoring the correspondences among different views. In this paper, we propose to integrate different views with the natural-language description of source code into a unified framework with Multi-View contrastive Pre-training, and name our model as CODE-MVP. Specifically, we first extract multiple code views using compiler tools, and learn the complementary information among them under a contrastive learning framework. Inspired by the type checking in compilation, we also design a fine-grained type inference objective in the pre-training. Experiments on three downstream tasks over five datasets demonstrate the superiority of CODE-MVP when compared with several state-of-the-art baselines. For example, we achieve 2.4/2.3/1.1 gain in terms of MRR/MAP/Accuracy metrics on natural language code retrieval, code similarity, and code defect detection tasks, respectively.

## introduction code intelligence that utilizes machine learning techniques to promote the productivity of software developers, has attracted increasing interest in both communities of software engineering and artificial intelligence @xcite @xcite @xcite . to achieve code intelligence, one fundamental task is code representation learning (also current approaches to code representation borrow ideas from the successful deep learning methods in natural language processing, mainly attributed to the naturalness hypothesis in source code @xcite . from our investigation, existing approaches mainly represent the source code from different views of code, including code token in plain text @xcite , abstract syntax tree (ast) @xcite , and control/data flow graphs (cfgs/dfgs) of code @xcite . recently, many attempts have been made to pretrain a masked language model for source code, such as codebert @xcite , graphcode-bert @xcite , syncobert @xcite , codegpt @xcite , plbart @xcite , cotext @xcite , and codet5 @xcite . despite much progress in code representation learning, most of them only consider a single view of source code independently, ignoring the consistency among different views @xcite @xcite . usually, a program, accompanied by a corresponding natural-language comment (nl), can be parsed into multiple views, e.g., the source code tokens, ast, and cfg. we argue that these different views contain complementary semantics of the program. for example, the source code tokens (e.g., method name identifiers) and naturallanguage comments always reveal the lexical semantics of code, while the intermediate structures of code (e.g., ast and cfg) always reveal the syntactic and executive information of code. in addition, a program can also be transformed (or rewritten) into different variants that have equivalent functionality. we think that different variants of the same program reveal the functional information of code. that is, those different program variants with the same functionality are expected to represent the same semantics. inspired by the aforementioned insights, this paper proposes a novel code-mvp for code representation, which aims to integrate multiple views of the code into a unified framework with multi-view contrastive pre-training. concretely, we first extract multiple views of code using several compiler tools, and learn the complementary information among them under a multi-view contrastive learning framework. meanwhile, inspired by the type checking in compilation process, we also introduce fine-grained type inference as an auxiliary task in the pre-training process to encourage the model to learn more fine-grained type information. to summarize, the contributions of this paper are two-fold: (1) we are the first to represent source code from multiple views, including the code tokens, ast, cfg, and various program equivalents, under a unified multi-view contrastive pre-training framework. meanwhile, we also introduce an auxiliary task of inferring type annotations for variables. (2) we extensively evaluate code-mvp on three program comprehension tasks. experimental results demonstrate the superiority of code-mvp when compared with several state-of-theart baselines. specifically, code-mvp achieves 2.4/2.3/1.1 gain on mrr/map/accuracy metrics in natural language code retrieval, code similarity, and code defect detection tasks, respectively. ## multiple views of code we borrow ideas from the way that computers process the source code in compilation, where a program would be converted into multiple views. figure 1 shows the process of converting a program from source code to machine code. during this process, the compiler would automatically utilize some program analysis techniques to verify the correctness of source code, including lexical, syntax, and semantic analyses. in the lexical analysis, a program is treated as a sequence of tokens and checked for spelling problems. in the syntax analysis, syntactic rules of programs are defined by the context-free grammar @xcite . then the program could be parsed as an ast, based on which many program transformation heuristics can be applied to rewrite the program while maintaining the same desired functionality. in the semantic analysis, semantic rules of the program are defined by the attribute grammar @xcite . then the compiler could check the types of code tokens, and a decorated ast could be obtained. after the three stages above, a translator will convert the source code to its intermediate representation (ir), which is then considered as the basis for building control/data flow graphs (cfgs/dfgs) for further optimizations in the static analysis. finally, the ir of the source code should be converted into machine code to execute through a code generator. next, we introduce how we extract different views of the source code. figure 2 illustrates multiple views of source code with an example. ## conclusion in this paper, we have proposed code-mvp, a novel approach to represent the source code with multi-view contrastive pre-training learning. we extract multiple code views with compiler tools and learn the complement among them under a contrastive learning framework. we also propose a fine-grained type inference task in the pre-training process. comprehensive experiments on three downstream tasks over five datasets verify the effectiveness of code-mvp when compared with several state-of-the-art baselines.