U ni X coder: Unified Cross-Modal Pre-training for Code Representation (2022)

Abstract: Pre-trained models for programming languages have recently demonstrated great success on code intelligence. To support both code-related understanding and generation tasks, recent works attempt to pre-train unified encoder-decoder models. However, such encoder-decoder framework is sub-optimal for auto-regressive tasks, especially code completion that requires a decoder-only manner for efficient inference. In this paper, we present UniXcoder, a unified cross-modal pre-trained model for programming language. The model utilizes mask attention matrices with prefix adapters to control the behavior of the model and leverages cross-modal contents like AST and code comment to enhance code representation. To encode AST that is represented as a tree in parallel, we propose a one-to-one mapping method to transform AST in a sequence structure that retains all structural information from the tree. Furthermore, we propose to utilize multi-modal contents to learn representation of code fragment with contrastive learning, and then align representations among programming languages using a cross-modal generation task. We evaluate UniXcoder on five code-related tasks over nine datasets. To further evaluate the performance of code fragment representation, we also construct a dataset for a new task, called zero-shot code-to-code search. Results show that our model achieves state-of-the-art performance on most tasks and analysis reveals that comment and AST can both enhance UniXcoder.

## introduction pre-trained models such as gpt @xcite and bert @xcite have substantially advanced the state of the art across numerous natural language processing (nlp) tasks. these pretrained models are pre-trained on large amounts of text data with self-supervised objectives, and can be fine-tuned to adapt to downstream tasks. inspired by the success of pre-trained models in nlp, pre-trained models for programming languages (pl) @xcite @xcite have been proposed to promote the development of code intelligence. @xcite proposes gpt-c that employs a left-to-right transformer @xcite to support generation tasks such as code completion, but the unidirectional framework is sub-optimal for understanding tasks. in contrast, other works @xcite pre-train a bidirectional transformer encoder on source code, which significantly improves the performance of code-related understanding tasks. however, its bidirectionality nature requires an additional decoder when applied to generation tasks, where this decoder initializes from scratch and cannot benefit from the pre-training. in this work, we present unixcoder, a unified cross-modal pre-trained model for programming languages to support both code-related understanding and generation tasks. unixcoder is based on a multi-layer transformer and follows @xcite to utilize mask attention matrices with prefix adapters to control the access to context for each token. compared with current unified encoderdecoder models @xcite on code intelligence, unixcoder can be better applied to auto-regressive tasks such as code completion that requires a decoder-only manner to perform efficient inference in practice. instead of taking code as the only input, we also consider multi-modal contents like code comment and abstract syntax tree (ast) to enhance code representation. generally, user-written code comments provide crucial semantic information about source code like "sort a given list" and ast contains rich syntax information like types of statements and nested relationship among them, which helps the model better understand source code. to encode ast that is represented as a tree in parallel, we propose a one-to-one mapping method to transform ast in a sequence structure that retains all information of the tree and then the sequence can be used as the input to enhance code representation. we pre-train unixcoder using three types of language modeling tasks: masked language modeling @xcite , unidirectional language modeling @xcite and denoising objective @xcite , which can enable the model to support various types of downstream tasks. furthermore, we introduce two pre-training tasks to learn a embedding that can represent semantics of a code fragment. one is multi-modal contrastive learning that leverages ast to enhance semantics of code fragment embeddings, and the other is cross-modal generation that utilizes code comment to align embeddings among programming languages. we evaluate unixcoder on five tasks over nine public datasets, including two understanding tasks: clone detection and code search, two generation tasks: code summarization and code generation, and an auto-regressive task: code completion. to further test code fragment embeddings, we propose a new task, called zero-shot code-to-code search, and construct a new dataset from codenet corpus (puri et al., 2021) for this task. experimental results show that our model achieves state-of-the-art performance on most tasks. further analysis reveals that ast and code comment can both enhance unixcoder to better capture code semantics. in summary, the contributions of this paper are: (1) we propose a unified cross-modal pre-trained model that leverages multi-modal contents, i.e. code comment and ast, to support code-related understanding, generation tasks and auto-regressive tasks. (2) we propose a one-to-one mapping function that converts ast into a sequence that retains all information of ast and can be encoded with source code and comment in parallel. (3) we further propose to utilize code comment to learn code fragment representation and construct a new dataset for zero-shot code-code search to evaluate the quality of code fragment representation. (4) experimental results show that unixcoder provides significant improvement on most downstream tasks. 1 1 all the codes and data are available at https:// github.com/microsoft/codebert . ## related works with the great success of pre-training in natural language (nl) processing @xcite @xcite , pre-trained models for programming languages have been proposed to promote the development of code intelligence. these pre-trained models can be generally divided into three categories: encoderonly, decoder-only, and encoder-decoder models. encode-only models @xcite @xcite @xcite pre-train a bidirectional transformer in which each token can attend to each other. @xcite pre-train cubert on a corpus of python source codes by masked language modeling and next sentence prediction objectives. codebert @xcite ) is pre-trained on nl-pl pairs in six programming languages with a new pre-training task, namely replace token detection. @xcite leverages data flow to enhance code representation, while syn-cobert @xcite incorporates abstract syntax tree by ast edge prediction and contrastive learning. however, encoder-only models require an additional decoder for generation tasks, where this decoder initializes from scratch and cannot benefit from the pre-training. as for decoder-only pre-trained models, svyatkovskiy et al. (2020) and @xcite respectively propose gpt-c and codegpt, which are both pre-trained using unidirectional language modeling that only allows tokens to attend the previous tokens and itself to predict the next token. decoder-only models are good at auto-regressive tasks like code completion, but the unidirectional framework is sub-optimal for understanding tasks. some recent works explore encoder-decoder models to support both understanding and generation tasks. plbart @xcite is based on the bart @xcite architecture and pre-trained on nl and pl corpus using denoising objectives. codet5 @xcite adapts the t5 @xcite model that considers the crucial token type information from identifiers and allow for multi-task learning on downstream tasks. treebert @xcite follows the encoderdecoder transformer framework but utilizes the tree structural information by modeling ast paths. different from current unified models, unixcoder is based on a multi-layer transformer and utilizes mask attention matrices with prefix adapters ## unixcoder in this section, we describe unixcoder, a unified cross-modal pre-trained model that leverages multimodal data (i.e. code comment and ast) to pretrain code representation. the model is based on transformer and utilizes mask attention matrices @xcite with prefix adapters to control the behavior of the model. in the following, we first introduce how to unify multi-modal data as the input of unixcoder ( §3.1), and then the model architecture ( §3.2) and pre-training tasks ( §3.3). ## experiments we evaluate unixcoder on five tasks over nine public datasets, including two understanding tasks ( §4.2), two generation tasks ( §4.3) and an autoregressive task ( §4.4). to further evaluate the performance of code fragment embeddings, we also propose a new task called zero-shot code-to-code search ( §4.5). more details about datasets and finetuning can be found in the appendix c. ## conclusion to support both code-related understanding and generation tasks, we present unixcoder, a unified pre-trained model that incorporates semantic and syntax information from code comment and ast. we propose a one-to-one mapping method to transform ast to a sequence structure and two new pre-training tasks to learn code fragment representation. to further investigate the performance of code representation, we propose a new downstream task of zero-shot code-to-code search and create a dataset for this task. experiments show that unixcoder significantly outperforms previous works on most tasks. further ablation studies also show that both ast and code comment can enhance unixcoder and reveal the effectiveness of our proposed mapping function and pre-training tasks.