C ode T 5: Identifier-aware Unified Pre-trained Encoder-Decoder Models for Code Understanding and Generation (2021)

Abstract: Pre-trained models for Natural Languages (NL) like BERT and GPT have been recently shown to transfer well to Programming Languages (PL) and largely benefit a broad set of code-related tasks. Despite their success, most current methods either rely on an encoder-only (or decoder-only) pre-training that is suboptimal for generation (resp. understanding) tasks or process the code snippet in the same way as NL, neglecting the special characteristics of PL such as token types. We present CodeT5, a unified pre-trained encoder-decoder Transformer model that better leverages the code semantics conveyed from the developer-assigned identifiers. Our model employs a unified framework to seamlessly support both code understanding and generation tasks and allows for multi-task learning. Besides, we propose a novel identifier-aware pre-training task that enables the model to distinguish which code tokens are identifiers and to recover them when they are masked. Furthermore, we propose to exploit the user-written code comments with a bimodal dual generation task for better NL-PL alignment. Comprehensive experiments show that CodeT5 significantly outperforms prior methods on understanding tasks such as code defect detection and clone detection, and generation tasks across various directions including PL-NL, NL-PL, and PL-PL. Further analysis reveals that our model can better capture semantic information from code. Our code and pre-trained models are released at https://github.com/salesforce/CodeT5 .

## introduction pre-trained language models such as bert @xcite , gpt @xcite , and t5 @xcite have greatly boosted performance in a wide spectrum of natural language processing (nlp) tasks. they typically employ a pre-train then fine-tune paradigm that aims to derive generic language representations by selfsupervised training on large-scale unlabeled data, which can be transferred to benefit multiple downstream tasks, especially those with limited data annotation. inspired by their success, there are many recent attempts to adapt these pre-training methods for programming language (pl) @xcite @xcite , showing promising results on code-related tasks. however, despite their success, most of these models rely on either an encoder-only model similar to bert @xcite or a decoder-only model like gpt @xcite , which is suboptimal for generation and understanding tasks, respectively. for example, codebert @xcite ) requires an additional decoder when applied for the code summarization task, where this decoder cannot benefit from the pre-training. besides, most existing methods simply employ the conventional nlp pretraining techniques on source code by regarding it as a sequence of tokens like nl. this largely ignores the rich structural information in code, which is vital to fully comprehend the code semantics. in this work, we present codet5, a pre-trained encoder-decoder model that considers the token type information in code. our codet5 builds on the t5 architecture @xcite ) that employs denoising sequence-to-sequence (seq2seq) pre-training and has been shown to benefit both understanding and generation tasks in natural language. in addition, we propose to leverage the developer-assigned identifiers in code. when writing programs, developers tend to employ informative identifiers to make the code more understandable, so that these identifiers would generally preserve rich code semantics, e.g., the "binarysearch" identifier in figure 2 directly indicates its functionality. to fuse such code-specific knowledge, we propose a novel identifier-aware objective that trains the model to distinguish which tokens are identifiers and recover them when they are masked. ## codet5 our codet5 builds on an encoder-decoder framework with the same architecture as t5 @xcite . it aims to derive generic representations for programming language (pl) and natural language (nl) via pre-training on unlabeled source code. as illustrated in figure 2 , we extend the de-noising seq2seq objective in t5 by proposing two identifier tagging and prediction tasks to enable the model to better leverage the token type information from pl, which are the identifiers assigned by developers. to improve the nl-pl alignment, we further propose a bimodal dual learning objective for a bidirectional conversion between nl and pl. in the following, we introduce how codet5 encodes pl and nl inputs ( §3.1) and our proposed identifier-aware pre-training tasks ( §3.2), followed by the fine-tuning with task-specific transfer learning and multi-task training ( §3.3). ## conclusion we have presented codet5, a pre-trained encoderdecoder model that incorporates the token type information from code. we propose a novel identifieraware pre-training objective to better leverage the identifiers and propose a bimodal dual generation task to learn a better nl-pl alignment using code and its comments. our unified model can support both code understanding and generation tasks and allow for multi-task learning. experiments show that codet5 significantly outperforms all prior work in most codexglue tasks. further analysis also reveals its better code comprehension capability across various programming languages.