C ode BERT : A Pre-Trained Model for Programming and Natural Languages (2020)

Abstract: We present CodeBERT, a bimodal pre-trained model for programming language (PL) and natural language (NL). CodeBERT learns general-purpose representations that support downstream NL-PL applications such as natural language code search, code documentation generation, etc. We develop CodeBERT with Transformer-based neural architecture, and train it with a hybrid objective function that incorporates the pre-training task of replaced token detection, which is to detect plausible alternatives sampled from generators. This enables us to utilize both “bimodal” data of NL-PL pairs and “unimodal data, where the former provides input tokens for model training while the latter helps to learn better generators. We evaluate CodeBERT on two NL-PL applications by fine-tuning model parameters. Results show that CodeBERT achieves state-of-the-art performance on both natural language code search and code documentation generation. Furthermore, to investigate what type of knowledge is learned in CodeBERT, we construct a dataset for NL-PL probing, and evaluate in a zero-shot setting where parameters of pre-trained models are fixed. Results show that CodeBERT performs better than previous pre-trained models on NLPL probing.

## introduction large pre-trained models such as elmo @xcite , gpt @xcite , bert @xcite , xlnet @xcite and roberta @xcite have dramatically improved the state-of-the-art on a variety of natural language processing (nlp) tasks. these pre-trained models learn effective contextual representations from massive unlabeled text optimized by self-supervised objectives, such as masked language modeling, which predicts the original masked word from an artificially masked input sequence. the success of pre-trained models in nlp also drives a surge of multi-modal pre-trained models, such as vilbert @xcite for language-image and videobert @xcite for language-video, which are learned from bimodal data such as language-image pairs with bimodal self-supervised objectives. in this work, we present codebert, a bimodal pre-trained model for natural language (nl) and programming language (pl) like python, java, javascript, etc. codebert captures the semantic connection between natural language and programming language, and produces general-purpose representations that can broadly support nl-pl understanding tasks (e.g. natural language code search) and generation tasks (e.g. code documentation generation). it is developed with the multilayer transformer @xcite , which is adopted in a majority of large pre-trained models. in order to make use of both bimodal instances of nl-pl pairs and large amount of available unimodal codes, we train codebert with a hybrid objective function, including standard masked language modeling @xcite and replaced token detection @xcite , where unimodal codes help to learn better generators for producing better alternative tokens for the latter objective. we train codebert from github code reposito-ries in 6 programming languages, where bimodal datapoints are codes that pair with function-level natural language documentations @xcite . training is conducted in a setting similar to that of multilingual bert @xcite , in which case one pre-trained model is learned for 6 programming languages with no explicit markers used to denote the input programming language. we evaluate codebert on two downstream nl-pl tasks, including natural language code search and code documentation generation. results show that fine-tuning the parameters of codebert achieves state-of-the-art performance on both tasks. to further investigate what type of knowledge is learned in codebert, we construct a dataset for nl-pl probing, and test codebert in a zero-shot scenario, i.e. without fine-tuning the parameters of codebert. we find that codebert consistently outperforms roberta, a purely natural language-based pre-trained model. the contributions of this work are as follows: ## codebert we describe the details about codebert in this section, including the model architecture, the input and output representations, the objectives and data used for training codebert, and how to fine-tune codebert when it is applied to downstream tasks. ## experiment we present empirical results in this section to verify the effectiveness of codebert. we first describe the use of codebert in natural language code search ( §4.1), in a way that model parameters of codebert are fine-tuned. after that, we present the nl-pl probing task ( §4.2), and evaluate code-bert in a zero-shot setting where the parameters of codebert are fixed. finally, we evaluate code-bert on a generation problem, i.e. code documentation generation ( §4.3), and further evaluate on a programming language which is never seen in the training phase ( §4.4). ## conclusion in this paper, we present codebert, which to the best of our knowledge is the first large bimodal pre-trained model for natural language and programming language. we train codebert on both bimodal and unimodal data, and show that finetuning codebert achieves state-of-the-art performance on downstream tasks including natural language code search and code-to-documentation generation. to further investigate the knowledge embodied in pre-trained models, we formulate the task of nl-pl probing and create a dataset for probing. we regard the probing task as a cloze-style answer selection problem, and curate distractors for both nl and pl parts. results show that, with model parameters fixed, codebert performs better than roberta and a continuously trained model using codes only. there are many potential directions for further research on this field. first, one could learn better generators with bimodal evidence or more complicated neural architecture to improve the replaced token detection objective. second, the loss functions of codebert mainly target on nl-pl understanding tasks. although codebert achieves strong bleu scores on code-to-documentation generation, the codebert itself could be further improved by generation-related learning objectives. how to successfully incorporate ast into the pretraining step is also an attractive direction. third, we plan to apply codebert to more nl-pl related tasks, and extend it to more programming languages. flexible and powerful domain/language adaptation methods are necessary to generalize well.