C o T ex T : Multi-task Learning with Code-Text Transformer (2021)

Abstract: We present CoTexT, a pre-trained, transformer-based encoder-decoder model that learns the representative context between natural language (NL) and programming language (PL). Using self-supervision, CoTexT is pre-trained on large programming language corpora to learn a general understanding of language and code. CoTexT supports downstream NL-PL tasks such as code summarizing/documentation, code generation, defect detection, and code debugging. We train CoTexT on different combinations of available PL corpus including both “bimodal” and “unimodal” data. Here, bimodal data is the combination of text and corresponding code snippets, whereas unimodal data is merely code snippets. We first evaluate CoTexT with multi-task learning: we perform Code Summarization on 6 different programming languages and Code Refinement on both small and medium size featured in the CodeXGLUE dataset. We further conduct extensive experiments to investigate CoTexT on other tasks within the CodeXGlue dataset, including Code Generation and Defect Detection. We consistently achieve SOTA results in these tasks, demonstrating the versatility of our models.

## introduction in recent years, pre-trained language models (lm) have played a crucial role in the development of many natural language processing (nlp) systems. before the emergence of large lms, traditional word embedding gives each word/token a global representation. large pre-trained models such as elmo @xcite , @xcite , bert @xcite , and xlnet @xcite ) can derive contextualized word vector representations from large corpora. these methods can learn generalized representations of language and have significantly improved a broad range of downstream nlp tasks. these lms make use of learning objectives such as masked language modeling (mlm) @xcite where random tokens in a sequence are masked and the model predicts the original tokens to learn the context. the success of pre-trained models in nlp has created a path for domain-specific pretrained lms, such as biobert @xcite on biomedical text, or tabert @xcite on nl text and tabular data. we introduce cotext (code and text transfer transformer), a pre-trained model for both natural language (nl) and programming language (pl) such as java, python, javascript, php, etc. cotext follows the encoder-decoder architecture proposed by @xcite with attention mechanisms. we then adapt the model to match t5 framework proposed by @xcite . we test cotext by performing exhaustive experiments on multi-task learning of multiple programming languages and other related tasks. we train cotext using large programming language corpora containing multiple programming languages (including java, python, javascript, ruby, etc.). here, we test different combinations of unimodal and bimodal data to produce the best result for each downstream task. we then finetune cotext on four codexglue tasks @xcite including codesummarization, codegeneration, defect detection and code refinement (small and medium dataset). results show that we achieve state-of-the-art values for each of the four tasks. we found that cotext outperforms current sota models such as codebert @xcite and @xcite . in this paper we offer the following contribution: recent work on domain adaptation of bert show improvements compared to the general bert model. biobert @xcite ) is further trained from bert base on biomedical articles such as pubmed abstracts and pmc articles. similarly, scibert @xcite is trained on the full text of biomedical and computer science papers. the experimental results of these models on domain-specific datasets show the enhanced performance compared to bert base . relating specfically to our work, codebert is @xcite trained on bimodal data of nl-pl pairs. this strategy allows codebert to learn general-purpose representations of both natural language and programming language. graphcode-bert @xcite is an extension of code-bert that moves beyond syntactic-level structure and uses data flow in the pre-training stage to capture the semantic-level structure of code. more recently, @xcite ) is a pretrained sequence-to-sequence model for nl and pl. through denoising autoencoding, this model can perform well on nl-pl understanding and generation tasks. ## experiments in this section, we will first describe the benchmark dataset for code intelligence codexglue, then we will explain the experimental setup on the tasks we perform and discuss the results of each task. the evaluation datasets are summarized in ## conclusion in this manuscript, we introduced cotext, a pretrained language representation for both programming language and natural language. cotext focused on text-code and code-text understanding and generating. leveraging the t5 framework @xcite , we showed that pre-training on a large programming language corpus is effective for a diverse array of tasks within the natural language and programming language domain. cotext achieves state-of-the-art results on 4 codexglue code intelligence tasks: code summarization, code generation, code refinement, and code detection. for future work, we plan to test cotext on a broader range of programming language and natural language generation tasks, such as autocompletion or code translation.