Unified Pre-training for Program Understanding and Generation (2021)

Abstract: Code summarization and generation empower conversion between programming language (PL) and natural language (NL), while code translation avails the migration of legacy code from one PL to another. This paper introduces PLBART, a sequence-to-sequence model capable of performing a broad spectrum of program and language understanding and generation tasks. PLBART is pre-trained on an extensive collection of Java and Python functions and associated NL text via denoising autoencoding. Experiments on code summarization in the English language, code generation, and code translation in seven programming languages show that PLBART outperforms or rivals state-of-the-art models. Moreover, experiments on discriminative tasks, e.g., program repair, clone detection, and vulnerable code detection, demonstrate PLBART’s effectiveness in program understanding. Furthermore, analysis reveals that PLBART learns program syntax, style (e.g., identifier naming convention), logical flow (e.g., “if“ block inside an “else“ block is equivalent to “else if“ block) that are crucial to program semantics and thus excels even with limited annotations.

## introduction engineers and developers write software programs in a programming language (pl) like java, python, etc., and often use natural language (nl) to communicate with each other. use of nl in software engineering ranges from writing documentation, commit messages, bug reports to seeking help in different forums (e.g., stack overflow), etc. automating different software engineering applications, such as source code summarization, generation, and translation, heavily rely on the understanding of pl and nl-we collectively refer them as plug (stands for, program and language understanding and generation) applications or tasks. summary: sort a list of tuples by first element note that the use of nl in software development is quite different than colloquially written and spoken language. for example, nl in software development often contains domain-specific jargon, e.g., when software engineers use code smell foot_0 , it means a potential problem in code (something other than smell in regular english language). in this work, our goal is to develop a generalpurpose model that can be used in various plug applications. recent advancements in deep learning and the availability of large-scale pl and developers' nl data ushered in the automation of plug applications. one important aspect of plug applications is that they demand a profound understanding of program syntax and semantics and mutual dependencies between pl and nl. for example, figure 1 shows two implementations of the same algorithm (sorting) in two pl and corresponding nl summary. an automatic translation tool must understand that function sorted in python acts similar to arrays.sort in java and the lambda operation in python is equivalent to instantiating a comparator object in java. similarly, a tool that summarizes either of these code must understand that x[0] in python or tuple.get(0) in java refers to the first element in the tuple list. most of the available data in pl and nl are unlabeled and cannot be trivially used to acquire plug task-specific supervision. however, plug tasks have a common prerequisite -understanding pl and nl syntax and semantics. leveraging unlabelled data to pretrain a model to learn pl and nl representation can be transferred across plug tasks. this approach reduces the requirement of having large-scale annotations for taskspecific fine-tuning. in recent years we have seen a colossal effort to pretrain models on a massive amount of unlabeled data (e.g., text, images, videos) @xcite @xcite @xcite to transfer representation encoders across a wide variety of applications. there are a few research effort in learning general purpose pl-nl representation encoders, such as codebert @xcite and @xcite that are pretrained on a small-scale bimodal data (code-text pairs). such models have been found effective for plug tasks, including code search, code completion, etc. language generation tasks such as code summarization is modeled as sequence-to-sequence learning, where an encoder learns to encode the input code and a decoder generates the target summary. despite the effectiveness of existing methods, they do not have a pretrained decoder for language generation. therefore, they still require a large amount of parallel data to train the decoder. to overcome this limitation, @xcite proposed denoising sequence-to-sequence pre-training where a transformer @xcite learns to reconstruct an original text that is corrupted using an arbitrary noise function. very recently, @xcite studied denoising pre-training using a largescale source code collection aiming at unsupervised program translation and found the approach useful. this raises a natural question, can we unify pretraining for programming and natural language? presumably, to facilitate such pre-training, we need unlabeled nl text that is relevant to software development. note that unlike other bimodal scenarios (e.g., vision and language), pl and associated nl text share the same alphabet or uses anchor tokens java python nl all size 352 gb 224 gb 79 gb all -nb of tokens 36.4 b 28 b 6.7 b all -nb of documents 470 m 210 m 47 m we introduce plbart (program and language bart), a bidirectional and autoregressive transformer pre-trained on unlabeled data across pl and nl to learn multilingual representations applicable to a broad spectrum of plug applications. we evaluate plbart on code summarization, generation, translation, program repair, clone detection, and vulnerability detection tasks. experiment results show that plbart outperforms or rivals stateof-the-art methods, e.g., codebert and graph-codebert, demonstrating its promise on program understanding and generation. we perform a thorough analysis to demonstrate that plbart learns program syntax, logical data flow that is indispensable to program semantics, and excels even when limited annotations are available. we release our code foot_1 to foster future research. ## plbart plbart uses denoising sequence-to-sequence pretraining to utilize unlabeled data in pl and nl. such pre-training lets plbart reason about language syntax and semantics. at the same time, plbart learns to generate language coherently. we tokenize all the data with a sentencepiece model @xcite ) learned on 1/5'th of the pre-training data. we train sentencepiece to learn 50,000 subword tokens. one key challenge to aggregate data from different modalities is that some modalities may have more data, such as we have 14 times more data in pl than nl. therefore, we mix and up/down sample the data following @xcite to alleviate the bias towards pl. we sample instances for pre-training according to a multinomial distribution with probabilities (q 1 , q 2 , . . . , q n ): architecture plbart uses the same architecture as bart base @xcite , it uses the sequence-to-sequence transformer architecture @xcite , with 6 layers of encoder and 6 layers of decoder with model dimension of 768 and 12 heads (∼140m parameters). the only exception is, we include an additional layernormalization layer on top of both the encoder and decoder following @xcite , which is found to stabilize training with fp16 precision. ## experiment setup to understand plbart's performance in a broader context, we evaluate plbart on several tasks. our evaluation focuses on assessing plbart's ability to capture rich semantics in source code and associated natural language text. ## results & analysis we aim to address the following questions. 1. does plbart learn strong program and language representations from unlabeled data? 2. does plbart learn program characteristics, e.g., syntax, style, and logical data flow? 3. how does plbart perform in an unseen language with limited annotations? ## related work pre-training for language understanding and generation transformer @xcite , a sequence-to-sequence architecture that includes an encoder and decoder, has shown tremendous promise in natural language processing (nlp), computer vision, software engineering, and more. @xcite first proposed to pre-train a large transformer architecture, called bert, to learn representations of natural language using large-scale unlabeled data in a self-supervised fashion. later, bert's task-independent pre-training approach is rigorously studied @xcite @xcite @xcite . while bert-like models have shown effectiveness in learning contextualized representation, it is not very useful in generation tasks. gpt @xcite style models improve upon bert for generative tasks with autoregressive pre-training; however, unlike bert, they are not bidirectional. @xcite introduced bart, a denoising autoencoder that uses a bidirectional encoder and an auto-regressing decoder. similar to bart, plbart uses denoising pre-training to cope with generative tasks and learns multilingual representations of programming and natural language jointly. ## conclusion this paper presents plbart, a sizeable pre-trained sequence-to-sequence model that can perform program and language understanding and generation tasks. plbart achieves state-of-the-art performance on various downstream software engineering tasks, including code summarization, code generation, and code translation. furthermore, experiments on discriminative tasks establish plbart's effectiveness on program understanding. we also show that plbart learns crucial program characteristics due to pre-training, such as syntax, identifier naming conventions, data flow. in the future, we want to explore ways to fine-tune plbart on all the downstream tasks jointly.