C ode T 5+: Open Code Large Language Models for Code Understanding and Generation (2023)

Abstract: Large language models (LLMs) pretrained on vast source code have achieved prominent progress in code intelligence. However, existing code LLMs have two main limitations. First, they often adopt a specific architecture (encoder-only or decoder-only) or rely on a unified encoder-decoder network for different downstream tasks, lacking the flexibility to operate in the optimal architecture for a specific task. Secondly, they often employ a limited set of pretraining objectives which might not be relevant to some tasks and hence result in substantial performance degrade. To address these limitations, we propose “CodeT5+”, a family of encoder-decoder LLMs for code in which component modules can be flexibly combined to suit a wide range of code tasks. Such flexibility is enabled by our proposed mixture of pretraining objectives, which cover span denoising, contrastive learning, text-code matching, and causal LM pretraining tasks, on both unimodal and bimodal multilingual code corpora. Furthermore, we propose to initialize CodeT5+ with frozen off-the-shelf LLMs without training from scratch to efficiently scale up our models, and explore instruction-tuning to align with natural language instructions. We extensively evaluate CodeT5+ on over 20 code-related benchmarks in different settings, including zero-shot, finetuning, and instruction-tuning. We observe state-of-the-art (SoTA) performance on various code-related tasks, and our instruction-tuned CodeT5+ 16B achieves new SoTA results of 35.0% pass@1 and 54.5% pass@10 on the HumanEval code generation task against other open code LLMs, even surpassing the OpenAI code-cushman-001 model.

## introduction large language models (llms) @xcite @xcite have recently demonstrated remarkable success in a broad set of downstream tasks in the code domain @xcite @xcite . by pretraining on massive code-based data (e.g. github public data), these code llms can learn rich contextual representations which can be transferred to various code-related downstream tasks. however, we found that many existing models are designed to perform well only in a subset of tasks. we argue that this is mainly due to two limitations in terms of architecture and pretraining tasks. from an architectural perspective, existing code llms often adopt encoder-only or decoder-only models that perform well only on certain understanding or generative tasks. specifically, encoderonly models @xcite are often used to facilitate understanding tasks such as text-to-code retrieval @xcite . for generative tasks such as code generation @xcite , decoder-only models @xcite often demonstrate stronger performance. however, these decoder-only models are often not ideal for understanding tasks such as detection tasks compared to encoder-only models @xcite . besides, several models have adopted more unified encoder-decoder architectures @xcite to adapt to different types of tasks. while these models can support both understanding and generative tasks, they still suffer from suboptimal performance on certain tasks. @xcite found that encoder-decoder models fail to beat (state-of-the-art) sota encoder-only or decoder-only baselines on retrieval and code completion tasks respectively. this shortfall is due to the limitation of the single-module architecture generally adapted to all tasks. in summary, prior approaches are not designed with compositionality such that individual components can be activated to better suit different types of downstream tasks. from a learning objective perspective, current models employ a limited set of pretraining tasks. these tasks can lead to performance degrade on certain downstream tasks due to the discrepancy between the pretraining and finetuning stage. for instance, t5-based models such as @xcite are often trained with a span denoising objective. however, in downstream tasks such as code generation @xcite , most sota models are pretrained with a next-token prediction objective which autoregressively predicts a program token by token. furthermore, many models are not trained to learn contrastive code representations that are vital for understanding tasks such as text-to-code retrieval. although recent attempts @xcite introduce a contrastive learning task to alleviate this issue, these approaches ignore the fine-grained text-code cross-modal alignments. to address the above limitations, we propose "codet5+", a new family of encoder-decoder code foundation llms for a wide range of code understanding and generation tasks (see fig. 1 for an overview). despite being an encoder-decoder based model, our codet5+ can flexibly operate in encoder-only, decoder-only, and encoder-decoder modes to suit different downstream applications. such flexibility is enabled by our proposed pretraining tasks, which include span denoising and causal language modeling (clm) tasks on code data and text-code contrastive learning, matching, and clm tasks on text-code data. we found that such a wide set of pretraining tasks can help learn rich representations from both code and text data, and bridge the pretrain-finetune gap in various downstream applications. besides, we show that the integration of the matching task with contrastive learning is crucial to capture the fine-grained text-code alignments and improve retrieval performance. furthermore, we scale up the model size of codet5+ with a compute-efficient pretraining strategy by leveraging off-the-shelf code llms to initialize the components of codet5+. specifically, we employ a "shallow encoder and deep decoder" architecture @xcite , where both encoder and decoder are initialized from pretrained checkpoints and connected by cross-attention layers. we freeze the deep decoder llm and only train the shallow encoder and cross-attention layers, largely reducing the number of trainable parameters for efficient tuning. finally, recent work in the nlp domain @xcite @xcite inspired us to explore codet5+ with instruction tuning to better align the models with natural language instructions. we extensively evaluate codet5+ on over 20 code-related benchmarks under various settings, including zero-shot, finetuning, and instructiontuning. results show that codet5+ yields substantial performance gains on many downstream tasks compared to their sota baselines, e.g., 8 text-to-code retrieval tasks (+3.2 avg. mrr), 2 line-level code completion tasks (+2.1 avg. exact match), and 2 retrieval-augmented code generation tasks (+5.8 avg. bleu-4). in 2 math programming tasks on mathqa and gsm8k benchmarks @xcite , codet5+ models of below billion-parameter sizes significantly outperform many llms of up to 137b parameters. particularly, in the zero-shot text-to-code generation task on humaneval benchmark @xcite , our instruction-tuned codet5+ 16b sets new sota results of 35.0% pass@1 and 54.5% pass@10 against other open code llms, even surpassing the closed-source openai code-cushman-001 model. finally, we showcase that codet5+ can be seamlessly adopted as a semi-parametric retrieval-augmented generation system. ## related work following the success of llms such as bert @xcite and gpt @xcite in natural language processing (nlp), recent years witness a surge of research work of llms in the code domain, leading to new sota results on various code-related tasks. typically, code llms can be categorized into three architectures: encoderonly models @xcite , decoder-only models @xcite , and encoder-decoder models @xcite . for encoder-only and decoderonly models, they are often ideal for either understanding tasks such as code retrieval @xcite or generation tasks such as code synthesis (chen et al., 2021) respectively. for encoderdecoder models, they can be adapted to both code understanding and generation but do not always achieve better performance @xcite . in this work, we propose a new family of encoderdecoder code llms "codet5+" that can flexibly operate in various modes, including encoder-only, decoder-only, and encoder-decoder models. prior code llms are also limited by their pretraining tasks, which are not perfect to transfer the models to some downstream tasks. for instance, t5-based models such as @xcite pretrained with span denoising objective are not ideal for auto-regressive generation tasks like next-line code completion @xcite , as these models are trained to recover short spans of limited lengths rather than a whole program. 1 inspired by recent advances in nlp research @xcite , we explore to combine span denoising with clm tasks to improve the model with better causal generation capability @xcite . additionally, most models do not have specific pretraining tasks (e.g. contrastive learning) to facilitate the learning of contextual representations that can distinguish code samples of different semantics. this can lead to suboptimal performance on code understanding tasks like code retrieval @xcite . in light of this, we include a contrastive learning task to learn better unimodal representations and a matching task to learn richer bimodal representations, which has been shown helpful in vision-language retrieval tasks @xcite . ## codet5+: open code llms we develop codet5+, a new family of open code llms for code understanding and generation tasks (see fig. 1 for an overview and more architecture/pretraining details in fig. 2 and fig. 3 ). based on the encoder-decoder architecture @xcite , codet5+ is enhanced with the flexibility to operate in various modes for different downstream tasks through our proposed mixture of pretraining objectives, which are performed on two stages of pretraining on unimodal (sec. 3.1) and bimodal data (sec. 3.2). we found that this stage-wise training approach can efficiently expose our models to more diverse data to learn rich contextual representations. finally, we explore initializing codet5+ with off-the-shelf code llms to efficiently scale up the model without training from scratch (sec. 3.3). such code completion setting is not the focus of this work. codet5+ def getpairscount (arr, n, sum): # count pairs with sum equal to 'sum' @xmath0, n): for j in range(i + 1, n): if arr[i] + arr[j] == sum: count += 1 return count [mask0] pairscount [mask1] with sum equal [mask2] range(0 [mask3] + arr[j] [clm] def getpairscount (arr, n, sum): # count pairs with sum equal to 'sum' @xmath1, n): def get[mask0](arr, n, sum): # count pairs [mask1] to 'sum' @xmath2, n): for j in range(i + 1, n): if arr[i] [mask3]== sum: count += 1 return count for j in range(i + 1, n): if arr[i] + arr[j] == sum: count += 1 return count [clm] figure 3 : self-supervised pretraining on code data: we pretrain codet5+ on code data using a mixture of tasks: (i) span denoising (top); (ii) decoder-only causal lm (middle); and (iii) seq2seq causal lm (bottom). this mixture of tasks lets the models learn meaningful representations of code contexts and recover missing information at different levels: code spans, partial programs, and complete programs. ## pretraining and instruction tuning additional pretraining and finetuning setups can be found in appendix b, c, and e. pretraining dataset. we enlarge the pretraining dataset of codesearchnet @xcite with the recently released github code dataset foot_1 . we select nine pls (python, java, ruby, javascript, go, php, c, c++, c#) and filter the dataset by preserving only permissively licensed code foot_2 and files with 50 to 2000 tokens. besides, we filter out the overlapped subset with codesearchnet and other downstream tasks covered in our evaluation by checking their github repository names. note that although we employ the deduplicated data version in which duplicates are filtered out based on the exact match, there might be some potential remaining duplicates. however, we do not expect any remaining duplication will impact our model performance significantly. we use the codet5 tokenizer to tokenize the multilingual dataset, resulting in 51.5b tokens, ∼50x larger than codesearchnet. pretraining setup. we pretrained two groups of codet5+ models: 1) codet5+ 220m and 770m that are trained from scratch following t5's architecture @xcite ) (t5-base and large respectively), 2) codet5+ 2b, 6b, 16b in which the decoders are initialized from codegen-mono 2b, 6b, 16b models @xcite and its encoders are initialized from codegen-mono 350m. note that following our model scaling strategy, the latter group of codet5+ models introduce insignificant trainable parameters (the 350m encoder plus one cross-attention layer of 36m, 67m, 151m for 2b, 6b, 16b models respectively) compared to the original codegen models. we employ the codet5 tokenizer and codegen tokenizer for these two groups of models respectively. in pretraining, we adopt a stage-wise strategy to pretrain codet5+ first on the large-scale unimodal dataset and then on the smaller bimodal dataset on a cluster with 16 a100-40g gpus on google cloud platform. instruction tuning. in the nlp domain, recent work @xcite studied the benefits of data augmentation techniques on pretrained lms with synthetic instruction data. models finetuned with this type of data can better understand natural language instructions and demonstrate improved alignment with the corresponding tasks @xcite . we are motivated to transfer this technique to the code domain to improve our codet5+ models. following @xcite , we employ over 20k instruction data in the code domain curated by @xcite . the data is generated by letting pretrained llms i.e. text-davinci-003, generate novel tasks, including task instructions, inputs (if any), and expected outputs. we trained our models on this augmented dataset for up to 3 epochs and denote the instruction-tuned models as "instructcodet5+". note that the instruction data are generated fully independently from any downstream evaluation tasks and we still evaluate these models in a zero-shot manner. ## experiments we extensively evaluate codet5+ on a wide range of code understanding and generation tasks over 20+ code-related datasets across 9 different programming languages (pls). in addition, we consider a variety of evaluation settings including zeroshot, instruction tuning, task-specific finetuning. additional results can be found in appendix d. baselines. we developed a family of codet5+ models, with model sizes ranging from 220m to 16b. we compare codet5+ with 3 types of models: encoder-only, decoder-only, and encoder-decoder. note that billion-parameter llms such as codex and codegen typically use most of the source code from github for model training and do not remove any overlap with the downstream tasks covered in this work as we did. therefore, it is difficult to ensure a fair comparison with these models in those tasks, especially the code completion tasks. moreover, these models are very expensive to perform task-specific finetuning, and hence, they are often employed only on the zero-shot evaluation. in this work, we mainly compare codet5+ with these llms in the zero-shot humaneval code generation task (sec. 5.1). in other experiments, we focus on the finetuning setting and compare our models with smaller-scale lms. ## conclusion we propose codet5+, a new family of open code llms with a dynamic architecture that can flexibly operate in different modes (encoder-only, decoderonly, and encoder-decoder) to support a wide range of code understanding and generation tasks. to train codet5+, we introduce a mixture of pretraining tasks to learn rich representations from both unimodal code data and bimodal code-text data. additionally, it achieves efficient model scaling and better task generalization through integration with frozen llms and instruction tuning. extensive experiments on over 20 code intelligence benchmarks have verified the superiority of our models.