The Vault: A Comprehensive Multilingual Dataset for Advancing Code Understanding and Generation (2023)

Abstract: We present The Vault, an open-source dataset of high quality code-text pairs in multiple programming languages for training large language models to understand and generate code. We propose methods for thoroughly extracting samples that use both rules and deep learning to ensure that they contain high-quality pairs of code and text, resulting in a dataset of 43 million high-quality code-text pairs. We thoroughly evaluated this dataset and discovered that when used to train common code language models (such as CodeT5, CodeBERT, and CodeGen), it outperforms the same models train on other datasets such as CodeSearchNet. These evaluations included common coding tasks such as code generation, code summarization, and code search. The Vault can be used by researchers and practitioners to train a wide range of big language models that understand code. Alternatively, researchers can use our data cleaning methods and scripts to improve their own datasets. We anticipate that using The Vault to train large language models will improve their ability to understand and generate code, propelling AI research and software development forward. We are releasing our source code and a framework to make it easier for others to replicate our results.

## introduction the advent of deep learning and advancements in large language models (llms) have spurred a revolution in the field of code representation learning. these developments, supported by the growing accessibility of vast open-source code repositories, have heralded the emergence of code large language models (codellms) for code generation and understanding tasks. the sheer volume of these repositories and the rich, unprocessed raw data they contain, serve as unparalleled resources for training llms. consequently, current state-ofthe-art models for coding tasks effectively utilize * equal contribution these expansive datasets for training. however, it is important to note that these datasets, including the stack @xcite and the pile @xcite , often comprise unprocessed data. alternatively, there are established datasets, such as concode @xcite , funcom @xcite , deepcom @xcite for code summarization tasks; apps @xcite for text-to-code generation; and code-searchnet @xcite for code search. these datasets contain carefully curated code-text pairs. although considerably smaller in comparison to raw code datasets (e.g., 2.3m functions in codesearchnet @xcite versus 197m files in the stack @xcite ), they provide high-quality code-text pairings that significantly enhance the effectiveness of model training. consequently, we identify two main types of datasets used to train codellms: large yet unprocessed, and smaller yet well-structured (e.g., arranged into code-text pairs). the scaling law @xcite @xcite indicates that the volume of training data is crucial for model performance. however, other studies underscore the importance of dataset quality over quantity in training superior llms @xcite @xcite @xcite . given these observations, we propose that an ideal dataset for training codellms should combine both elements: it should be expansive in volume and meticulously processed to ensure quality. in this paper, we present the vault dataset, detailing its creation process, the toolkit developed for constructing and quality-controlling code-text pairs from raw source code, as well as an analysis of the vault's metrics. we also share empirical results obtained from utilizing the vault to finetune well-known foundational models. our specific contributions include the following: code large language models for understanding and generation code large language models facilitate various code understanding and code generation tasks, including but not limited to code generation @xcite @xcite @xcite , code completion @xcite @xcite , program repair @xcite , program classification @xcite and code translation @xcite . a significant portion of recent research employs language models, originally developed for natural language processing, for handling code @xcite @xcite @xcite @xcite @xcite . such approaches largely regard code as analogous to text and adapt pretraining strategies that mirror those used for natural languages. codebert @xcite , for instance, modifies a roberta model @xcite to pretrain a code model on multiple programming languages. codet5 @xcite and codet5+ @xcite employs unique identifier information from source code to pretrain the t5 model @xcite for code in a multi-modal fashion. ## empirical evaluation in this section, we aim to assess the quality of the vault in comparison with other datasets, such as csn. ## conclusion in this paper, we presented the vault, a large dataset of high-quality code-text pairs from ten programming languages, with over 43 million samples. the vault was carefully curated to ensure that each pair meets quality standards, with detailed and informative descriptions and consistent coding styles. our analysis uncovered a number of intriguing patterns and trends that shed light on the characteristics of programming languages and coding practices. we believe that the vault will be a valuable resource for researchers and practitioners in this rapidly evolving field, providing a solid foundation for developing novel approaches and advancing state-of-the-art code large language models.