Constructing Multilingual Code Search Dataset Using Neural Machine Translation (2023)

Abstract: Code search is a task to find programming codes that semantically match the given natural language queries. Even though some of the existing datasets for this task are multilingual on the programming language side, their query data are only in English. In this research, we create a multilingual code search dataset in four natural and four programming languages using a neural machine translation model. Using our dataset, we pre-train and fine-tune the Transformer-based models and then evaluate them on multiple code search test sets. Our results show that the model pre-trained with all natural and programming language data has performed best in most cases. By applying back-translation data filtering to our dataset, we demonstrate that the translation quality affects the model’s performance to a certain extent, but the data size matters more.

## introduction code search is the task of finding a semantically corresponding programming language code given a natural language query by calculating their similarity. with the spread of large-scale code-sharing repositories and the rise of advanced search engines, high-performance code search is an important technology to assist software developers. since software developers worldwide search for codes in their native language, we expect code search models to be multilingual. although many previous studies focus on multilingual code tasks other than code search (e.g., code generation, code explanation) @xcite @xcite , the existing code search datasets @xcite @xcite contain only monolingual data for search queries. in this research, we construct a new multilingual code search dataset by translating natural language data of the existing large-scale dataset using a neural machine translation model. we also use our dataset to pre-train and fine-tune the transformer @xcite -based model and evaluate it on multilingual code search test sets we create. we show that the model pretrained with all natural and programming language data performs best under almost all settings. we also analyze the relationship between the dataset's translation quality and the model's performance by filtering the fine-tuning dataset using backtranslation. our model and dataset will be publicly available at https://github.com/ynklab/ xcodesearchnet . the contributions of this research are as follows: ## dataset construction using machine translation a possible way to construct a code search dataset for multiple languages is to translate an existing monolingual dataset. however, csn's large data size makes manually translating all of its docstrings difficult. in their fine-tuning data, the numbers of positive and negative labels are balanced. note that we do not use javascript and ruby data, whose sizes are much smaller than those of other programming languages. as a translation model, we use m2m-100 @xcite , which supports translations in 100 languages. foot_1 m2m-100 achieved high accuracy in translations of low-resource languages by classifying 100 languages into 14 word families and creating bilingual training data within those families. we use m2m_100_1.2b model, which is provided by easynmt foot_2 , a public framework of machine translation models. we set the model's beam size to 3. we manually annotate the labels to some data of our fine-tuning dataset to check the correlation with the original labels, which is found to be 0.911 (see appendix b for the details). ## baseline experiments we conduct baseline experiments, where we train the transformer-based model with our multilingual dataset under various settings of the data sizes and evaluate it on multiple code search test sets. ## conclusion we created a large multilingual code search dataset by a neural machine translation model. we then constructed a multilingual code search model using our dataset. we found out that the models pre-trained with all of the multilingual natural language and programming language data achieved the best performance on a code search task almost all the time. we also investigated the relationship between the translation quality of our dataset and the model's performance. the results indicated that the data size contributed more to the model's code search performance than the data translation quality. overall, this research introduced that using a publicly available machine translation model helps to translate texts in the programming domain. we can apply our method to extend datasets for languages other than french, japanese, and chinese to construct models for various natural languages.