ERNIE -Code: Beyond E nglish-Centric Cross-lingual Pretraining for Programming Languages (2023)

Abstract: Software engineers working with the same programming language (PL) may speak different natural languages (NLs) and vice versa, erecting huge barriers to communication and working efficiency. Recent studies have demonstrated the effectiveness of generative pre-training in computer programs, yet they are always English-centric. In this work, we step towards bridging the gap between multilingual NLs and multilingual PLs for large language models (LLMs). We release ERNIE-Code, a unified pre-trained language model for 116 NLs and 6 PLs. We employ two methods for universal cross-lingual pre-training: span-corruption language modeling that learns patterns from monolingual NL or PL; and pivot-based translation language modeling that relies on parallel data of many NLs and PLs. Extensive results show that ERNIE-Code outperforms previous multilingual LLMs for PL or NL across a wide range of end tasks of code intelligence, including multilingual code-to-text, text-to-code, code-to-code, and text-to-text generation. We further show its advantage of zero-shot prompting on multilingual code summarization and text-to-text translation. We release our code and pre-trained checkpoints.

## introduction recent trends in generative pre-training of programming languages @xcite @xcite have led to a proliferation of improvements in code intelligence scenarios, including program understanding and generation @xcite . in this context, a transformer-based large language model (llm) is pre-trained on a large corpus of open source code (e.g., from github) and then finetuned or zero-shotly evaluated on downstream tasks, such as program synthesis @xcite ., 1 https://github.com/paddlepaddle/paddlenlp/tree/ develop/model_zoo/ernie-code english deutsch français español русский ‫ع‬ ‫ر‬ ‫ب‬ ‫ي‬ english php python javascript go java ruby 2021; @xcite , code search @xcite , clone detection @xcite , and text-to-code generation @xcite . although there has been a surge of interest in learning general-purpose multilingual llms for source code @xcite @xcite @xcite , research in this area has been essentially connecting english texts (e.g., comments or docstring) and multiple computer programs (e.g., python, c++, and java), as shown in figure 1 (a), and primarily focused around english-centric corpora and benchmarks. this english-centricity issue dramatically limits their use and practice given that 95% of the world population does not have english as their native language @xcite . as such, it is crucial to mitigate barriers and draw connections between non-english natural languages (nls) and multiple programming languages (pls). one engineering solution is to use english translation of non-english texts by engaging neural machine translation (nmt) systems before/after the code llm as a pipeline. unfortunately, most general-purpose nmt systems @xcite are not designed for codespecific texts and can be prone to accumulative errors due to cascaded prediction stages. a more general way is to learn a multilingual llm that encodes a mixture of multiple nls and pls into a shared cross-mode representation space. the success in learning universal representations of many languages @xcite @xcite @xcite that focuses on pls or nls suggests that it is possible to build a universal multilingual model that jointly represent multiple pls and nls. in this work, we present ernie-code, a unified cross-lingual pre-trained llm for multiple nls and pls in hopes of mitigating the english-centric bias for program pre-training, as illustrated in figure 1. our model builds on the t5 @xcite encoder-decoder architecture that has been demonstrated to be effective in understanding and generation tasks for multilingual nl @xcite and pl @xcite . for monolingual pre-training on mono-mode data (i.e., unpaired multilingual code or text), we follow the same t5 recipe to employ the "span-corruption" denoising objective in the text-to-text format. the good-quality parallel corpus between lowresource nls and multilingual pls is usually unavailable. instead, most popular pls, accompanying api documentation, code examples, and discussion forums are primarily written in english, which poses a bottleneck to drawing connections between low-resource nls and pls. inspired by the pivotbased machine translation @xcite ) that uses a pivot language and decomposes the source↔target translation into source↔pivot and pivot↔target bilingual translation, we introduce the pivot-based translation language modeling (ptlm) with prompting that disassembles multi-nl↔multi-pl into multi-nl↔english and english↔multi-pl with pivoting through english. specifically, we leverage the ptlm training in dual direction for parallel corpus in different modes: to summarize, this paper makes the following contributions: (1) we first propose a unified cross-lingual pre-trained llm for both multilingual nls and multilingual pls, enlarging the capacity of llms towards jointly learning the universal multilingualism. (2) we employ the pivotbased translation language modeling with prompting to build connections between multi-nls and multi-pls (with english pivots) and mitigate the problem when the parallel corpus of multilingual-nl↔multilingual-pl is unavailable. (3) we obtain superior performance compared with previous multilingual llms across a wide range of code intelligence tasks, including text-to-code, code-to-text, code repair, and code documentation translation. (4) to some extent, our model has shown zeroshot prompting ability on multilingual code-to-text, text-to-code, and text-to-text generation. moreover, ernie-code is well-performed at naming a function and completing corresponding arguments given multilingual nl instructions. ## related work as text-based formal languages with strict syntax and semantics, pl differs from nl because nl is only used for human communication while pl requires the interaction between humans and computers. this work targets bridging the gap between human languages and computer programs in a cross-lingual manner for unified multilingual pre-training, which is closely related to llms in either multilingual pl or nl. multilingual pl pre-training the success of large-scale pre-training has led to impressive advances in computer programs. this line of research involves pre-training on multilingual pls using bidirectional transformer encoders @xcite , casual transformer decoders @xcite @xcite @xcite , and transformer encoder-decoder architectures @xcite @xcite . those with bidirectional encoder focus on program understanding tasks, such as code search @xcite , while the encoder-decoder ones target at building unified llms for both program understanding and generation. we observe that a large body of pre-trained models for pl tend to scale up their parameters under the framework of causal language modeling, mainly focusing on program synthesis @xcite @xcite @xcite . nevertheless, all of these works are almost english-centric, posing significant challenges to coping with pl end-tasks in non-english scenarios. multilingual nl pre-training this work is also related to the continual trend of multilingual llms. one line of this work focuses on encoding multiple nls into a shared representation space @xcite , while some make efforts to extend the efficient monolingual pre-training method into multilingual settings @xcite . inheriting the recent success of llms in multilingualism, this work lies in the intersection between multilingual nl and pl pre-training. in contrast to the previous work that attends to either multilingual nl or multilingual pl, we seek to explicitly learn multiple nls and pls in a shared representation space in hopes of breaking the language barriers between these two modes. ## cross-lingual nl-pl pre-training in this section, we introduce pre-training tasks ( §3.1), model ( §3.2), and pre-training data ( §3.3) we use throughout this work. ## experiments in this section, we first introduce multilingual pretrained models for comparison ( §4.1), downstream tasks, and evaluation metrics ( §4.2). then we evaluate and show consistent performance gains on several multilingual nl/pl benchmarks, including code-to-text ( §4.3), text-to-code ( §4.4), text-to-text ( §4.5), and code-to-code ( §4.6) end tasks. ## conclusion this work makes the first step towards explicitly connecting computer programs to human languages in a universal multilingual fashion. by virtue of cross-lingual pre-training on 116 nls and 6 pls, our model exhibits strong performance in various tasks across computer programs and natural languages, including pl-to-nl, nl-to-pl, nl-to-nl, and pl-to-pl. our model shows descent zero-shot performance via prompting on pl summarization and nl translation. finally, we provide discussions about limitations and future work for improvement.