MC o N a L a: A Benchmark for Code Generation from Multiple Natural Languages (2023)

Abstract: While there has been a recent burgeoning of applications at the intersection of natural and programming languages, such as code generation and code summarization, these applications are usually English-centric. This creates a barrier for program developers who are not proficient in English. To mitigate this gap in technology development across languages, we propose a multilingual dataset, MCoNaLa, to benchmark code generation from natural language commands extending beyond English. Modeled off of the methodology from the English Code/Natural Language Challenge (CoNaLa) dataset, we annotated a total of 896 NL-Code pairs in three languages: Spanish, Japanese, and Russian. We present a systematic evaluation on MCoNaLa by testing state-of-the-art code generation systems. Although the difficulties vary across three languages, all systems lag significantly behind their English counterparts, revealing the challenges in adapting code generation to new languages.

## introduction there are an increasing number of applications related to "code intelligence", such as code summarization @xcite @xcite and natural language (nl) to code generation @xcite @xcite @xcite , accompanied by code-specific tasks and benchmarks @xcite @xcite . however, in the cases where these benchmarks include natural language, that language is almost invariably english. there are a few exceptions, but most of them either focus on languages of specific domains @xcite @xcite figure 1 : examples in the mconala dataset, that aim to generate general-purpose python code snippets from source intent of multiple natural languages. @xcite or types of code @xcite , or contain nl intents collected via automatic translation @xcite (appendix a). however, similarly to how @xcite argue that "natural questions" are necessary to appropriately benchmark qa systems, we argue that ensuring the naturalness and coverage of questions is essential for benchmarking code generation systems as well. a dataset for english code generation based on natural programming questions is the conala dataset @xcite . it is based on natural developer questions harvested from the stack overflow (so) question answering forum. in fact, in addition to english, so also supports four other languages (spanish, portuguese, japanese, and russian) that have strong developer communities and engage in non-english programming environments. in this work, we utilize this resource to construct the mconala dataset, consisting of 341, 210, and 345 manually curated parallel samples with natural intents in spanish, japanese, and russian, along with corresponding python code snippets. like conala, these snippets are collected from language-specific so sites and annotated by na-tive speakers who are also proficient in the python programming language. to provide insights in the state of code generation on this new resource, we conduct comprehensive experiments with three state-of-the-art text generation models in the context of crosslingual transfer, by unifying training and testing nl via translation @xcite @xcite , or utilizing a multilingual nl encoder such as mbart @xcite . our results suggest that cross-lingual nl-to-code generation is challenging. among all languages and experiment settings, the highest average bleu score is 7.28, far behind that of english, which achieves 33.41, presumably because english resembles python more than other nls. in addition, we find models with task-specific modules and training outperform generic seq2seq models, yet the discrepancy between languages are consistent across all baseline models. in all, our corpus and experiments demonstrate the varied difficulty of the nl-to-code generation task under different languages, emphasizing the need to develop a language-comprehensive approach to code intelligence. ## method to provide insights about evaluating on mconala, we demonstrate potential dataset usage in three train-test settings ( § 3.1), and propose to adapt three baseline models from either multilingual (nl) or code understanding to achieve both ends ( § 3.2). because the size of mconala allows only testing purposes, we resort to its larger english counter-part, conala @xcite , to allow training. conala contains 2,879 manually annotated samples and 600k samples extracted from english so forum and api documents, which can serve as a sufficient source for training. given this usage, we denote the three test languages as target languages. ## conclusion in this work, we extend the task of nl-to-code generation from english-centric to multilingual scenarios. we establish the mconala benchmark that contains nl intent and code snippet pairs available in spanish, japanese, and russian. our benchmark serves for the multilingual code generation task, requiring models of both multilingual understanding and code synthesis. we conduct systematic experiments on three baseline models and show varying difficulty across languages and settings. we hope to reveal the necessity to develop, and serve as a solid test bed for language-comprehensive approaches regarding code intelligence.