M ULTI - LINGUAL E VALUATION OF C ODE G ENERATION M ODELS
AWS AI Labs
A BSTRACT
We present two new benchmarks,
MBXP and Multilingual HumanEval, designed to evaluate code generation models in over 10 programming languages.
These datasets are generated using a conversion framework that transpiles prompts and test cases from the original MBPP and HumanEval datasets into the corresponding data in the target language.
By using these benchmarks, we are able to assess the performance of code generation models in a multi-lingual fashion and discovered generalization ability of language models on out-of-domain languages, advantages of multi-lingual models over mono-lingual, the ability of few-shot prompting to teach the model new languages and zero-shot translation abilities.
In addition, we use our code generation model to perform large-scale bootstrapping to obtain synthetic canonical solutions in several languages, which can be used for other coderelated evaluations such as code insertion, robustness or summarization tasks
I NTRODUCTION
Code completion by machine-learning models has great potential to improve developer productivity (Barke et al., 2022).
This line of research has seen tremendous progress with several models recently proposed such as Codex (Chen et al., 2021), CodeGen (Nijkamp et al., 2022), PaLM (Chowdhery et al., 2022), BLOOM (Mitchell et al., 2022), and InCoder (Fried et al., 2022).
One key component for code generation research is how to evaluate such program synthesis abilities.
In the literature, two primary evaluation approaches emerged, namely, the match-based and the execution-based evaluations.
For both approaches, each problem contains a prompt which a model uses as input to generate a candidate body of code.
The match-based evaluation compares the candidate code against reference source code using n-gram metrics such as BLEU, whereas the execution-based evaluation executes the candidate code against test cases and calculates success rate.
The execution-based evaluation has benefits over the n-gram evaluation in that it permits solutions that are functionally correct but might not be equivalent to the reference solution in terms of the exact implementation.
Since the release of datasets such as HumanEval (Chen et al., 2021) or
MBPP (Austin et al., 2021), the community has been widely adopting the execution-based approach as a primary tool to evaluate program generation capabilities.
However, creating execution-based evaluation datasets is time-consuming since it requires careful construction of test cases to check the correctness of the code’s functionality.
Such difficulty leads to limited available of executionbased evaluation data.
For instance, to date, many execution-based datasets contain only problems in Python.
[Figure]
d i s ca r d a n d r e g e n e r a t e source prompt source solution (E) Other Evaluation
[Figure]
prompt (E-1) translation (E-2) code insertion prompt with extracted summary (E-3) summarization
[Figure]
prompt left context right context
[Figure]
solution test
[Figure]
test
[Figure]
In this work, we propose a scalable framework for dataset convertion from Python to many different languages.
While translating code from one language to another is typically a non-trivial task, it is possible to convert existing execution-based datasets to another language by transforming only prompts and test statements (see Figure 1 part A and Figure 2).
That is, the purpose of evaluating function completion ability, we do not need the canonical solution since it is not used during evaluation.
The function signature prompts and test cases of basic programming problems involve sufficiently simple data structures that can be analyzed to synthesize dataset in new languages.
Without having to translate the generic function body of code to another language, the conversion process becomes possible via a rule-based transpiler.
The result of such conversion are two benchmarks, MBXP ‡ and Multilingual HumanEval, which are derived from the original Python dataset MBPP (Austin et al., 2021) and HumanEval
(Chen et al., 2021).
We provide the evaluation data in many languages besides the original Python, namely, Java,
JavaScript, TypeScript, Go, Ruby, Kotlin, PHP, C#, Scala, C++, Swift, and Perl, with plans for more language expansion in the future.
Along with these datasets, we also release a code package to perform execution in all supported languages.
In addition, our conversion framework is easily extensible and allows us to obtain the multi-lingual version of other existing datasets such as MathQA (Schubotz et al., 2019).
In the main paper, we provide results and analyses mostly on MBXP where the results on Multilingual HumanEval and MathQA can also be found in Appendix D.
Our benchmarks also support other code completion tasks such as code insertion or translation in many languages.
This extension is made possible by performing large-scale bootstrapping to synthetize solutions (Section O.1.11).
The result of our dataset conversion framework and the solution synthesis process is, to date, the first multi-lingual execution-based evaluation benchmark equipped with canonical solutions, which can be adapted for many code-related evaluations.
In this paper, we process MBXP for multiple use cases, namely, for zero-shot translation t-MBXP, prompt robustness r-MBXP, code insertion i-MBXP, and the summarization s-MBXP.
Overall, the constructed datasets provides us new opportunities to explore many facets of code generation abilities.
In this work, we conduct a large scale evaluation where we train models of various sizes spanning three orders of magnitude (from ∼ 100M to ∼ 10B parameters) in both multi-lingual and mono-lingual settings.
We analyze results from hundreds of thousands of code generation samples to investigate the models’ code generation abilities with respect to in-domain versus out-ofdomain languages, the effectiveness of few-shot prompting, zero-shot translation abilities, robustness to prompt perturbation, code summarization, and code insertion.
F INDING H IGHLIGHTS
We provide the highlights of out findings below.
Given the same model size, a multi-lingual model often outperforms the best of monolingual models trained with equivalent training resources, especially when the models are sufficiently large.
This observation indicates that it is beneficial to train a single model on all programming languages, and provided that the model size has enough capacity, the performance will be better than the best of monolingual models.
Language models are able to generate code with correct syntax and pass unit tests in programming languages they are not intentionally trained on.
We hypothesize that the data “spillover” effect, where code in one language is present in
other languages through code def binomial_coeff(n,k): """ Write a Python function to find binomial coefficient. >>> binomial_coeff(5,2)""" if k > n : returnif k==0 or k ==n : returnreturn binomial_coeff(n-1,k-1) + binomial_coeff(n-1,k) assert binomial_coeff(5,2) =
= 10 assert binomial_coeff(4,3)
=
=assert binomial_coeff(3,2)
==
[Figure]
/** Write
[Figure]
to find
[Figure] [Figure]
[Figure] [Figure]
, int
[Figure] [Figure]
[Figure]
[] args)
[Figure]
{ int x0 = BinomialCoeff.binomialCoeff(5, 2); if (!x0.equals(10)) { throw new java.lang.
Exception ("Exception -- test case 0 did not pass.
x0 = " + x0); if (n < k) return 0; int res = 1; for ( int i = 1; i <=
k; i++) { res *= (n + 1 - i); res /= i; } return res;
M
[Figure]
l F un c t i o n s i g n a t u r e a n d d o c s t r i n g
F un c t i o n b o d y
T e s t ca s e s comments or co-occurrences.
Such amount of spillover data are enough for large language models to learn different languages that are embedded within the main language.
The occurrences of multi-lingual data in natural data also explains the superior performance of multi-lingual over mono-lingual models.
That is, the
multi-lingual model can perform better on language A since it can pick up and combine all knowledge of language A from the training data in languages A , B , C , etc.
in the multi-lingual setting.
Few-shot prompting can effectively help teach provide knowledge on a new language the model has not seen, significantly improving out-of-domain code generation abilities.
Through error analysis, few-shot prompting helps reduce compilation or parsing errors that are the major sources of errors when it comes to a programming language the model is not familiar with. 5.
Language models have zero-shot code translation abilities; that is, even though they are not specifically trained to perform translation, they are able to use reference code in one language to improve code generation in another language.
Problems that are difficult can become much easier with access to another language’s solution.
This observation holds for mono-lingual as well as multi-lingual models.
Multi-lingual models are also more robust to prompt perturbation and better at summarizing code.
C ONVERSION OF E XECUTION -B ASED E VALUATION D ATASETS
In this section, we provide high-level details on the data conversion process.
Figure 2 illustrates the mapping of the original Python prompt, consisting of a function signature and a docstring, to an equivalent prompt in Java (which we call a target prompt).
The target prompt is a valid code including a function signature from which the model can use to complete the function body.
In the case of Java or typed languages, constructing the target prompt requires inferring input and output types.
We perform such type inference by parsing the original test cases, taking into account heterogeneous data types.
For instance, if the first argument includes values of types int and float , we deduce it to have the most general type of all types encountered.
The converted prompt also needs to work in harmony with the converted test cases.
For instance, the Java test case in
Figure 2 refers to the defined class BinomialCoeff and the defined method binomialCoeff in the converted prompt with appropriate function call based on the defined argument list.
For more details including data validation and generated solutions via bootstrapping, see Appendix O.
M ULTI - LINGUAL E VALUATION OF C ODE G ENERATION M ODELS
From the previous section, we have established a framework to perform dataset conversion, from which we obtain a collection of execution-based evaluation datasets in 10+ programming languages.
These evaluation datasets contain rich information in the prompts including natural language description as well as appropriate function signatures that help steer a model to generate code in a particular language.
Most importantly, they also contain test cases in the respective language that can be used to check code correctness, which is applicable for most types of evaluation in MBXP+.
This section describes the training, evaluation setup, and findings from each evaluation task.
4. D ATA AND M ODELS
For the purpose of this work, we collected training data in three primary programming languages, namely, Python, Java, and JavaScript, containing permissively licensed code data from GitHub.
Following Chen et al.
; Nijkamp et al.
, we perform filtering, deduplication, and remove data that contains a significant amount of non-English text or is not parsable with respect to that language’s syntax parser.
We also ensure the original MBPP and HumanEval datasets are not included in data.
After all the post processing steps, our dataset contains 101 GB Python, 177 GB Java, and 216 GB JavaScript data.
We use a decoder-only transformers as the model architecture and train the models via next-token prediction loss (Vaswani et al., 2017; Brown et al., 2020).
We design our training to compare multilingual versus mono-lingual settings by using the same compute budget for each language in both cases.
In particular, we train mono-lingual models on 210 billion tokens with their respective languages (Python, Java, and JavaScript) and train multi-lingual models on 210 billion tokens from each language, with 630 billion tokens in total.
To study effects of model sizes, we train models of various number of parameters, namely, 125M, 672M, 2.7B and 13B. For the synthetic canonical solution process, we use a separate 13B multi-lingual model which we refer to as the 13B ∗ model.
4. E XECUTION -B ASED F UNCTION C OMPLETION
We use pass @
k scores (Kulal et al., 2019) with the unbiased estimate presented in (Chen et al., 2021) as the metrics for our evaluation, where each task is considered successful if any of the k samples are correct.
We generate up until the end of the function, such as end of indented function block for Python or until the closing curly brace for PHP or Go, for example (see Appendix C.2 for end of scope details).
We refer to an evaluation language that the model is not specifically trained on as out-of-domain with respect to that model.
Otherwise, the language is considered in-domain .
For instance, Java is out-of-domain for a Python-only model and PHP is out-of-domain for our multilingual model trained on Python, Java, and JavaScript.
4.2. A CCURACY VS . S AMPLING B UDGET
Overall, we observe sigmoid-like relationships between pass @
k and sampling budget k across all datasets in MBXP where the performance increases smoothly as k increases (Figure 3, and Appendix
F.2).
This trend is consistent with the original MBPP and HumanEval which are manually-annotated.
This sigmoid-like performance with respect to sampling budget indicates that problems vary in terms of difficulty, where certain problems require many more attempts to get them right.
We do not find a degenerate case in any evaluation language where all problems are either trivial to solve (pass @
k saturated near 100%), or impossible (pass @ k all zeros).
The consistency of the observed performance trend across all programming languages in the MBXP benchmark provides reassurance regarding the benchmark’s applicability as a multi-lingual evaluation tool for assessing a model’s capabilities at different levels.
4.2. G ENERALIZATION TO OUT - OF - DOMAIN LANGUAGES
As demonstrated in Figure 3, our model can achieve non-zero pass @
k scores for out-of-domain languages.
We emphasize that our models are not specifically trained on out-of-domain languages since we filter languages based on file extensions and verify that the data have correct syntax with respect to each language (refer to Section 4.1).
However, we hypothesize that cross-language data
[Figure]
the out-of-domain case, the curve breaks out upward, similar to the earlier part of the sigmoid function.
The behavior also applies for models of other sizes as well as mono-lingual models (not shown in this figure).
spillover are quite typical, since there can be data related to other languages mentioned in code comments, natural texts, or intentionally embedded in cross-lingual code projects.
Examples of such projects are Django or Flask, where JavaScript pieces of code can be embedded in Python files for web development, or mixed use of Java and Python code in projects such as Jython.
We provide further discussion of types and examples of cross-lingual data occurrences in Appendix E.
In Figure 4a, we also observe that the out-of-domain scores are not symmetric for a given language pair; i.e., Python models perform well on Java but Java models have negligible performance on
Python.
The data spillover hypothesis supports this observation where it is likely that there are many languages embedded in, e.g. Python files, whereas not as many languages are embedded in
Java files.
We provide further analyses related to data spillover hypothesis in Section 4.2.3.
4.2. M ULTI - LINGUAL VERSUS MONO - LINGUAL MODELS
Figure 4a shows a plot of pass @
k scores versus model sizes for multi- and mono-lingual models, where we observe approximate log-linear relationships similar to those found in the literature (Chowdhery et al., 2022; Chen et al., 2021; Nijkamp et al., 2022; Austin et al., 2021;
Li et al., 2022a).
For small model sizes, we see that multi-lingual models can perform slightly sub-par or on-par to mono-lingual models.
For instance, at size 125M and 672M, mono-lingual models outperform multi-lingual models in some evaluation languages such as Python and Ruby.
However, once we reach a certain size such as 2.7B or 13B parameters, a large multi-lingual model begins to outperform the best of mono-lingual models in all evaluation languages.
The performance gains of multi-lingual over mono-lingual models are particularly significant for out-of-domain languages such as MBPHP and also noticeable for in-domain ones such as MBJSP and MBJP.
We observe that for MBPP, the mono-lingual Java and JavaScript models obtain close to zero pass @ k, suggesting that the amount of spillover Python code in Java or JavaScript training data is likely low.
This finding coincides with the Python and multi-lingual models achieving near identical
MBPP scores in Figure 4a, suggesting that both Python and multi-lingual models observed similar amount of Python code during training.
This evidence is consistent with the previous observation that there is little Python code in Java or JavaScript training data.
In contrast, for the JavaScript evaluation (MBJSP) shown in Figure 4a, each of the mono-lingual models obtain reasonable pass @
k scores, suggesting that the spillover of JavaScript code is prevalent (at least in Python and Java data).
This finding also explains why the multi-lingual model performs significantly better to the JS model on JS evaluation (MBJSP), as the multi-lingual model learn JS knowledge from other sources, while the mono-lingual JS model’s source of knowledge is more limited.
[Figure]
[Figure]
evaluation datasets in MBXP.
On languages such as PHP, Ruby, Kotlin which are outside of the core training data (Python, Java,
JS), multi-lingual models are also more capable of learning such languages, as demonstrated in
Figure 4a.
Overall, the performance in the multi-lingual setting tends to improve more rapidly as they are able to draw knowledge from many sources at once, as observed by higher slopes in the plots (Figure 4a).
Interestingly, we note that even though the multi-lingual models perform better during evaluation, the validation losses per language for multi-lingual models are higher than those of mono-lingual models (See Figure 4b).
We provide further discussion on validation losses in Appendix N.2.
4. Z ERO -S HOT C ODE T RANSLATION
Our dataset conversion framework yields parallel data in many different languages.
These parallel datasets provide a valuable resource for studying the translation abilities of code generation models, as we can evaluate how well the models generate code in any other supported language using the canonical solutions in our source language.
For this study, we prepend the function in a source language to the beginning of the function-completion prompt of the target language (Figure 5).
We can also think of this setup as a function completion with augmented information, where we provide a reference solution in another language.
Therefore, we also refer to the usual function completion setup as the non-translation setting.
Zero-shot translation abilities Figure 6a showcases the ability of language models to perform translation by using reference solutions in a different language to aid in function completion.
Examples in Figures 5 and 8 illustrate how the models are able to produce code that retains the same underlying logic as the reference solution in Python, while conforming to the syntax of the target language, such as PHP (e.g., using $ before variable names)
Specifically, the generated code in the translation mode mimics the content of the reference solution, including the same loop and control flow structures, as shown in the upper part of Figure 8.
Additionally, the generated code cab exhibit similar semantics, such as sorting by the summation, as illustrated in the lower part of Figure 8.
Interestingly, our analysis in Figure 7c suggests that the translation setting can enable the models to solve problems that are otherwise difficult without the aid of reference solutions.
For instance, on the MathQA dataset, which requires complex reasoning but has solutions with simple arithmetic syntax, our models are able to translate to a different language with near-perfect accuracy, achieving almost 100% pass @ 100 scores (see Appendix D.1).
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table] [Table]
[Figure]
Unequal effects of different source languages We find that different source languages can interact quite differently with each target language.
For instance, JavaScript yields better translation performance as a source language compared to Python, when evaluated on datasets such as Kotlin (MBKP) or Ruby (MBRBP).
Languages that are too close in terms of syntax can confuse the model when used together in the translation setting.
For instance, Python as a source language for translation to Ruby can sometimes lead the model to generate code in Python, which is undesirable.
For each evaluation language, the best source language is fairly consistent, relatively consistent across models.
We discuss the language compatibility with respect to translation further in Appendix H.1.
4. F EW -S HOT P ROMPTING
Few-shot prompting is a technique that can provide additional information to a language model, which can help it perform better on specific tasks (Brown et al., 2020).
In our experiment, we construct few-shot prompts consisting of three correct functions from the respective MBXP dataset (see Appendix G.2 for prompt format).
We observe consistent improvement in execution accuracies, especially for out-of-domain evaluations, as shown in Figure 6b.
One possible explanation for this improvement is that the few-shot prompt can help disambiguate programming languages, which is most beneficial in out-of-domain evaluations when the models
[Figure] [Figure]
in translation mode if (2*l <= r): return (l,2*l) else: return (-1) reference solution var sumByRow: (row : Int) ->
Int = (r : Int) -> m.get(r).reduce { it + it } var sumByCol: (col : Int) ->
Int = (c : Int) -> m.mapIndexed { it.get(c) } return m.sortedBy { it.sum() } result =
sorted(M, key=sum) return result
[Figure]
Kotlin Kotlin
[Figure]
are not familiar with the target language.
For example, in MBRBP evaluation (Ruby), the Ruby function signature can be very similar to that of Python, which can lead to confusion and the model generating Python code without the few-shot prompt.
The error analysis in Figure 7a demonstrates that compilation, syntax, or parsing errors (non-assertion errors) drop significantly due to the fewshot prompts.
The improvement due to few-shot prompts also applies to other datasets such as MathQA (Appendix
D.1).
These observations suggest that soft prompts obtained via prompt tuning or its variants (Lester et al., 2021; Liu et al., 2021b;a; Li & Liang, 2021) could further help condition models to perform better in out-of-domain or scarce programming languages.
4. R OBUSTNESS E VALUATION : R -MBXP
We evaluate the robustness of models across r-MBXP datasets perturbed by common transformations in NL-Augmenter (Dhole et al., 2021), a standard collection of data augmentations for robustness evaluation on text.
Our experiments show that multi-lingual models are more robust on average, with less percentage of performance drops ( 7 . 80% vs 9 .
39% for multi- and mono-lingual models) and higher pass @ 1 scores across most perturbed datasets compared to mono-lingual models.
For more details and other interesting observations on robustness, we refer readers to Appendix J.
As the first code-generation robustness benchmark, we encourage researchers to further investigate robustness evaluation metrics, data augmentations, adversarial attacks, and defenses based on our released datasets.
4. C ODE I NSERTION : I -MBXP
We introduce i-MBXP, an insertion-based variant of our MBXP benchmark, which is the first execution-based multi-lingual code insertion benchmark.
Each data sample consists of left and right contexts where we split the original function signature and the canonical solution into left context, right context, and ground truth insertion code.
Code insertion is evaluated in an execution-based manner by using the same test statements as in MBXP.
We benchmark using the publicly available insertion-based model, InCoder (Fried et al., 2022).
Both models show that incorporating right context can significantly boost performance compared to using only the left context.
In InCoder, we observed 23 .
2% , 14 . 4% , and 37 .
6% relative improvements on Python, JavaScript, and Java respectively compared to the case without right context.
Ablation studies on the performance versus the number of right context lines show a positive correlation, indicating the models’ abilities to incorporate partial right context information to improve prediction.
We provide further details on dataset construction and results in Appendix K. This work demonstrates the usefulness of our MBXP benchmark for code insertion and highlights the need for further research in execution-based multi-lingual code insertion evaluation.
4. C ODE S UMMARIZATION : S -MBXP
We evaluate the ability of models to perform code summarization, where we use a function signature along with its solution as the prompt, with the natural language description in the docstring removed.
Based on this prompt, we induce the model to generate the description of the code’s functionality.
Our results show that, in both zero-shot and few-shot settings, multi-lingual models generally outperform mono-lingual models, consistent with the performance trends observed in other evaluation tasks discussed in Section 4.2.3.
In the few-shot case, we observe noticeable improvements compared to the zero-shot setting, with more significant improvement on larger models.
We provide examples and detailed results in Appendix L.
R ELATED W ORK
Many other evaluation datasets can be considered for the conversion to multi-lingual counterparts such as APPS (Hendrycks et al., 2021) and CodeContest (Li et al., 2022a).
These datasets in its original forms are execution-based datasets containing challenging algorithmic competition problems and tests that are language-agnostic, but can be converted to Python and many other languages.
Existing benchmarks for code generation are primarily either match-based or focused mostly on
Python, if not language-agnostic.
Our work fills a gap in the literature by providing a multi-lingual code evaluation framework that includes synthetic solutions, handles datasets beyond HumanEval (e.g., MBPP and MathQA), and investigates various types of code generation abilities.
Concurrent work by Cassano et al.
(2022) converts prompts and test cases of HumanEval into multiple languages.
Recent work by Orlanski et al.
(2023) presents BabelCode, a framework for executionbased evaluation, and investigates the effectiveness of balancing the distribution of languages in a training dataset.
Together, these works provide a valuable resource for researchers to evaluate multilingual code generation.
We provide further discussion of related work in Appendix B.
D ISCUSSION
Our release of these datasets is a significant contribution to the field of code generation research, providing researchers with a valuable resource to evaluate various aspects of code generation abilities.
The findings from our evaluations have shed light on interesting areas such as multi- vs mono-lingual models, out-of-domain performance, zero-shot translation abilities, and multi-lingual code insertion, all of which hold potential for advancing the state-of-the-art in code generation.
Our observations suggest that large multi-lingual models are more effective than multiple monolingual models in code generation tasks, benefiting from the data spillover across languages.
The success of our multi-lingual models in out-of-domain evaluations and robustness testing demonstrates their potential to generalize to new languages and tasks.
However, to comprehensively evaluate the complexities of real-world software development tasks, it may be necessary to include additional language-specific evaluations where appropriate.
Overall, our datasets provide a solid foundation for future research to explore and enhance various aspects of code generation, with the potential to lead to significant advancements in the field.
Table of Contents
A Extended Discussion
A.1 Implication of findings . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
A.2 Implication of Evaluation Data at Scale . . . . . . . . . . . . . . . . . . . . . .
A.3 Possibilities of true generalization . . . . . . . . . . . . . . . . . . . . . . . . .
A.4 Potential proxy for general coding capabilities . . . . . . . . . . . . . . . . . .
A.5 Limitations . .
A.6 Generation tendency versus generation ability . . . . . . . . . . . . . . . . . .
B Other Related Work
C
Evaluation Setup
C.1 Sample Generation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
C.2 Stopping Criteria . .
C.3 Code Execution . .
D Evaluation Results on Additional Datasets
D.1 Multi-lingual MathQA . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
D.2 Multi-lingual HumanEval . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
E Language “Spillover” in Training Data
E.2 Example
Embedded JavaScript
Python files . . . . . . . . . . . . . . . . .
E.
Example
F Execution-Based Function Completion Results
F.1 Performance Trend with Respect to Model Size . . . . . . . . . . . . . . . . . .
F.2 Comprehensive Sampling Results . . . . . . . . . . . . . . . . . . . . . . . . .
G Few-Shot Prompting
G.1 Evaluation Results . .
G.2 Qualitative Examples . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
H Translation
H.1 Translation Results from Various Language Sources . .
H.2 Comparing translation performance of multi-lingual and mono-lingual models . .
H.3 Generated Translation Examples . . .
I Analysis: Effects of few-shot and translation prompts
I.
Test case error versus non-assertion error . . . . . . . . . . . . . . . . . . . . .
I.
Solve rate per problem due to few-shot prompting and translation . . . . . . . .
J Robustness Evaluation: r-MBXP
J.
Dataset Preparation and Evaluation Setup . .
J.
Evaluation Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
J.
Qualitative Examples . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
K Code Insertion: i-MBXP
K.
[Table]
Preparation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
K.2 Evaluation Setup . .
K.3 Evaluation Results . .
[Table] [Table]
[Table] [Table]
[Table]
Examples . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
M
[Table]
Models
[Table] [Table]
[Table]
O
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
R
[Table]
MBXP . .
[Table]
[Table]
lingual MathQA . .
A E XTENDED D ISCUSSION A. I MPLICATION OF FINDINGS
From our findings, it is clear that a large multi-lingual model compared to multiple mono-lingual is a better choice if we are to consider deploying code generation models.
This is due to the data spillover from each language source which reinforces the knowledge of the model in the multilingual training.
However, such model needs to be of sufficient size to capture all the available knowledge.
For our controlled setting, model sizes 2 .
7 B and above begin to clearly outperform all mono-lingual models.
It is possible that as the number of languages in the training set increase, the required size for the multi-lingual model to be superior to individual mono-lingual models can increase.
A. I MPLICATION OF E VALUATION D ATA AT S CALE
Our parallel datasets provide a valuable resource for studying the translation abilities of code generation models.
By leveraging the canonical solutions in our source language, we can evaluate how well the models generate code in any other supported language.
This opens up a range of research questions, such as how well the models generalize across languages, what factors contribute to successful or unsuccessful translations, and how different modeling strategies affect translation performance.
A. P OSSIBILITIES OF TRUE GENERALIZATION
Out-of-domain evaluations from our controlled experiments reveal interesting behavior of how code in multiple languages present themselves in natural data.
We hypothesize that the out-of-domain code generation abilities are mainly due to the data spillover.
However, we believe it is also possible that a true generalization plays a role where the model is able to complete code in a new language that is not in the training data at all.
To test this, we can design a new language which avoids the complication of data spillover in the any training dataset.
We can use our framework to construct the evaluation set in such language and use it to evaluate the existing models.
However, we note that such new language likely are similar to existing languages in the training set in some aspects.
For instance, the control flows (if clause), loops, variable declaration, or objects such as lists or dictionaries potentially might not differ much from each component of existing languages.
Even with the new language constructed, the boundary between evaluating a true generalization versus generalization between data spillover can be somewhat unclear.
A. P OTENTIAL PROXY FOR GENERAL CODING CAPABILITIES
MBXP and other code completion benchmarks such as HumanEval measure the general understanding of basic tasks from natural language description with function signature and the model’s ability to complete such tasks.
Given the description of these problems in natural language and function signature where a competent human coder can complete, this benchmark helps measure if a code generation model can perform such tasks.
The scores of these evaluations can be a useful proxy for overall code capabilities if they correlate with the performance on all coding-related tasks.
We believe that such correlation is possible or likely the case if the models are not trained to adapt to a specific distribution of evaluation datasets.
By using these evaluations as proxies of general coding abilities, we implicitly accept the premise that zero-shot evaluation on a slice of all possible problems (the slice being MBXP, for instance) is an unbiased proxy to measure overall model’s capabilities in each language.
Hence, in this paper, we particularly avoid finetuning even though results in the literature demonstrate increased performance so that the results established can be less biased towards specific kinds of coding problems and can better reflect models’ true coding capabilities.
A. L IMITATIONS
The proposed conversion framework is well suited for basic programming problems that are applicable to a wide set of programming languages.
While the original MBPP dataset is meant for basic programming problems, some tasks can be more appropriate for certain languages than others.
For instance, string manipulation problems can be naturally encountered in languages such as Python or PHP more than C++.
By design, our conversion “naively” assumes that a problem is relevant to the target language which might not be true for all problems in a given dataset.
That is, the scores obtained from MBXP benchmark might not align with the distribution of natural usage in different languages equally.
In addition, the programming problems in MBXP do not cover language-specific functionalities; for instance, there are no specific questions related to web development for JavaScript, or memory allocation for C++.
Therefore, it can be unclear how the conclusion from MBXP benchmark transfers to coding performance in the wild given the complexity of real-world software development.
The test conversion we support are value-oriented which do not cover all possible types of testing.
The value-oriented test performs assertion by checking if the values match.
If the assertion process is more complex such as in deep integration tests with specific code packages or APIs, the conversion process is not applicable.
In fact, we explicitly define the types of Python objects that we support converting from in Appendix O.
We suggest that it can be beneficial to complement MBXP evaluation with other language-specific evaluation, if available.
A. G ENERATION TENDENCY VERSUS GENERATION ABILITY
The prompts in our benchmark heavily guide the model to generate in a particular language.
For example, when a prompt contains function method_name( , the model is highly encouraged to generate code that has such syntax, in this case PHP, but not Python where the function signature would have started with def method_name( .
In that sense, this benchmark measures the ability of a model to conform to the guided prompt and the completion ability based on the function signature that has already started, and not necessarily the tendency to generate in particular languages.
Note that without explicit guidance or special tags, models can generate code in any languages, especially multi-lingual models, which makes fair evaluation of code completion harder since we might not penalize correct code that is correct but in a different language.
Our prompt format helps isolate evaluation of the generation ability in a desired language from the generation tendency.
This is contrast to free-form prompt style in datasets like the original MBPP, APPs, or CodeContests where the model generates its own function signature.
However, in our case, if the evaluation is outof-domain, it is still possible that with explicit guidance of function signature, the model can still generate in a similar yet different language, as in the case of confusion between Ruby and Python with similar function signature syntax.
We also observe that while this benchmark is about generic understanding of basic programming tasks and does not particular attempt to measure the knowledge of a model in terms of specific syntax in the desired language , we observe that language-specific syntax usage typically emerge, for instance, the usage of list.select in Ruby, or nums.filter in Kotlin to select elements of a list, instead of a generic for loop.
We provide sample generations for all converted languages in
Section R.1.
B O THER R ELATED W ORK
Code Generation Models Language models for code generation is a rising domain in recent years.
CodeBERT (Feng et al., 2020) is the first BERT-based language model trained on code.
GraphCodeBERT Guo et al.
(2021) improves upon CodeBERT by leveraging AST and data flow.
CodeT5 Wang et al.
(2021) and PLBART Ahmad et al.
(2021a) pretrained encoder-decoder based generative language models for code.
More recently, various work have been proposed to use large language models for code generation.
Codex (Chen et al., 2021) was pretrained on Python on top of GPT-3, resulting in up to 175B parameters code generation models.
CodeGen (Nijkamp et al., 2022) was pretrained on multiple programming languages and optimized for conversational code generation with up to 16B parameters.
InCoder (Fried et al., 2022), along with CoditT5 Zhang et al.
(2022a)
on a similar line of research, is able to perform program synthesis (via left-to-right generation) as well as editing (via infilling) with up to 6.7B parameters.
Further, researchers also found that generic (natural) language models are also able to perform code completion to a certain degree, e.g., PaLM (Chowdhery et al., 2022) and BLOOM (Mitchell et al., 2022).
In addition, researchers proposed various ways of improving code generation models.
For example,
Poesia et al.
(2022) propose Target Similarity Tuning for code retrieval augmentation and Con- strained Semantic Decoding to improve code generation by constraining the output to a set of valid programs in the target language.
Shi et al.
(2022) introduce execution result–based minimum Bayes risk decoding that improves choosing a single correct program from among a generated set.
Another line of work is to “repair” the generated code by language models, e.g., (Fan et al., 2022; Li et al., 2022b).
Our work is model-agnostic and complimentary to all the above works that serves as a testbed of code generation.
Code Completion Resources
Many code completion evaluation benchmarks have been proposed recently, but they differ in style and focus.
Lu et al.
(2021) composed a token and line completion datasets in Java and Python based on existing benchmarks (Allamanis & Sutton, 2013; Raychev et al., 2016).
Clement et al.
(2021) presented a method generation dataset in Python based on
CodeSearchNet (Husain et al., 2019).
All these datasets are primarily collected from open-source projects or GitHub and focus on match-based evaluation (using n-gram metrics).
In contrast to these efforts, recent works promote unit tests-based execution evaluation to assess the functional correctness of ML-based code completion techniques.
In this line of work, Austin et al.
(2021) introduced the MBPP dataset focusing on basic programming problems where the prompt format consists of a natural language description and assert statements in Python.
HumanEval (Chen et al., 2021) focuses on more challenging algorithmic problems with prompts containing function signatures in Python along with docstring descriptions of the problems, including test case descriptions.
APPS (Hendrycks et al., 2021) and CodeContest (Li et al., 2022a) contain language-agnostic problems in algorithmic competition style and tend to be very challenging.
Both datasets expect solutions (complete programs, unlike functions in other datasets) in any language that uses standard input and output to consume and return values.
The output is compared directly with the expected values without test cases written in any particular language to test for correctness.
In contrast, HumanEval and MBPP use test statements written directly in
Python.
We show all the dataset formats for comparison in Section R.1.
We find that the HumanEval format aligns best with how programmers would write in a typical coding environment; therefore, we use this format for our converted MBXP benchmark.
We also convert the original Python MBPP dataset to be of this format as well for comparison consistency.
Our benchmark, MBXP, is the first execution-based function completion benchmark available in multiple languages for all (or most) tasks in parallel.
Code Translation Resources Several works in the literature have developed parallel corpus to facilitate source code translation.
Earlier works (Nguyen et al., 2013; Karaivanov et al., 2014) focused on building semi-automatic tools to find similar functions in Java and C# from open source projects.
Subsequent works used libraries and transcompilers to construct parallel corpora in Python 2 and Python 3 (Aggarwal et al., 2015), and CoffeeScript and JavaScript (Chen et al., 2018).
Among the recent works, Lachaux et al.
(2020a) collected a corpus of parallel functions in Java, Python, and C++ from GeeksforGeeks and provided unit tests for execution-based evaluation.
Very recently, Szafraniec et al.
(2022) extended the dataset in Go and Rust languages.
On a similar line,
Zhu et al.
(2022) introduce a new dataset which is parallel across 7 programming languages on both snippet level and program level based on GeeksforGeeks data.
Another work (Ahmad et al., 2021b) aggregated a comparatively larger parallel corpus in Java and Python by collecting programming problem solutions from several sources.
Different from the prior works, our proposed dataset,
MBXP, covers a wide range of languages with unit tests to facilitate the evaluation of functional accuracy of ML-based code translation models.
Multi-lingual Evaluation of Code Generation Models Wang et al.
(2022) proposed MCoNaLa, a multi-lingual version of CoNaLa Yin et al.
(2018) in various natural languages.
This is orthogonal to our work that extends multi-linguality on programming languages.
Similar approaches could be applied to MBXP to expand the dataset to multiple natural languages and we leave it as one of the future directions.
C E VALUATION S ETUP C. S AMPLE G ENERATION
We use nucleus sampling with p = 0 .
95 (Holtzman et al., 2020).
For all experiments, limit the input length to be 1792 and generate up to 256 tokens.
If the context exceeds 1792 tokens, we perform truncation from the left.
Note that the truncation can happen more often in the case of few-shot prompting or translation settings.
C. S TOPPING C RITERIA
We categorize languages into groups where each group has the same stopping criteria.
• Curly-brace style with standalone function: JavaScript, TypeScript, Go, Kotlin, PHP, C++, Rust.
We truncate right after the closing } character.
• Curly-brace style with function wrapped under class: Java, C#.
We truncate right after the closing } and add \n} to close the higher-level class wrapper.
This is slightly different from letting the model generate a closing } for the wrapper class.
We find that if we do let the model generate a closing } on its own, it can go on to generate another function, which technically should not harm the evaluation, but it can cause the generation to be too long and can hit the maximum token limit.
Therefore, we find that it is fair and more efficient to close out the class right away after the current function is generation.
• Other keywords: ‘end’ for Ruby
Note that it is possible to extend these stopping criteria to include multi-function evaluation, where the first function can refer to other functions that follow.
However, it is out of scope for this current paper.
C. C ODE E XECUTION
We adapted the human-eval † repository by OpenAI which provides multi-thread execution-based evaluation framework in Python along with unbiased pass @
k calculation.
Our adaptation supports execution in all languages in MBXP where we use Python’s subprocess to execute native command in each respective language.
For instance, we execute with node file.js for JavaScript.
The test statements for each language are such that exceptions are thrown if the test cases fail.
Each task can also fail due to improper generated code that does not parse or compile.
We capture the failure or success of each execution via exit code as well as standard error message for further analysis.
D E VALUATION R ESULTS ON A DDITIONAL D ATASETS D. M ULTI - LINGUAL M ATH QA
We evaluate our 13B multi-lingual and mono-lingual models on MathQA datasets in different programming languages.
The original MathQA dataset was formatted to the Python version by Austin et al.
(2021), after which we use our conversion framework to obtain the corresponding data in different languages for analyses.
We do not leverage the training and validation sets of MathQA to finetune our models.
The purpose of this evaluation is to investigate the generalization capability of our models on complex context, which requires mathematical reasoning.
Similar to Section 4.4 and 4.3, we compare the models with respect to adding few-shot prompts, conducting canonical solution translation, as well as the normal function completion.
[Table]
Specifically, for the few-shot setup, we prepend the first 4 examples in the MathQA training data with their canonical solutions.
For MathQA-Python, the canonical solutions are given; we manually adapt the Python solutions to other languages for these four examples.
For solution translation, we evaluate the models on Java and JavaScript with the Python canonical solutions given in the context.
The mono-lingual models are only evaluated on the MathQA dataset of the same language.
Following findings are summarized below based on Table 1.
• Both multi-lingual and monolingual models perform very well for solution translation ( > 94%), probably because the solutions required for solving MathQA problems are usually simple mathematical calculations.
Converting them to different languages are straightforwards, if python solutions are provided.
• Prepending few-shot examples achieves better performances than normal predictions for both multi-lingual and monolingual models.
As illustrated in the MathQA example in Section R.3, the context are significantly different from the training corpus.
Involving a few examples from MathQA domain in the context does help alleviate the domain divergence.
The multi-lingual models do not consistently outperform the monolingual counterparts.
This may be due to the nature of MathQA problems that emphasizes the models’ mathematical reasoning capability.
Below, we present a correct prediction and a failed case for MathQA-Java examples under the translation mode.
The case below demonstrates that the model understands that exponentiation in Python ( a**b ) and is able to translate to Math.pow(a,b) .
[Figure]
[Table]
[Figure]
[Table] [Table]
return answer
[Table]
[Table] [Table]
return answer;
[Table]
Illustrated by the failed example below, despite the good overall performance, the model sometimes fails to translate mathematical built-in functions from Python to Java (eg. max in Python vs.
Math.max in Java).
Additionally, math.log in Python can take a second argument for logarithmic base, while Math.log in Java specifically means natural logarithm, taking only one argument.
The translation model ignores this difference.
[Figure]
789 class Problem {
[Table]
D. M ULTI - LINGUAL H UMAN E VAL
We present the results on multi-lingual HumanEval in Section M using our models as well as publicly available models.
We find that the results on few-shot prompting and translation are generally consistent with MBXP.
Details on multi-lingual HumanEval dataset preparation can be found in
Section R.
E L ANGUAGE “S PILLOVER ” IN T RAINING D ATA
Our evaluation indicates that code generation models typically have out-of-domain generalization performance (see Section 4.2).
We hypothesize that it is due to the effect of data spillover that are quite common especially in cross-lingual code projects where each file can have multiple languages present.
In this section, we provide discussion and examples of such cross-lingual code occurrences.
E. T YPES OF C ROSS -L ANGUAGE D ATA S PILLOVER
We provide discussion on types of data observed for code in the wild where multiple languages can co-occur.
In particular, there are four categories:
Source code from two programming languages occurring in the same file via explicit language embedding mechanism other than “putting code in strings”.
There are actually two categories — “deep” and “shallow” embeddings of the guest language into the host language.
A good example of this in Python is https://nyu-cds.github.io/ python-numba/05-cuda/ which uses python syntax but does not have the semantics of the corresponding python program.
Source code from two programming languages occurring in the same file, where the “guest language” is included in the “host language” via the host language’s string type.
Most web code will fit in this category, but also stuff like code generators (e.g. https://github.com/LS-Lab/KeYmaeraX-release/blob/master/ keymaerax-webui/src/main/scala/edu/cmu/cs/ls/keymaerax/ codegen/CExpression.scala ) 3.
Source code from two programming languages occurring in the same project, but always in separate files.
This is another potential source of cross-lingual data, but it does not apply to the models trained in our paper since we filter languages per file, not per project.
Combinations of programming languages via a Foreign Function Interface, where the host language does not explicitly use any source code from the source language but does, e.g., refer to
identifiers or function names in compiled bytecode.
E. E XAMPLE 1: E MBEDDED J AVA S CRIPT IN P YTHON FILES
The example below taken from https://github.com/brython-dev/brython/blob/ master/scripts/make_encoding_js.py#L30 shows JavaScript written in Python strings throughout the code file make_encoding_js.py .
[Figure]
[Table] [Table]
[Table] [Table]
[Table] [Table]
A simple search query ‡ on Github can reveal multiple other examples.
E. E XAMPLE 2: J AVA AND P YTHON INTEGRATION AS J YTHON
This example is taken from https://jython.readthedocs.io/en/latest/
JythonAndJavaIntegration/ which shows a combination of Java and Python code in a cross-lingual project Jython.
[Figure] [Figure]
[Figure]
print ’Initializing’pass def calculateCost(self, salePrice, tax):return salePrice + (salePrice * tax) 1213 package org.jython.book.interfaces; 1415 public interface CostCalculatorType { public double calculateCost(double salePrice, double tax); 1819 }
F E XECUTION -B ASED F UNCTION C OMPLETION R ESULTS F. P ERFORMANCE T REND WITH R ESPECT TO M ODEL S IZE
Figure 9 shows pass @ 1, pass @ 10, and pass @ 100 for many evaluation datasets in MBXP.
We can observe that the trends for pass @
k for different k are consistent, but simply different in terms of scale for scores.
That is, the observation that multi-lingual models begin to clearly outperform mono-lingual models when the model size becomes sufficiently large holds for any k
[Figure]
) Temperature
[Figure]
k =
[Figure]
F. C OMPREHENSIVE S AMPLING R ESULTS
[Figure]
[Figure]
PythonJavaJavaScript
[Figure]
[Figure]
PythonJavaJavaScript
[Figure]
[Figure] [Figure]
PythonJavaJavaScript
[Figure]
G F EW -S HOT P ROMPTING
In this section, we present more detailed results on few-shot prompting with prompt consisting of three correct functions from the respective MBXP dataset.
The few-shot prompts are selected from three correct samples for each language.
We note that this gives an automatic performance gain of roughly 0 .
3% since there are ≈ 1000 cases for each evaluation language.
However, this is quite small compared to the gains observed.
We do not tune on the few-shot examples selected; that is, these examples are chosen once and fixed for all usage.
It is possible that this can be further tuned, such as the case of prompt engineering in the literature.
G. E VALUATION R ESULTS
From Figure 14, we observe that the performance gain is quite clear especially for out-of-domain languages (pass @ 1 with temperature 0.2).
In some cases, there are large performance boosts for mono-lingual models evaluated on out-of-domain languages.
For instance, with few-shot prompting, the pass @ 1 of the 13B Python model evaluated on MBJP increases from 5 .
7% to 10 .
Similarly, the pass @ 1 of the 13B multi-lingual model increases from 5 .
9% to 12 .
2% with few-shot prompting.
G. Q UALITATIVE E XAMPLES
We demonstrate the few-shot prompts for select languages.
Each of these prompts precede the function completion prompt for each evaluation.
G.2. P YTHON FEW - SHOT PROMPT
[Figure]
def find_char_long(text):
[Figure]
""Write a function to find all words which are at least 4 characters long in a string by using regex.
[Figure]
find_char_long(’Please move back to stream’)
[Figure]
[Figure]
, ’move’, ’back’, ’stream’]
[Figure]
>>> find_char_long(’Jing
Eco and Tech’)
[Figure] [Figure]
, ’Tech’]
[Figure]
>>> find_char_long(’Jhingai wulu road Zone 3’)
[Figure]
[’Jhingai’, ’wulu’, ’road’, ’Zone’
]"""import re return re.findall(r"\b\w{4,}\b", text) 141516 def square
[Figure]
Write a function to find squares of individual elements in a list using lambda function.
>>> square_nums([1, 2, 3, 4, 5, 6, 7, 8, 9, 10])[1, 4, 9, 16, 25, 36, 49, 64, 81, 100]>>> square_nums([10,20,30])([100,400,900])>>> square_nums([12,15])([144,225])"""return list ( map (lambda x: x**2, nums))
29 def test_duplicate(arraynums):"""
Write a function to find whether a given array of integers contains any duplicate element.>>> test_duplicate(([1,2,3,4,5]))
False>>> test_duplicate(([1,2,3,4, 4]))
True>>> test_duplicate([1,1,2,2,3,3,4,4,5])
True"""if len (arraynums) =
=
len ( set (arraynums)):return Falseelse:return True
[Table]
pa ss @
k 6.
[Table] [Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
pa ss @
k 24.
[Figure] [Figure]
[Figure] [Figure]
nan 0.
Few
[Figure]
G.2. J AVA S CRIPT FEW - SHOT PROMPT
[Table]
G.2. PHP FEW - SHOT PROMPT
php
[Table] [Table]
[Table] [Table]
G.2. R UBY FEW - SHOT PROMPT
[Table]
H T RANSLATION H. T RANSLATION R ESULTS FROM V ARIOUS L ANGUAGE S OURCES
In this section, we show translation results using (1) multi-lingual and mono-lingual models of various scales and (2) three different languages as source solutions (Python, Java, JavaScript).
We note that the canonical solutions from Java and JavaScript are from the data bootstrapping using a separately trained model, as detailed in Section P. For tasks that we do not have solutions for, we do not prepend anything to the usual target-language prompt.
While the training data can potentially consist of translation-like data which allow the model to perform zero-shot translation, we do not know the volume of such translation-related data and suspect such volume to be low.
In addition, the model is not trained specifically on certain languages such as Kotlin or PHP for multi-lingual models, or even on Java for Python-only models, for instance.
Figures 15 illustrate the zero-shot translation results for multi-lingual, Python, JavaScript and Java mono-lingual models respectively.
In most settings, we observe improvements due to zero-shot translation over the baseline.
Out-of-domain evaluation languages benefit more from translation We can observe consistent performance gains due to translation as opposed to without using reference solution.
The performance gain is drastic in certain cases.
For example, for Ruby, the 13B multi-lingual model obtains 5.9% pass @ 1 in the normal mode and 15.9% in the translation mode with JavaScript as a source language, or for PHP, the performance improvement is from 19 .
1% to 46 .
5% with Java as a source language.
Effects of language compatibility or affinity for zero-shot translation Based on the trends of performance gains from the translation settings, we observe that different source languages have unequal effects as reference solutions.
For instance, based on the multi-lingual 672M and 13B models, Java is the source language that yields the highest performance for MBPHP, whereas JavaScript seems to be the best for MBRBP and MBKP.
These compatibility trends can change slightly but are roughly consistent.
For instance, for MBJSP, JavaScript is the best source language for the 13B
JavaScript and Java monolingual models, whereas Python is the best source language for MBJSP in many other settings.
However, for MBKP and MBRBP, JavaScript consistently is the best source language across all model types.
We summarize the best model types for each evaluation set below in Table 2.
We observe that it is not necessarily the source languages that are closest in syntax that is the best source language, since it has potential to confuse the models during translation and lead the model to generate in an incoorect syntax.
[Table]
We provide some examples in Section H.3.
Mono-lingual versus multi-lingual models For mono-lingual models, we observe large performance boost, partly due to mono-lingual models not performing well for baseline to start with.
Trends with respect to model sizes Larger models typically perform better, as observed in the normal code completion case and also in the translation case as well.
Model knowledge of source language versus target language It is likely the case that the knowledge of the target language is more important than the source language for translation performance.
We note that Python model obtains high scores with translation on MBJSP with Python as source ( 13 . 8% → 30 . 7% ).
JavaScript model also obtains high scores with translation on MBJSP with
Python as source, with better performance compared to Python model, which is in part due to better baseline performance to start with ( 23 .
3% → 32 . 8% ).
[Figure]
[Figure] [Figure]
[Table]
[Figure]
[Table]
[Figure]
source language.
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table] [Figure] [Table] [Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure] [Table] [Figure] [Table]
[Figure]
[Table] [Figure] [Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
H.
G ENERATED T RANSLATION E XAMPLES
H.3.
[Figure]
(PHP
[Figure]
find_Sum(arr,n)
[Figure]
Write a python to the sum repeated elements in a given array.>>> find_Sum([1,2,3,1,1,4,5,6],8)3>>> find_Sum([1,2,3,1,1],5)
[Figure]
>>
[Table]
find
[Table]
Sum([1,1,2],3)
[Table]
[Figure]
" return sum( for x in arr if arr.count(x) > 1])
[Table]
php
[Table]
[Table] [Table]
function
[Table]
) { $
[Table]
=
[Table]
values
[Table]
);return array
[Table]
sum(array filter($arr
($
) use ($
) {return $count[$value] > 1;})); 31 } 1 def find_Product(arr,n):"""
Write a python function to find the product of non-repeated elements in a given array.>>> find_Product([1,1,2,3],4)
[Figure]
>>> find_Product([1,2,3,1,1],5)
[Figure]
>>> find_Product([1,1,4,5,6],5)
[Figure]
arr.sort()prod
[Table]
for i in range(
[Table] [Table]
if (arr[i - 1] !=
arr[i]):prod = prod * arr[i]return prod; 1819 <
[Table]
php
[Table]
You are an
[Table]
PHP
[Table]
task
[Table]
$
[Table]
foreach($arr as $key =>
[Table] [Table]
if($key =
=
|| $value !
=
$arr[$key - 1]) {$prod = $prod * $value;}}return $prod; 40 }
1 /** 2 * Write a javascript function to find the element occurring odd number of times.
3 * > getOddOccurrence([1, 2, 3, 1, 2, 3, 1], 7) 4 *5 * > getOddOccurrence([1, 2, 3, 2, 3, 1, 3], 7) 6 *7 * > getOddOccurrence([2, 3, 5, 4, 5, 2, 4, 3, 5, 2, 4, 4, 2], 13) 8 *9 */ 10 function getOddOccurrence(arr, arrSize) { for i in range(0, arr_size):count =
0for j in range(0, arr_size):if arr[i]
=
=
arr[j]:count += 1if count % 2 !=
:return arr[i]return202122 ## 23 #
You are an expert Ruby programmer, and here is your task.
24 # Write a to find the element occurring odd number of times.
25 # irb> get_odd_occurrence([1, 2, 3, 1, 2, 3, 1], 7)
# =>27 # irb> get_odd_occurrence([1, 2, 3, 2, 3, 1, 3], 7)
# =>29 # irb> get_odd_occurrence([2, 3, 5, 4, 5, 2, 4, 3, 5, 2, 4, 4, 2], 13) 30 # =>3132 def get_odd_occurrence(arr, arr_size)occurrences =
Hash.newfor i in 0...arr_sizeoccurrences[arr[i]] ||= 0occurrences[arr[i]] += 1endresult = niloccurrences.each_pair do |k, v|if
v % 2 != 0 && result =
=
nilresult =
kendendresult 4546 end
H.3.
MBKP (K OTLIN )
[Table]
23 */ 24 fun sortMatrix(m :
List< List<
Int >>) :
List< List<
Int >> {return
I A NALYSIS : E FFECTS OF FEW - SHOT AND TRANSLATION PROMPTS
Below, we show an extended version of the analysis in main text (Figure 7) where we demonstrate the differences in qualitative behaviors of few-shot prompting versus translation settings and their effects on helping the usual function completion task.
The differences in these two modes are simply the prompts the precede the function completion prompt.
That is, the few-shot setting uses 3 examples in the corresponding language and the translation mode uses the solution from the same problem in a different language.
The translation setting helps the model solve difficult tasks that are very difficult to solve without a reference solution (Figure 21) whereas the the few-shot prompting helps condition the model to generate code properly in that respective syntax (See Section I.1).
I. T EST CASE ERROR VERSUS NON - ASSERTION ERROR
We categorize the failure of each generation code sample into two main categories: assertion or testbased errors versus non-assertion errors, which consist of all other errors such as compile, parsing, or runtime error not related to test cases.
We use the results from temperature 0 .
2 with 30 samples for each problem and calculate the fraction of non-assertion errors over the number of all samples.
The results in Figure 19 show that the few-shot prompting results in lower non-assertion errors for out-of-domain languages, indicating that few-shot prompts help models generate code with more precise syntax in each language.
In contrast, there is little effect even evaluated on the in-domain languages, since the models already are fluent in these languages where the additional signals from the few-shot prompts do not help further.
For the translation case, interesting we observe higher non-assertion errors on in-domain evaluation.
[Figure]
I. S OLVE RATE PER PROBLEM DUE TO FEW - SHOT PROMPTING AND TRANSLATION
We perform sampling to generate 100 samples with temperature 0 .
For each task id, we calculate the fraction of number of code samples that pass the test over a total of 100 samples.
We repeat this experiment for the function completion baseline, few-shot prompting
[Figure]
and translation settings.
In
Figure 21, we sort the task ids by the solve rate of the function completion baseline
[Figure]
which indicates the difficulty of various tasks in each dataset.
We observe differences in how the solve rates for fewshot prompting or translation settings accumulate.
For the few-shot case, the accumulation of the solve rates per task revolve near the baseline solve rate, indicating that the difficulty of the problem given the few-shot prompts do not deviate much from the difficulty in the baseline case.
However, in the translation case, some tasks ids that correspond to low baseline solve rate have much higher solve rate in the translation case, sometimes with perfect rate 1 . 0 .
[Figure] [Figure]
number of samples (solve rate), where the task indices are ranked to show increasing difficulty.
In the translation setting, tasks that are previously difficult (low solve rate for the baseline) can become easily solvable, demonstrating that models can leverage reference solutions in the source language to solve hard tasks.
In contrast, the solve rates with fewshot prompting do not deviate as much from the baseline solve rate.
Translation from different language sources exhibit similar trends where the source solution can help solve hard tasks where we observe unequal effects from different source languages.
Figure 21 also demonstrates consistent effects of source languages.
For a language such as Java, we observe that it can help solve many of the hard problems for MBPHP (PHP) evaluation, based on the high concentration of points around solve rate 1 . 0 .
This analysis also complements Section H which demonstrates unequal effects of source languages.
J R OBUSTNESS E VALUATION : R -MBXP J. D ATASET P REPARATION AND E VALUATION S ETUP
Robustness is an important indicator of the reliability of code generation models in practice.
Here we provide a robustness benchmark for all the trained models across MBXP datasets.
Specifically, we consider three natural data augmentations (1) Paraphrase by Back Translation (Li & Specia, 2019; Sugiyama & Yoshinaga, 2019)
(e.g., “create a function” to “write one function”) (2) Character
Case Change (“Create A FunctioN”) and (3) Synonym Substitutions (Miller, 1995) (“generate a function”) as basic transformations to perturb the docstrings in prompts.
We use the default settings and implementations of these three transformations from NL-Augmenter § , a standard collection of data augmentations for robustness evaluation on text (Dhole et al., 2021).
We select these three transformations since they can mostly maintain the naturalness for the tasks of code generations based on our observations.
We then measure the average pass@1 with greedy decoding for all the models on datasets perturbed by each transformation.
Here multi-lingual models are trained with multiple languages including Python, Java, Javascript (JS) while mono-lingual models are trained with each language individually.
To simplify the comparisons, we call Ruby, PHP, and Kotlin as out-of-domain datasets for all the evaluated models while Python, Java, JS as in-domain datasets.
J. E VALUATION R ESULTS
We present the detailed results in Figure 22 and summarize several interesting observations below.
(1) The percentages of pass@1 drops on perturbed datasets over regular ones are consistent across different sizes of the models.
In specific, the average pass@1 over all the datasets drops from 2.
to 2.07 for 125M models (8.56% drop), 6.40 to 5.87 for 672M models (8.25% drop), 9.20 to 8.
for 2.7B models (9.40% drop),
and 12.63 to 11.62 (8.02% drop).
(2) For in-domain datasets, multilingual models have less percentage of performance drops compared to mono-lingual models under perturbations.
On average, pass@1 of multi-lingual models drops from 21.36 to 19.73 (7.63% drop) while pass@1 of mono-lingual models drops from 16.90 to 15.24 (9.81% drop).
(3) For out-ofdomain datasets, multi-lingual models also have less percentage of performance drops compared to mono-lingual models under perturbations.
On average, pass@1 of multi-lingual models drops from 6.78 to 6.24 (7.98% drop) while pass@1 of mono-lingual models drops from 2.72 to 2.47 (8.97% drop).
[Figure]
J. Q UALITATIVE E XAMPLES
In this subsection, we provide qualitative examples to illustrate the three types of perturbations applied to the datasets.
A successful and a failure MBPP sample completions before and after each type of perturbation are provided based on the code completion results by 672M python model on MBPP dataset.
We performed similar perturbations for all the other datasets for robustness evaluation (quantitative results shown in Section J.2).
MBPP Example 1 for Back Translation Paraphrasing
[Figure]
a
[Figure]
function to left
[Figure]
a Python function to rotate the bits of a given number to the
[Figure]
left_
[Figure] [Figure]
n
[Figure]
MBPP Example 2 for Character Case Changing
[Figure]
[Table]
MBPP Example 3 for Synonym Substitutions
[Table]
[Table] [Table]
## for key in
[Table]
:if dict [key]
[Table]
return Falsereturn True 17 ### End of code completion ###
[Table]
672M
[Table]
completion on
[Table]
prompt] (
[Table] [Table]
20 def check
[Table]
value(
[Table] [Table]
Write a function to check if all values are like in a dictionary
[Table] [Table]
[Table]
Cox
[Table] [Table]
>
[Table] [Table]
[Table] [Table]
Cox
[Table]
False""
[Table]
30 ### Begin of code completion ### for key in dict :if type ( dict [key]) !
=
int :return Falsereturn True 35 ### End of code completion ###
K C ODE I NSERTION : I -MBXP K. D ATASET P REPARATION
Assume there are n lines of code in canonical solution, we skip problems with n < 2 and randomly mask out m =
[1 , 8] consecutive lines for remaining problems.
For each problem, we run multiple times (say, 1 if n < 5 , 2 if n < 12 , otherwise 3) to generate variants and remove duplicate masks.
We report data statistice in Table 3.
[Table]
K. E VALUATION S ETUP
We evaluate InCoder-6.7B model (Fried et al., 2022) pretrained on 159 GB of code, 52 GB in
Python and 107 GB in other 28 languages, and 57 GB of text content from StackOverflow.
We do greedy search and report averaged execution accuracy over all problems.
We feed a sequence left <Mask:0> right <Mask:0> to the model as prompt, where left denotes the right context and <Mask:0> denotes a sentinel token of Incoder model.
We apply two stopping criteria: (1) <EOM> is generated and (2) any token from the predefined list [" \ n class", " \ n def", " \ n # ", " \ n if"] is generated for Python or braces that close the function scope for Java and
Javascript.
After that, we match generated tokens with right context and remove duplicated tokens.
We compare it against left-right (L-R) baseline where we feed left to the model as prompt and follow the same stopping criteria.
K. E VALUATION R ESULTS
Results in Table 4 show that right context can significantly boost performance across all languages.
We also studied the effect of the number of lines of right context and observed increasing accuracy as we add more lines of right context (in Table 5), which is intuitive since more context is beneficial.
Furthermore, qualitative examples (K.4) show that models are able to leverage right context to fill in the blank.
In example example 1, given index j and k in the right context, InCoder model can fill in two inner loops L15-L16.
Otherwise, it fails to do so.
In example 2, given add operator in the right context, the InCoder model can mimic the behavior in L33-L38.
Otherwise, model might generate irrelevant operator remove .
In example 3, given result [ char ] = 1 in the right context, InCoder model generate result [ char ]+ = 1 .
Otherwise, the model will replace 1 with countStr [ char ] which results in wrong outputs.
[Table] [Table]
Q UALITATIVE EXAMPLES FOR I -MBXP
The example below shows that the model is able to use the right context information and generate appropriate insertion code that are consistent with the right context.
In contrast, the left-to-right approach does not make use of the right context which leads to a very different implementation that would not be consistent with the right context.
Example 1: Python insertion mode
return False
Example 1:
Python left-to-right mode
None
Example 2:
Java insertion mode
[Figure] [Figure]
{ /*** Write a function to iterate over elements
[Table] [Table]
.countVariable(4,
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table]
finalResult.add(countableList.get(2))
;} countFor = d;for
<
{finalResult.add countableList get
;} return finalResult;} 51 }
1 ### begin left-right ###countableList.remove(
;}countFor
;for
;countableList
;}countFor
;for
;countableList
;for
finalResult
;countableList remove(
;}return finalResult;} 21 } 22 ## left right ###
Example
JavaScript insertion
mode
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table]
" xyz"], " peek") 8 * [" keep"] 9 */ 10 function anagramLambda(texts, str) { const countFunc = (str) =
> {let result =
{};for (let i = 0; i < str.length; i++) {let char =
str[i];if (result[char]) {result[char] += 1;} else {result[char] = 1;}}return result;};const countStr = countFunc(str);return texts.filter((word) =
> {let result =
{};for (let i = 0; i < word.length; i++) {let char = word[i]; 2829 ### begin of insertion ###if (result[char]) {result[char] += 1; 32 ### end of insertion ### } else {result[char] =
1;}}return Object.keys(countStr).every((char)
=> {return countStr[char]
=
== result[char];});});
Example 3: JavaScript right mode
[Table]
if (result[char]) {result[char] += countStr[char];} else {result[char] = countStr[char];}}return
[Table]
[Table]
L C ODE S UMMARIZATION : S -MBXP L. D ATASET P REPARATION AND E VALUATION S ETUP
Here we re-purpose the original MBXP datasets for code summarization task.
We remove the natural language description from the original prompt and use the function signature and the canonical solution as the model input for code summarization.
To induce the model to generation natural language in comments, we design two types of prompt in zero-shot and few-shot setting, respectively.
Zero-shot Evaluation.
In this setting, we append “The above code writes a ” in the format of code comment after the original code prompt.
For example, in Python, the appended sequence is “# The above code writes a ”.
See examples in different languages in Section L.3.
Few-shot Evaluation.
In this setting, we select three code-summary pairs and prepend them before the original prompt.
Examples are shown in Section L.3.
To evaluate the code summarization performance, we use smoothed BLEU score as the metrics following the setting in CodeXGLUE (Lu et al., 2021) that compare the generated outputs with the groundtruth docstrings.
In MBXP datasets, the summarizations are short paragraphs with one or two sentences which makes smoothed BLEU score a suitable metrics (Feng et al., 2020).
L. E VALUATION R ESULTS
Experimental results are shown in Figure 23 covering Python, JavaScript and Java.
Overall, we found that performances are improved along with the increasing of the model size.
For example, the
BLEU-4 scores on Python language in 13B, 672M and 125M models are 6.07, 5.59, 3.20 under zeroshot settings, and 34.10, 24.72, 20.75 under few-shot settings.
We also noticed that multi-lingual models achieve better performances compared with monolingual models trained on individual languages.
An interesting observation is that though the monolingual models are trained on a specific language, they can generalize to other languages well when few-shot examples are provided.
From the table we can also notice that the improvements brought by few-shot settings are more significant on larger models.
Comparing the multi-lingual models and monolingual models under few-shot settings, we found that the multi-lingual models are more robust to the few-shot examples while monolingual models in smaller sizes show unstable performances.
L. Q UALITATIVE E XAMPLES
Zero-shot Prompt Examples.
Here we list some examples in Python, JavaScript and Java to show the zero-shot prompts, generation from models and their ground-truth.
[Figure]
1314 # Groundtruth: Write a function to calculate the discriminant value.
1516 # Generation: Write a "function" that returns the value of the discriminant of a quadratic equation.
1718 Example Python-1 19 ------ begin of prompt ------ 20 from collections import Counter 21 def add_dict(d1,d2): add_dict = Counter(d1) + Counter(d2)return add_dict 24 #
The above code writes a 25 ------ end of prompt ------
# Groundtruth: Write a function to combine two dictionaries by adding values for common keys.
28 # Generation: Write a "add_dict" function that adds two dictionaries together.
1 Example Java-0 2 ------ begin of prompt ------ 3 import java.io.
*; 4 import java.lang.
*; 5 import java.util.*;78 class ChangeDateFormat { public static String changeDateFormat(String dt) {
String[] tokens = dt.split("-");return tokens[2] + "-" + tokens[1] + "-" + tokens[0];} 14 } 15 //
The above code writes a 16 ------ end of prompt ------ 1718 //
Groundtruth: Write a function to convert a date of yyyy-mm-dd format to dd-mm-yyyy format.
Generation: Write a .txt file with the date in the format "dd-mm-yyyy" 1 Example JavaScript-0 2 ------ begin of prompt ------ 3 function swapList(newlist) {let first = newlist[0];let last = newlist[newlist.length - 1];newlist[0] = last;newlist[newlist.length - 1] = first;return newlist; 9 } 10 //
The above code writes a 11 -
Few-shot Prompt Examples.
The following examples show the few-shot prompts in different languages.
21 ### 22 # Code 23 def square_nums(nums):square_nums = list ( map (lambda x: x ** 2, nums))return square_nums 26 #
Summary: Write a function to find squares of individual elements in a list using lambda function.
27 ### 28 # Code 29 def check_tuples(test_tuple, K):res = all (ele in K for ele in test_tuple)return (res) 32 # Summary: 33 ------ end of prompt ------ 3435 # Groundtruth: Write a function to check if the given tuple contains only k elements.
36 # Generation: Write a function to check if a tuple is a subset of a list.
1 Example Java-0 2 ------ begin of prompt ------ 3 // summarize the functionality of the code 45 // Code 6 import java.io.
*; 7 import java.lang.
*; 8 import java.util.
*; 91011 class FindCharLong { public static List<String> findCharLong(String text) {
List<
String> result = new ArrayList<>(); for (String str : text.split(" "))
{if (str.length() >= 4) {result.add(str);}}return result;} 23 }n// Summary: Write a to find all words which are at least 4 characters long in a string by using rege 24 x. 25 /// 26 // Code 27 import java.io.
*; 28 import java.lang.
*; 29 import java.util.
*; 303132 class FindRotations { public static int findRotations(String str) {
String tmp = str + str;int n = str.length();for (int i = 1; i <= n; i++) {
String substring = tmp.substring
Code 51 import
*; 52 import java.lang.
*; 53 import java.util.
*; 545556 class SquareNums { public static List<
Integer> squareNums(List<
Integer> nums) {
List<
Integer> squareList = new ArrayList<
Integer>();
for (Integer num : nums)squareList.add(num*num); return squareList;} 67 } 68 // Summary: Write a function to find squares of individual elements in a list using lambda function.
Code 7172 import java.io.
*; 73 import java.lang.
*; 74 import java.util.
*; 757677 class RecurGcd { public static int recurGcd(int a, int b) {int low =
Math.min(a, b);int high =
Math.max(a, b);if (low == 0) {return high;} else if (low == 1) {return 1;}return recurGcd(low, high % low);} 89 } 90 //
Summary: 91 ------ end of prompt ------ 9293 //
Groundtruth: Write a function to find the greatest common divisor (gcd) of two integers by using recursion.
94 // Generation: Write a function to find the greatest common divisor of two numbers.
1 Example JavaScript-0 2 ------ begin of prompt ------ 3 // summarize the functionality of the code 45 // Code 6 function findCharLong(text) { let regex
= /\b\w{4,}\b/g;return text.match(regex);
9 } 10 // Summary: Write a function to find all words which are at least 4 characters long in a string by using regex.
11 /// 12 // Code 13 function findRotations(str) {let tmp = str + strlet n = str.length for (let i = 1; i <= n; i++) {let substring =
tmp.substring(i, i + n)if (str =
== substring) {return i}}return n 24 }
let square_nums
=
nums.map(function (num) {return num ** 2;});return square_nums; 34 }
36 /// 37 // Code 38 function arr
) {while (low < high) {mid = low
+ (high low)
> 1;if (arr[mid] > arr[high]) {low = mid + 1;
M E VALUATING P UBLIC M ODELS
We used MBXP to evaluate several public models such as OPT (Zhang et al., 2022b), BLOOM (Mitchell et al., 2022), and CodeGen (Nijkamp et al., 2022).
We use pass @ 1 to evaluate these models, where we generate the samples using greedy decoding.
We generate 256 tokens per example and truncate the output to one function for evaluations.
The trends we observe with public models are aligned with those observed with our models.
In general, we observe a log-linear performance gain with model sizes, across all model families, and better execution accuracy in in-domain languages.
Among the general large-language models we observe that BLOOM models outperform OPT models (See Table 6).
This can be attributed to the fact that 13 .
4% of the pretraining data used for
BLOOM models is code, while OPT does not train on code specifically.
BLOOM’s pretraining data includes code in PHP, Java, Python, Javascript, and Ruby among others, making them all in-domain languages.
This explains relatively similar performance across all languages barring Kotlin and
Ruby.
CodeGen models are trained in three stages, first is text pretraining, which is followed by code pretraining and python-only training.
Here pretraining code data includes code in Python, Java,
Javascript, C++, and Go.
The CodeGen-multi refers to the models at the end of the code pretraining stage without the python-only training, while the CodeGen-mono is models at the end of all three training stages.
Experiments with CodeGen models show similar performance trends as our models listed in Section 4.2.
Specifically, we observe that large models show better than log-linear performance on outof-domain languages (Kotlin, Java, and PHP).
Interestingly, when compared with CodeGen-multi,
CodeGen-mono 16B models show 6% , and 8% improvements on JavaScript, and PHP, respectively (See Table 6).
We speculate the additional training with python data has improved model performance in other languages as well.
With few-shot prompting, we observe significant improvements in out-of-domain languages.
Specifically, accuracy with Ruby (which is typically confused with python by the models) increased from 3 .
5% to 16 .
46% with few-shot prompting on CodeGen-multi models with few-shot learning (See
Table 8).
In translation mode, barring Ruby, we find significant improvements in all languages (See
Table 10).
[Table]
2B 31.
6B 37.
16B 40.
Ours-multi 125M 6.
672M 19.
2.7B 27.
13B 35.
Ours-mono 125M 8.
- - - 672M 19.
- - - 2.7B 24.
- - - 13B 33.
[Table]
Model Size go c++ c#
Typescript
Perl swift scala
OPT 1.3B 0.
2.7B 0.
6.7B 1.
13B 1.
Bloom 1.1B 3.
1.7B 4.
3B 6.
7.1B 7.
CodeGen-Mono 350M 1.
2B 5.
6B 3.
16B 10.
CodeGen-Multi 350M 6.
2B 12.
6B 11.
16B 15.
Ours 125M 0.
672M 7.
2B 15.
13B 5.
[Table]
[Table]
-mono
350M 17.
2B 28.
6B 39.
16B 46.
Ours-multi 125M 6.
672M 19.
2.7B 24.
13B 31.
Ours-mono 125M 8.
- - - 672M 18.
- - - 2.7B 21.
- - - 13B 31.
Model Family
Model Size
Go
C++
C#
Typescript
Perl
Swift
Scala
CodeGen-Mono 350M 1.
2B 5.
6B 6.
16B 13.
CodeGen-Multi 350M 6.
2B 14.
6B 12.
16B 20.
[Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
6B
[Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
2B
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table]
16B 49.
CodeGen-Multi 350M 10.
2B 23.
6B 40.
16B 46.
Ours 125M 1.
672M 16.
2B 28.
13B 33.
N T RAINING N. M ODEL ARCHITECTURE AND TRAINING DETAILS
We train using 210B tokens for mono-lingual models, and 630B tokens for multi-lingual models with 210 B tokens from each language.
Across all models, we use max sequence length of 2048, and use larger batch size for larger models, while reducing max steps accordingly to train all models with same amount of per-language tokens.
For example, for 13B, we use batch size of 1024 and max steps of 100,000 with 2048 sequence length, resulting in total 210B training tokens for each language.
For multi-lingual models, there are three languages and we increase max steps by three times to 630B tokens.
We use AdamW optimizer (Loshchilov & Hutter, 2018) with β 1 = 0 .
9 , β 2 = 0 .
95 , and ϵ = 10 − 8 .
We use warm up steps of 2000 steps with cosine annealing after peak learning rate, and the mininum learning rate being 10% of corresponding peak learning rate, weight decay of 0.01, and gradient clipping of 1.0.
We rescale the initialization weight standard deviation for larger models following (Shoeybi et al., 2019) for better training stability.
Our training pipeline is based on PyTorch Lightning ¶
and we use bfloat16 (Kalamkar et al., 2019) and DeepSpeed (Rasley et al., 2020) for training optimization.
We randomly split 0 .
1% data as validation set.
The validation loss curve for different sizes of multi-lingual and monolingual models are shown in Fig.
[Figure]
v a li d a t i o n l o ss
[Figure]
N. O BSERVATIONS ON VALIDATION LOSSES VERSUS PERFORMANCE
We plot the validation loss of multi/mono-lingual models on each programming languages in Figure 25.
We can see that the trend of validation loss roughly follows log-linear relationship with respect to model sizes.
By comparing the validation loss curves between multi-lingual models and mono-lingual models
[Figure]
we can see that mono-lingual models consistently achieves lower loss than multi-lingual ones.
This
[Figure]
demonstrate that it is more difficult for the same size of model to fit multi-lingual datasets since with limited model capacity it needs to learn more diverse information while mono-lingual models can be more concentrated.
However, although the validation loss of mono-lingual models is generally lower, from Section 4.2, we observe that in terms of execution performance pass @
k, multi-lingual models actually outperform mono-lingual ones especially when model sizes go beyond 672M.
In fact, as model size increases, the improvement of multi-lingual models over mono-lingual models get more and more significant.
The reason could be that although models get distracted to fit multiple languages, the knowledge sharing across different languages helps model to learn better in solving problems.
For example, similar tasks might exist in different programming languages, hence models are easier to learn to transfer from one language to another.
And larger models have better capability in knowledge sharing/transfer learning, with the evidence that the zero-shot learning performance of multilingual models on unseen programming languages get significantly better than mono-lingual ones as model sizes increases.
O D ATASET C ONVERSION F RAMEWORK
In this section, we describe a dataset conversion framework that transforms an execution-based evaluation in one programming language to another.
In particular, we focus on a function completion format of execution-based evaluation as shown in Figure 2.
Each problem in a function completion dataset consists of a prompt , a test statement , and a canonical solution .
The prompt contains a function signature along with a docstring describing the desired functionality of the code.
The canonical solution is an example of a function body that fulfills such functionality, usually written by human annotators.
Given a candidate function body generated by the model, we can test whether the corresponding function is correct by executing the test statement against the candidate function.
To construct a evaluation dataset for function completion in a new language, we recognize that it is sufficient to convert only the prompts and the test statements (Section O.1).
That is, we do not need to transform the canonical solutions, since they are simply examples and are not used to measure correctness in the test-based framework.
This key feature of a test-based evaluation makes it possible to perform mapping of an evaluation set from one language to many others by static analyses, as outline below.
For other code-related evaluations that require access to canonical solutions, we synthesize solutions by generating many code versions based on the converted prompt and use our converted test statement to filter for correctness (Section O.1.11).
O. L ANGUAGE C ONVERSION OF P ROMPTS AND T EST C ASES O.1. F ORMAT CHOICE
The purpose of this work is to build datasets that allow us to measure multi-lingual code generation abilities.
The function completion format helps steer the model to generate code in a specific language since the prompt consists of a partial function that has already been started, i.e. a function signature.
This is in contrast to other formats such as that of the original MBPP where the prompt does not consist of a function signature, but contains more implicit information such as assert statements, example function calls, and a description such as “Write code in Python” (see Appendix
R.1.1 for examples).
Compared to other formats, the execution-based function completion aligns well with how an ML-driven model would perform code suggestion in a typical coding environment.
Therefore, we process our converted datasets and the original datasets (MBPP, MathQA) to be of this format, except for the original HumanEval dataset whose format is already consistent.
O.1. I NFERRING ARGUMENT AND RETURN TYPES
This step is applicable for statically typed target languages such as Java, C#, etc.
The process starts from inferring the types of function arguments, which can be done by inspecting the argument values.
We perform mapping of types from Python to types in a target language; for instance, to convert to Java, we map list → ArrayList or dict → HashMap .
Values for different test cases can have different types, therefore we infer the common superclass of all observed types for each argument.
Since there can also be many levels of types, due to containers such as list or sets, we recursively infer the types among each level to be consistent.
For example, “list of list” and “list of object” has a common type of “list of object”.
The return type is inferred via expected return values in the test cases.
O.1. S UPPORTED T YPES OF O BJECT C ONVERSION
Our conversion framework depends on the structure of basic programming problems which involve object types of the following:
• Integer or long version of integer • Float or double • Boolean • String.
We assume any string of single character is also of type string for the purpose of conversion.
• None.
This depends whether the target language also supports None/null/nil types.
• List.
Tuples in Python are also converted to list in all languages.
• Dictionary • Set
For any container type, we recursively perform object conversion for all nested structure within the container.
O.1. C ONSTRUCTING REPRESENTATIONS OF CODE OBJECTS
We convert the argument and return values from Python to a target language by generating strings that represent the target language’s objects.
For example, the object [1,2,3] in Python is converted to Arrays.asList(1, 2, 3) in Java, as shown in Figure 2.
We recursively construct container elements for any nested structures.
O.1. C ONVERTING TEST STATEMENT
We construct objects for function argument inputs, using above information regarding constructed objects and types, as well as expected output.
We build the test statement in a target language using appropriate assertion to match the returned value with the expected output.
We perform deep object comparison with an appropriate comparator for each language.
O.1. P ROMPT CONSTRUCTION
We ensure that the converted function, argument, and class names are stylistically appropriate for each language, e.g. camel case versus snake case, etc.
We construct function call examples in the docstring to look representative without being too verbose, e.g., we use [1,2,3] in Java’s docstring to represent a list, instead of an actual ArrayList .
We avoid using language-reserve words for variable names such as end for Ruby or char for Java or C++ and escape certain substrings that are keywords such as /* or // .
We also deal with all formats in the prompt with great care.
For instance, docstrings for Java and
JavaScript are to be before the function signature, following the convention.
For Java, this is crucial, otherwise it would be too out of distribution and the model would not generate anything, if docstring is below function signature.
O.1. D OCSTRING AND NATURAL LANGUAGE CONVERSION
The natural language statements for datasets such as MBPP can contain python-specific statements that might not be applicable to Java or Javascript such as “Write a function in Python to ...” or “... if the object does not exist, return None”.
We substitute ”Python” with the target language name, ”None” as appropriate null values.
O.1. V ALIDATION
We validate that converted objects, test statements and function signatures parse and/or compile with respect to each language.
O.1. Q UALITY C HECK VIA R EVIEWERS
To gauge the quality of our conversion, we also request annotators to manually review the converted programming problems in sample languages, namely, Java and JavaScript.
We ask language experts to identify issues with converted examples consisting of natural language statement, test cases, and function signature and use this process to help iteratively improve our conversion algorithm.
For the final review, annotators have not found issues specifically related to language conversion, but observed ambiguity in some cases attributed to the original dataset.
In the future, any updates to the original datasets can propagated to all converted languages programmatically.
We provide the detailed analysis of evaluation by annotators in Appendix Q.
O.1. C OMPARISON TO OFF - THE - SHELF ML TRANSLATOR
We find ML translators to be insufficient to perform the dataset conversion, due to limited support for language pairs, restriction on format, as well as transformation errors related to types and object construction.
In contrast, our framework can convert data to many target languages and does not have non-deterministic errors related to type inference or object mapping.
We provide further discussion and examples in Appendix O.2.
O.1. S YNTHETIC CANONICAL SOLUTIONS
The availability of canonical solutions in each converted language can open up the possibilities to perform other types of evaluation.
To generate such synthetic solutions for each language, we sample up to 10 , 000 versions of code per problem and filter them for correctness with our converted tests.
In order to generate at least one correct solution for as many problems as possible, we use both the function completion and zero-shot translation settings (see Section 4.3) where we prepend the Python solution, provided in the original datasets by human annotators, to the beginning of the function signature prompt.
With high-temperature sampling, we are able to generate correct solutions for a large portion of all problems, with 96% coverage for JavaScript, 93% for Java, and even on an out-of-domain language such as PHP with 93% coverage (more details in Appendix P).
O. P OTENTIAL U SE OF T RANSCODER FOR D ATASET C ONSTRUCTION
We conduct preliminary experiments using a publicly available code translation model Transcoder (Lachaux et al., 2020b) to perform dataset conversion.
Overall, there are two main limitations of this approach.
First, these models typically support a limited number of language pairs, which means that we would not be able to perform conversion to 10+ languages like with our proposed framework.
Second, we find that there are some common errors associated with type inference, for instance, when the return type should be boolean , the translation model can predict int as a return type.
These types of error cause false negatives and can impact overall quality of the converted datasets.
In contrast, we do not have these types of errors in our conversion framework due to the static analysis implementation.
In particular, Transcoder (Lachaux et al., 2020b) supports Python, Java, and C++.
In this setup, we use a complete function in Python as an input prompt.
The transcoder model then generates a complete function in Java and C++.
Here, we are interested in whether the model is able to translate function signatures that capture necessary information.
Example 1 While the model seems able to translate the function signature, the function name for
Java and C++ appear to be in snake case, which is not the standard for these languages.
[Figure]
Example
The model seems to adapt the function name to be entirely different, i.e., is_undulating to isAbundulating or isSkundulating .
[Figure]
[Table]
is
[Table]
:if (len(n) <= 2):return Falsefor i in range(2, len(n)):if (n[i - 2] !
= n[i]):return Falsereturn True 10 -------------------- Translation in Java -------------------
[Figure]
Example 3 Incorrect type inference where a nested list nestedList is Python is mapped to a string in C++, or simply a flat list of strings in Java.
[Figure]
Example 4 A list test_list is incorrectly inferred to have a type string in C++.
[Figure]
in Python ----------------------- 3 def list_to_float(test_list):res =
[]for tup in test_list:temp = []for ele in tup:if ele.isalpha():temp.append(ele)else:temp.append(float(ele))res.append((temp[0],temp[1]))return (str(res)) 14 -------------------- Translation in Java -------------------
15 public static String listToDouble ( ArrayList < String > testList ) { ..
Translation in C++ -------------------- 20 string listToDouble ( string testList ) { ..
P S YNTHETIC C ANONICAL S OLUTIONS P. M ULTI - STAGE DATA BOOTSTRAPPING
We combine solutions for (1) normal function completion or with few-shot prompting if the language is out-of-domain and (2) translation settings.
This is because these different generation modes can synthetic correct solutions for different problems, according to our evaluation analyses in Section 4.2.
We perform data generation in multiple stages.
We first sample n = 100 samples for all cases, after which we sample for n = 1000 cases for the problems where we have not found at least one correct solution.
The last step contains uses n = 10000 samples.
We use temperature 0 . 8 , 1 . 0 and 1 . 2 respectively.
P. D ISCUSSION : G ROUND TRUTH ASSUMPTIONS OF TEST CASES
Our synthetic generation of canonical solutions make heavy use of test cases to filter whether each code is correct or not.
The process implicitly assumes that the test cases act as a ground truth verifier that provides necessary and sufficient conditions for the correctness of each task’s functionality.
Such assumptions might not hold if the test cases are not thoroughly written in the original dataset.
In fact, false positives of execution-based datasets can typically occur as indicated in several previous work (Li et al., 2022a).
Our framework do not aim to inject additional knowledge per each test case but merely acts to translate a task in one language to another, while carrying the information captured in test cases of the original dataset to the corresponding datasets in other languages.
Any corrections to the benchmark shall be done upstream in the original dataset, and can easily propagate to the rest of the converted datasets, since the conversion process for prompts and test cases is purely programmatic.
The usefulness of the conversion framework lies in the automated conversion into many languages which can be repeatedly done if the test cases in original datasets are updated, or new tasks are added.
This helps reduce human effort to perform such manual translation of a dataset in one language to many others.
One step that can be done to improve thoroughness of the test cases is to use the provided canonical solutions in the original dataset to synthetically generate additional test cases, with the hope that additional test cases can provide test coverage for the functionality.
However, this proposal also relies heavily on the canonical solution as the ground truth that captures the true functionality of the task, which might not necessarily be true since during the annotation process, annotators might write canonical solutions that are only partially correct but pass the specified test cases.
Therefore, we leave this investigation as future work.
Q Q UALITY C HECK OF C ONVERTED D ATASETS
During the manual review process, we provide the converted programming problem in Java and
Javascript to annotators and ask them to check if the problem is correct and clear.
Below we categorize the issues identified by annotators, with examples.
Almost all of these cases can be attributed to the source dataset we use for conversion.
On the other hand, the conversion process itself does not introduce additional errors.
For future work, we plan to thoroughly check for errors in the original MBPP.
Any changes from there can be easily propagated to the converted datasets due to the automatic conversion.
• Natural language statement is ambiguous.
[Figure]
[Table]
[Figure]
[Table]
• Spelling mistake.
[Table]
• One or more test cases are wrong.
[Figure]
[Table]
[Figure] [Figure]
[Figure]
[Table]
[Figure]
[Table] [Table]
[Table] [Table]
R D ATASETS R. MBXP
Below, we show examples of the samples from the original dataset as well as the converted dataset.
[Table]
Note that we convert the original MBPP dataset (Austin et al., 2021) which has a slightly different format into HumanEval format (Chen et al., 2021) with function signature and docstring, as shown below.
The use of function signature and docstring in the formatted MBPP makes it consistent with the converted datasets in all other languages.
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table] [Table]
] + cost.get(i).get(j);} else if (j == 0) {dp[i][j] = dp[i - 1][j] + cost.get(i).get(j);} else {dp[i][j] = Math.min(dp[i - 1][j], dp[i][j - 1])
+ cost.get(i).get(j);}}}return dp[m - 1][n - 1];} 37 } 38 // ---------------------
TEST --------------------- 39 class Main { public static boolean compare(Object obj1, Object obj2) {return obj1.equals(obj2);}
[Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
R.1.
MBJSP: J AVA S CRIPT 1 // ---------------------
MBJSP/1:
PROMPT ---------------------
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Figure]
* >
[Figure] [Figure]
[Figure] [Figure]
[Figure]
*10 */ 11 function minCost(cost, m, n) { 12 // ---------------------
COMPLETION (model generated)
---------------------const dp = new Array(m + 1).fill(0).map(() => new Array(n + 1).fill(0));for (let i = 1; i <= m; i++) {for (let j = 1; j <= n; j++) {dp[i][j] = cost[i - 1][j - 1] + Math.min(dp[i - 1][j], dp[i][j - 1]);}}return dp[m][n]; 20 } 2122 // ---------------------
TEST ---------------------
[Table]
_ = require("lodash") 2425 function compare(object1, object2){return _.isEqual(object1, object2) 27 } 2829 var
[Table]
=
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
else
[Table]
throw
[Table]
at
[Table]
Value
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table]
at
[Table]
Value = ’
+
[Table] [Table]
(x
[Table]
R.1. MBTSP: T YPE S CRIPT
[Table]
[Table] [Table]
j
[Table] [Table]
j
[Table]
n
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
1, n {tc[i][j] :=
min(tc[i-1][j-1], tc[i-1][j], tc[i][j-1]) + cost[i][j]}}return tc[m][n] 31 } 32 // ---------------------
TEST --------------------- 33 func deepCompare(a interface{}, b interface{}) bool { aJSON, err := json.
Marshal(a)if err !
= nil {panic(err)}bJSON, err := json.
Marshal(b)if err !
= nil {panic(err)}var a2 interface{
[Table]
var
[Table]
interface{
[Table]
err = json.
[Table] [Table]
if err !
= nil {
[Table] [Table]
err = json.
Unmarshal
[Table]
, &b2)if err !
= nil {panic(err
[Table]
}result := reflect.
DeepEqual(a2, b2)if result {return true} else {panic(fmt.
Sprintf("%
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table]
MBCSP
C
MBCSP/1:
[Table]
--------------------- 2 using System; 3 using
[Table]
.Collections.
Generic; 4 using
[Table]
.Linq;
[Table]
.Text.
RegularExpressions;
[Table]
R.1.
MBPHP: PHP 1 // ---------------------
MBPHP/
[Table]
PROMPT ---------------------
[Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
return $min
[Table] [Table]
R.1.
MBRBP: R UBY
[Table]
if cost[m][n
[Table]
=
[Table]
:return 0if m =
= 0 and n == 0:return cost[m][n]if m =
= 0:return cost[m][n] + min_cost(cost, m, n - 1)if n == 0:return cost[m][n] + min_cost(cost, m - 1, n)return cost[m][n] + min(min_cost(cost, m - 1, n), min_cost(cost, m, n - 1)) 24 end 25 # ---------------------
TEST --------------------- 26 x = min_cost([[1, 2, 3], [4, 8, 2], [1, 5, 3]], 2, 2) 27 if x != 8raise StandardError, "Error at test case 1" 29 end 30 x = min_cost([[2, 3, 4], [5, 9, 3], [2, 6, 4]], 2, 2) 31 if x != raise StandardError, "Error at test case 2" 33 end 34 x = min_cost([[3, 4, 5], [6, 10, 4], [3, 7, 5]], 2, 2) 35 if x !
= raise StandardError, "Error at test
[Table]
R.1.
MBKP:
K OTLIN
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table]
R.1. MBCPP: C++
[Table]
2) 13 *14 */ 15 int minCost(vector<
vector<int>> cost, int m, int n) {17 // ---------------------
COMPLETION (model generated) ---------------------vector<int> dp(m + 1, n + 1);dp[0][0] = 0;for
(int i = 1; i <= m; i++) {dp[i][0] = dp[i - 1][0]
+ cost[i - 1][0];}for
(int j = 1; j <= n; j++) {dp[0][j] =
dp[0][j - 1] + cost[0][j - 1];}for (int i = 1; i <= m; i++) {for (int j = 1; j <= n; j++) {dp[i][j] = cost[i - 1][j - 1] + Math.min(dp[i - 1][j], dp[i][j - 1]);}}return dp[m][n];
TEST --------------------- 3536 template <typename T> bool compare(T a, T b){return a == b; 38 } 3940 int main(int argc, char* argv[]) {int x0 =
minCost(vector<vector<int>>{{1, 2, 3}, {4, 8, 2}, {1, 5, 3}}, 2, 2);if (!
(compare(x0, 8)))
{throw runtime_error("Exception -- test case 0 did not pass.");} int x1 =
minCost(vector<vector<int>>{{2, 3, 4}, {5, 9, 3}, {2, 6, 4}}, 2, 2);if (!(compare(x1, 12))) {throw runtime_error("Exception -- test case 1 did not pass.");} int x2 =
minCost(vector<vector<int>>{{3, 4, 5}, {6, 10, 4}, {3, 7, 5}}, 2, 2);if (!
(compare(x2, 16)))
{throw runtime_error("Exception -- test case 2 did not pass.");} return 0; 57 }
R.1.
MBPLP: P ERL 1 # ---------------------
MBPLP/1:
PROMPT --------------------- 2 #
You are an expert Perl programmer, and here is your task.
3 # Write a function to find the minimum cost path to reach (m, n) from (0, 0) for the given cost matrix cost[]
[Table]
while ($i < $m and $j < $n) {if ($cost[$i][$j] < $min) {$min = $cost[$i][$j];$min_i = $i;$min_j = $j;}if ($i < $m - 1) {$i += 1;} else {$i = 0;$j += 1;}}$path = $min_i + 1;
[Table] [Table]
[Table] [Table]
[Table] [Table]
R.1.
MBSCP:
S CALA
[Table]
[Figure]
[Table] [Table]
[Table]
R.1. MBSWP: S WIFT
MBSWP/1:
PROMPT ---------------------
[Table] [Table]
[Figure]
[Table]
R. M ULTI - LINGUAL H UMAN E VAL
HumanEval contains 164 cases, most of which are compatible with our conversion framework.
For some cases where the tests are not explicit, such as using Python for loop to iterate over many test cases, we expand them out explicitly to make it compatible with the conversion framework.
For instance, the test statement below
[Table]
is expanded to
There are some cases that we filtered out such as cases that involve a user defined function.
In total, we keep 161 out of 164 cases.
We format of multi-lingual HumanEval are similar to that of MBXP in each language; therefore, we skip the display of examples in this section for brevity.
R.
[Table]
By extending MathQA-python datasets Austin et al.
(2021) for other programming languages, we obtained MathQA-Java and MathQA-JavaScript, for the purpose of evaluating the ability of the models to reason and synthesize code from more complex text, under multiple languages.
The original MathQA-python problem contains a short text (which describes a mathematical question), an answer (usually a real number) and a canonical solution in Python.
Based on this, to build a version in a different language, we perform following two transformation steps:
• Convert MathQA-Python problem into our canonical MBXP format (Section 3).
Specifically, we construct a unified function signature, ie.
def problem(): , followed by a docstring, which is equivalent to the short text of the original MathQA-Python.
Additionally, a single test case can be generated based on the given answer, ie.
assert problem() =
= answer .
• Obtain prompts and test cases in another language for execution-based evaluation using our proposed rule-based conversion framework.
• For the conversion framework outlined in Section 3, we emphasize that we handle floating point comparing numbers to be within ϵ = 1 e − 8 instead of exact comparison.
This handling is suitable for floating points and helps avoid potential false negatives.
It is also compatible with all conversions in other datasets since it is handled within the abstract compare function in each target language.
Below, we show converted examples after the first step (including Python prompts, the canonical solution and a single test case) and its counterparts for Java and JavaScript generated from the second step.
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table] [Table]
[Table] [Table]
[Table]
if (obj1 == null && obj2 == null){
[Table]
} else if (obj1 == null || obj2 =
= null){return false
[Table]
} else
[Table]
if ((obj1 instanceof Double || obj1 instanceof Float) &&(obj2 instanceof Double || obj2 instanceof Float)){if (
obj1 instanceof Float){obj1 = ((Float) obj1).doubleValue();
[Table]