L EVERAGING A UTOMATED U NIT T ESTS FOR U NSUPERVISED C ODE T RANSLATION
Anonymous authors
Paper under double-blind review
A BSTRACT
With little to no parallel data available for programming languages, unsupervised methods are well-suited to source code translation.
However, the majority of unsupervised machine translation approaches rely on back-translation, a method developed in the context of natural language translation and one that inherently involves training on noisy inputs.
Unfortunately, source code is highly sensitive to small changes; a single token can result in compilation failures or erroneous programs, unlike natural languages where small inaccuracies may not change the meaning of a sentence.
To address this issue, we propose to leverage an automated unit-testing system to ﬁlter out invalid translations, thereby creating a fully tested parallel corpus.
We found that ﬁne-tuning an unsupervised model with this ﬁltered data set signiﬁcantly reduces the noise in the translations so-generated, comfortably outperforming the state-of-the-art for all language pairs studied.
In particular, for
Java → Python and Python → C++ we outperform the best previous methods by more than 16% and 24% respectively, reducing the error rate by more than 35%.
I NTRODUCTION
Ancient languages such as COBOL still underpin much of the ﬁnancial industry and government services.
Their outdated structures and thinning developer bases induce costs and severely slow down development, prompting businesses to modernize their codebases.
For instance, the Commonwealth
Bank of Australia spent around $750 million over 5 years to migrate its COBOL codebase to a more recent language.
More generally, most large companies own code written in several programming languages, which can hinder interoperability and make programmers less efﬁcient.
Automatic translation systems could make codebase migrations faster and cheaper, and help programmers learn new languages or understand existing code.
Systems to automatically translate between programming languages with approximately the same level of abstraction are called transpilers or sourceto-source compilers.
They need to be distinguished from compilers which translate source code to a lower-level language. The particularities of some languages allow the creation of very successful rule-based transpilers for a few language pairs (e.g. Java → Scala, CoffeeScript → JavaScript). Methods leveraging veriﬁed lifting (Kamil et al., 2016), which offer formal guarantees, can signiﬁcantly speedup some pre-deﬁned code fragments (Ahmad & Cheung, 2016; Ahmad et al., 2019). However, source-to-source translation for arbitrary programming languages is still an open problem.
Rule-based systems are commonly used, but they are never exhaustive due to the considerable number of translation rules that should be written to translate every function and object from every standard library.
Unlike in natural languages, there is little to no parallel data available for source code, making it impossible to train standard machine translation models.
Recently, TransCoder (Roziere et al., 2020) showed that unsupervised methods can be used to translate source code.
However, it is trained without any supervised signal and only learns the semantics of tokens from their contexts.
As shown in Figure 1, it can confuse tokens that have different semantics in different languages, for instance the ﬂoat division in Python and integer division in C++ and Java which use the token / or more subtle operator priority differences (e.g. Java prioritizes =
= over & unlike Python and C++).
While small inaccuracies often merely hinder comprehension in natural languages, they often make the entire translation erroneous in the context of programming languages.
TransCoder leverages back-translation (Sennrich et al., 2015), an effective data-augmentation scheme where the model translates source sequences to generate training data for the target-to- [Figure] source direction, and vice versa.
Although being highly effective in low-resource translation, backtranslation also has issues, as the model is trained on potentially invalid input-output pairs.
Neural machine translation models being highly sensitive to input noise (Belinkov & Bisk, 2018; Khayrallah & Koehn, 2018), this can severely deteriorate the performance.
Fortunately, many programming languages come with relatively mature tools and technologies for automated test data generation.
In this paper, we propose to leverage these tools to guide the translation process, weeding out unsuccessful translations, thereby increasing the overall conﬁdence in the machine translation process.
The topic of automated test data generation has been active for over three decades in the software engineering research community (Myers, 1979; Miller & Spooner, 1976).
There are now many existing mature tools for test data generation, both open source research tools (Fraser & Arcuri, 2011; Lakhotia et al., 2013; Cadar et al., 2008), and production testing systems (Alshahwan et al., 2018; Tillmann et al., 2014).
Because of its pivotal impact on practical software engineering, automated testing remains a highly active research area (Anand et al., 2013), with the result that future automated testing advances will lead to ongoing improvement in automated translation.
We use one such open source automated test generation tool, EvoSuite (Fraser & Arcuri, 2011), in this paper.
EvoSuite is a well-established test generation tool for Java which uses coverage metrics (Chekam et al., 2017) and mutation scores (Jia & Harman, 2011) to generate high-quality tests.
It has been widely used in the Software Testing research literature for test data generation although it has not, hitherto, been used as part of an automated translation approach guided by mutation, the topic of the present paper.
More generally, software testing tools have been largely ignored by the machine learning community.
In this paper, we propose to use automatically created unit tests to guide unsupervised translation models for programming languages.
More precisely, we create unit tests automatically for a large number of functions from the source dataset.
Since the unit tests are composed of simple inputs and asserts, they can easily be translated to semantically equivalent tests in the target languages using simple scripts.
Using our unit-tests and a pre-trained unsupervised translation model, we create parallel datasets by translating functions and selecting the translations that have the same semantics as the original function for the tested inputs.
Overall, we make the following contributions:
• We introduce a novel approach, TransCoder-ST (for Self-Trained), that leverages an automated unit test generation pipeline to ﬁlter out invalid translations and reduce the noise coming from the back-translation process in unsupervised machine translation.
• We present two implementations of this approach (online and ofﬂine), and show that it signiﬁcantly outperforms the previous state of the art in code translation on all the language pairs we considered.
In particular, we improve the state of the art for translating between Java, Python and C++ by an average of 12.6% Computational Accuracy (CA@1), corresponding to an average relative improvement of 25.5%.
For Python → C++, we improve the CA@1 by 24%, reducing the error rate by 35.7% compared to previous models.
• We generate multilingual unit tests for hundreds of thousands of Java functions and create a large parallel dataset of 135,000 parallel functions between Java, Python, and C++.
• Our method is completely unsupervised and could easily be generalized to other programming languages and unit test creation tools.
R ELATED WORK
Unit Test Generation.
Software testing is challenging due to the large number of possibilities to be tested, and the inherent cost of covering reasonable representative sample (Myers, 1979).
When test design is performed by humans, the cost can be prohibitive so much research over the last three decades has focused on automating the process of test generation (Anand et al., 2013).
Although automated test generation has been studied since the mid-1970s (Miller & Spooner, 1976), it was only in the last decade that industrial-strength tools have become widely available.
There are now several test data generation tools for languages, including C (Cadar et al., 2008; Lakhotia et al., 2013) and Java (Fraser & Arcuri, 2011).
Popular test data generation techniques include symbolic execution of the code (Cadar & Sen, 2013), dynamic execution guided by a ﬁtness function (Harman et al., 2015), and hybrids of these two techniques (Baars et al., 2011).
Recently, neural networks have also been used successfully to generate unit tests (Tufano et al., 2020).
One of the most well-established and widely-used open source tools for test data generation is the
EvoSuite system (Fraser & Arcuri, 2011).
EvoSuite uses search based software engineering (SBSE)
(Harman et al., 2012) to generate test cases.
Like all SBSE techniques, EvoSuite is guided by ﬁtness functions, in this case aimed at capturing the test suite’s coverage of the code being tested.
We use
EvoSuite in our work for three reasons: it is publicly available in open source (thereby facilitating replication), it is under current active development (thereby supporting future work), and it is widely used by other researchers (thereby enabling interoperability).
The test framework can be considered as a parameter in our overall approach and could be substituted with another.
In order to assess the effectiveness of the test suites generated, we use mutation testing, a topic also widely-studied since the 1970s (DeMillo et al., 1978).
A mutant is a version of the program into which a fault is deliberately inserted, thereby assessing the test suite’s ability to detect this fault (Jia & Harman, 2011).
For a given set of mutants and a test suite, the mutation score is deﬁned to be the proportion of mutants for which the test suite distinguishes the behavior of the mutant from that of the original program.
The mutant score is thus a proxy for the fault-revealing power of the test suite on a set of simulated faults (the mutants).
Mutation scores have been empirically demonstrated to be correlated to real fault revelation (Chekam et al., 2017), motivating our adoption of this approach.
Machine Learning for Programming Languages.
In recent years, deep learning methods have been used to tackle various tasks in software engineering, with a particular interest in bug detection and repair (Wang et al., 2018; Chen et al., 2019; Allamanis et al., 2018; Tarlow et al., 2020; Murali et al., 2021; Dinella et al., 2020; Yasunaga & Liang, 2020; Tufano et al., 2019; Drain et al., 2021) and code completion (Li et al., 2018; Liu et al., 2020; Kim et al., 2021; Svyatkovskiy et al., 2021).
Unsupervised pre-training methods for code based on BERT (Kanade et al., 2020;
Feng et al., 2020), BART (Ahmad et al., 2021) or other objectives tailored to source code (Guo et al., 2020; Roziere et al., 2021) have shown strong results on benchmarks such as CodeXGLUE (Lu et al., 2021).
Recently, Hendrycks et al.
(2021) evaluated the competence of several language models for solving coding challenges.
Chen et al.
(2021) trained a a large model to generate programs from docstrings and are able to solve 28.8% of the problems in their HumanEval dataset.
Austin et al. (2021) also evaluated the capabilities of large language models for generating code solving problem statements written in natural language.
The goal of code translation is also to generate code solving a speciﬁc problem, but the input (code written in a different language) is more precise and often more concise.
Several studies used statistical methods to translate between programming languages.
Early methods extracted parallel datasets and trained phrasebased models to translate between C# and Java (Nguyen et al., 2013; Karaivanov et al., 2014) or from Python 2 to Python 3 (Aggarwal et al., 2015).
Later, Chen et al.
(2018) proposed a tree-to-tree neural network to translate between CoffeeScript and JavaScript and between C# and Java using the dataset created by Nguyen et al.
However, these approaches are limited to a few language pairs for which small parallel datasets were created manually (e.g. C#-Java) or can be created with rule-based tools (e.g. Python 2-Python 3 and CoffeeScript-JavaScript).
Instead, Roziere et al.
(2020) proposed TransCoder, an unsupervised model that leverages the principles of unsupervised machine translation (Lample et al., 2018), to translate between Python, Java and C++.
They showed that their method outperforms well-established rule-based baselines, does not require any parallel data or expert knowledge, and can easily be generalized to other languages.
They pre-trained their model with the Masked Language Modeling (MLM) objective of Devlin et al.
(2018), and trained it with the denoising auto-encoding (DAE) (Vincent et al., 2008) and the back-translation (BT) (Sennrich et al., 2015) objectives.
Later, Roziere et al.
(2021) showed that augmenting MLM with a deobfuscation objective (dubbed DOBF) can substantially improve the performance of TransCoder.
In the rest of the paper, we will refer to their model as DOBF.
Even though unsupervised methods can be trained on large amounts of data, they sometimes lack the signal needed to differentiate between semantically different tokens that often occur in similar contexts (see Figure 1).
There is a need for a method providing supervised signal directly related to the semantics of the code without manually crafted parallel datasets.
M ETHOD 3. P ARALLEL DATA CREATION
Parallel unit test generation: We use EvoSuite to automatically generate unit tests for Java functions.
EvoSuite is a well-established open source tool for automated test generation in Java, which is still under active development and frequently used.
It is designed for Java programs but its searchbased technique is general and could be used for any programming language.
Unit tests can be thought of as lists of inputs and asserts testing the semantics of a program (e.g., the output of the function, the side effects on its arguments such as sorting the input list).
EvoSuite uses evolutionary methods to derive tests that maximize criteria such as code coverage or mutation score.
During its search, each candidate solution in EvoSuite is a test input.
The candidate inputs are evolved using crossover and mutation, and ﬁltered by a ﬁtness function (e.g., mutation score).
With each generation the ﬁtness improves until it reaches a plateau or the budget is exhausted.
The ﬁnal test inputs are wrapped up as test cases.
Each program is associated to a test suite containing a series of test cases.
Figure 3 shows an example of a test case generated by EvoSuite.
[Figure]
Parallel test suites selection: Some test suites created by EvoSuite only cover a few parts of the semantics of functions.
We only trust the translations veriﬁed by test suites which examine the function semantics thoroughly.
We use the mutation score, which is the most effective test assessment metric in the literature (Jia & Harman, 2011), to pick out these test suites.
The mutation score is computed through mutation testing, in which mutants (i.e., program variants with syntactic changes) are generated from the original program based on a set of transformation rules (more details in Appendix A.2).
A mutant is said to be killed if at least one test from the test suite has different results on the mutant and the original program.
Otherwise, the mutant is said to survive.
The mutation score is the ratio of killed mutants.
A test suite with a higher mutation score checks the code semantics more thoroughly.
We adopt a strict strategy in test suite selection: we keep only the Unit test suites with a mutation score larger than 90% for building the parallel dataset.
Parallel dataset building: The generated test suites can be used to test the semantics of programs written in any programming language as long as there is a clear mapping between the types of the output and parameters in the original language and the language of the translated unit tests.
We transform the generated Java tests into C++ and Python tests with identical inputs and expected outputs and side effects (i.e., assertions).
In practice, we selected the Java functions which can be compiled and run in isolation and with simple output and parameter types.
These types are the
Java primitive types (e.g. int , long , bool , float ...), standard data types (e.g. Integer ,
Double , String ...), array and List or ArrayList types of elements of supported types (e.g. double[] , List<
Integer> ...).
We use the best unsupervised translation models available for Java to Python and Java to C++ translation, namely TransCoder (Roziere et al., 2020) for Java to C++ and DOBF (Roziere et al., 2021) for Java to Python.
For each Java function, we generate 20 Python and C++ translations with beam search and select the ﬁrst element in the beam that passes the unit tests.
The created tests are executed against the translated functions.
If all the tests pass, the Python and C++ functions have the same semantics assessed by the generated tests.
Our method is illustrated in Figure 2.
3. T RAINING METHOD
Our parallel data generation method relies on a pre-existing model to translate from Java to Python and C++.
There is little parallel data for these tasks and the best performing published models are unsupervised.
TransCoder (Roziere et al., 2020) is trained using the MLM, denoising and backtranslation objectives and is able to translate between Java, C++ and Python.
DOBF (Roziere et al., 2021) provides clear improvements over TransCoder for translating between Java and Python but was not trained on C++.
Therefore, we use DOBF to translate from Java to Python and TransCoder to translate from Java to C++.
When ﬁne-tuning, we also reload these models.
For DOBF, we initialize the C++ language embeddings with those of Java.
The parallel examples we generate can be used to improve the performance of pre-existing translation models.
Since the number of examples we generate also depends on the performance of the translation model, it creates a positive feedback loop where improving the model allows to improve the parallel dataset which in turn can be used to improve the model again.
We propose ofﬂine and online approaches to use our method to maximize the unsupervised translation performance.
[Table]
Ofﬂine training.
With the ofﬂine training method, we use the method described in Section 3.
to create parallel Java ↔ Python, Java ↔ C++ and Python ↔ C++ datasets using every input Java function we selected.
For the ﬁrst iteration, we ﬁne-tune the model on these parallel examples until convergence.
We can iterate this process by selecting the best checkpoints for Java →
Python and
Java →
C++ using the validation dataset and using them to generate new parallel datasets, which can in turn be used to train a better model.
We iterate this process until convergence, i.e. when we see no signiﬁcant improvements on the validation set.
Online training.
With the online method, we create parallel examples on the ﬂy while training the model.
Compared to the ofﬂine method, it allows to always use the last model to generate new examples and it is much more convenient to automate.
However, this process can be unstable if done naively.
For instance, the model can start over-ﬁtting only a few examples and stop generating anything that passes the unit tests for any other example.
In order to stabilize the training, we follow
Likhomanenko et al.
and implement a cache mechanism storing the previous examples that passed the unit tests.
At each step, the model can either train on parallel functions sampled from the cache or create new parallel functions to add to the cache.
When an example is sampled, we remove it from the cache with a given probability.
The online training allows the model to always beneﬁt from the performance of the latest model and the cache mechanism ensures that the model does not forget the correct examples that it was able to generate at previous time steps.
3. E VALUATION
In the context of natural languages, machine translation models are generally benchmarked against a reference solution using the BLEU score (Koehn, 2009; Bahdanau et al., 2015; Vaswani et al., 2017).
Early studies on source code translation used the same metric to evaluate the quality of the generated functions (Nguyen et al., 2013;
Karaivanov et al., 2014; Aggarwal et al., 2015; MiceliBarone & Sennrich, 2017), or the exact match score which requires the translation to be exactly equal to the ground truth (Chen et al., 2018).
However, these metrics fail to capture the semantics of the code and typically correlate poorly with the correctness of the generated function, prompting the use of new metrics checking if the generated solution passes series of test cases (Kulal et al., 2019; Roziere et al., 2020;
Hendrycks et al., 2021; Chen et al., 2021; Drain et al., 2021).
We evaluate our models on the full validation and test sets of TransCoder.
It contains a few hundreds of parallel functions extracted from GeeksforGeeks along with associated unit tests.
As our TransCoder and DOBF baselines, we evaluate our models with the CA@N metric, which checks if any of the top-N solutions proposed by the model passes all the corresponding unit tests.
This metric can be computed independently of the beam size (as long as the beam size is greater or equal to N).
E XPERIMENTS 4. T RAINING DETAILS
Model architecture.
We use a sequence-to-sequence model with attention composed of an encoder and a decoder model with a transformer architecture (Vaswani et al., 2017).
In order to provide fair comparisons, we use the exact same architecture as TransCoder: an encoder and a decoder of 6 layers each, a hidden dimension of 1024 and 8 attention heads.
We limit the size of the input to 512 tokens.
Roziere et al.
(2021) train models with two different architectures.
For Java ↔ Python, we compare ourselves to the version of DOBF using the same architecture as TransCoder.
We initialize our models with either the best TransCoder checkpoint for Java → C++ or the best DOBF checkpoint for Java → Python with C++ language embeddings initialized with those of Java. [Table]
Datasets.
As TransCoder and DOBF, we use the GitHub public dataset available on Google BigQuery ﬁltered to keep only projects with open-source licenses 1 .
As our unit test creation tool can only be used on Java code, we only use the Java ﬁles and we select only the functions that can be compiled in isolation.
We obtain a dataset containing 333,542 Java functions.
We run EvoSuite with a budget of 20 seconds and a criterion including the line, branch, cbranch and output coverages, as well as the weak and strong mutation scores.
We set the maximum absolute value of integers that can be generated as an input to √ 2 31 − 1 to limit the number of overﬂows.
We manage to obtain high-quality (mutation score > 0.9 and at least two asserts) test cases for 103,488 functions.
See
Figures 3 and 5, 6 in the appendix for examples of selected and ﬁltered out test suites.
Training details.
During the training, we alternate between batches for every source and target language so that language pairs for which we managed to create more parallel examples are not overrepresented in our training batches.
For the online version, we set a cache warm-up parameter to ensure that we always generate new parallel examples if there are less than 500 examples in the cache for any language pair.
Otherwise, we sample from the cache with probability 0 .
5 , or generate new examples, train on them once and put them in the cache also with probability 0 .
The sampled elements are removed from the cache with probability 0 .
3 , so that each element we create is trained on about 4 times in average before being removed from the cache.
We initialize the cache with parallel examples created ofﬂine.
During beam decoding [Table] we compute the score of generated sequences by dividing the sum of token log-probabilities by l α where l is the sequence length.
We found that taking α = 0 .
5 (and penalizing long generations) leads to the best performance on the validation set.
4. R ESULTS AND DISCUSSION
[Table]
[Table]
Results.
In Tables 2 and [Table] we compare the results of our ofﬂine and online training methods with those of TransCoder and DOBF.
DOBF outperforms TransCoder for the Java ↔ Python pair.
We compare our models against the best baseline for each language pair and direction.
Training on the generated parallel examples brings substantial improvements for every language pair, direction, and metric.
Ofﬂine training already provides clear improvements over the baseline after one iteration.
The computational accuracy (CA@1) computed with beam size 10 is higher for every direction and it is substantially higher for the language pairs involving Python.
It allows to reduce the error rate of the best baseline by 22.4% for Java → Python.
In average, it increases the
CA@1 by 7.9% over the best previous models, and reduces the error rate by 17.0%.
In the second iteration, the model is trained on more than 25% more examples for Java ↔ Python and C++ ↔
Python and a few more examples for Java ↔ Python (see Table 1).
It results in improvements in every direction except for C++ →
Java where there is no signiﬁcant improvement and C++
→ Python where we observe a small regression.
The average improvement over the ﬁrst iteration is 1.5% point.
Although the model for the third iteration is trained on more parallel samples, its performance on the test set of TransCoder is actually worse than after the second iteration.
After two iterations, the model learned to generate more samples that pass the unit tests but some of them are actually incompatible with the types of translations expected by TransCoder (e.g. example with overﬂows in
Figure 4), causing the computational accuracy score to go down.
The online self-training method provides further improvements over training on the pseudo-labeled examples ofﬂine.
It outperforms every other method in every case except the second iteration of ofﬂine training for Java → C++.
In average, this model outperforms the baseline by 12.6% points, corresponding to an error rate reduction of 25.5%.
For Python → C++, it improves previous performance by more than 24% points, which corresponds to reducing the error rate by 35.7%.
Examples of avoided errors can be found in Figure 1 and Appendix B. Overall, all our models signiﬁcantly improve previous results.
As shown in Table 3, these improvements are stable across several beam sizes and CA@n metrics.
The CA@20 metric shows that the number of examples for which none of the 20 elements in the beam are correct is reduced by more than 22% with online self-training.
It indicates that, even though we train only on the output of the model, our method does much more than reordering the elements in the beam and allows the model to ﬁnd correct solutions that were not assigned a high probability by the baseline model.
See Table 6 in the appendix for more results.
Ablation study.
The results of our ablation study are shown in Table 4.
Training online with no cache makes the training much less stable.
The model improves at the beginning of training and we can select a few checkpoints where it performs well, but it ends up over-ﬁtting a few examples it generated and the performance drops after a few epochs.
Starting with an empty cache slows down the training and hinders generalization, leading to a clear drop in performance.
We also try removing the minimum mutation score requirement for the model with no initial cache, which leads to even lower scores as the model is trained partly on lower-quality parallel data.
All these models were trained using a self-training beam size of 20 when generating new examples.
Training with greedy decoding is much faster since computing the results for all the 20 elements of the beam is costly.
However, generating new examples with greedy decoding leads to a loss of about two percentage points in average compared to our full model using beams of size 20.
It shows that initializing the cache of the model with beam size 20 is not sufﬁcient and creating new examples with beam search is necessary to reach our best performance.
Our full model provides some improvements over the ablated versions for every language pair and direction, except over the model trained with greedy decoding for Python → C++ translation.
Evaluating with beam size(still returning only the ﬁrst element) leads to some improvements for every language pair.
Limitations.
We found that the unit tests we create with this method are sometimes incompatible with those of the test set of TransCoder, and that the capacity of a model to generate functions that pass these unit tests is not perfectly correlated to its score on the test set.
It raises the deeper issue of deﬁning what constitutes a correct translation.
For instance, most programmers would translate a factorial function implemented with long integers into a factorial function implemented with Python’s integer type.
However, these functions are not semantically equivalent since the Java implementation would return a negative number for the input 21 due to integer overﬂow while the
Python implementation would return 21!
correctly.
The human developers who wrote the parallel functions in the test set of TransCoder often assumed that these functions would only be used on a limited domain where no overﬂow occurs (see Figure 4).
However, the test cases of EvoSuite and TransCoder are not limited to this domain and they sometimes assert different semantics.
By using the test suites from EvoSuite as source of truth, we sometimes train the model to generate translations that are more rigorous but also less natural.
[Figure]
C ONCLUSION
In this paper, we introduced a novel method to grow a parallel corpus for automated code translation, from completely monolingual data.
We leverage multilingual unit tests to ﬁlter good pseudo-labels, improving the model, and in turn the candidate translations.
We show that both ofﬂine and online methods substantially improve the state of the art in unsupervised code translation, with an average improvement of 12.6% points in computational accuracy, and up to 24% points for Python → C++, corresponding to translation error rate reductions of 25.5% and 35.7% respectively, without using any unit test generation tool for Python and C++ (exclusively for Java).
Our method would automatically gain from improvements of automatic unit test generation tools.
We could also increase the size of the dataset we generate by using test creation tools written for other languages than Java, or by generating tests with EvoSuite on translated examples.
Similarly, we could also extract the semantics of human-written unit tests found in open-source projects to obtain larger, and possibly higher-quality datasets.
In this paper, we focused on translation correctness and our parallel example validation criterion was only based on semantics.
It could be supplemented with other requirements, such as a speciﬁc code formatting or the output of linters to generate code verifying arbitrary criteria.
Finally, the approach presented in this paper could easily be transferred to natural languages.
Although there is no concept of unit tests in natural language, traditional grammar and syntax checkers could be used to ﬁlter out some incorrect generations, and reduce the noise coming from the back-translation process.
Neural machine translation systems being highly sensitive to noise coming from parallel data (Belinkov & Bisk, 2018; Khayrallah & Koehn, 2018), this may improve the performance in low-resource machine translation signiﬁcantly.
R EPRODUCIBILITY
We made sure to use the same architecture and framework as previous works in source code translation so that our results are comparable (see Section 4.1).
We submit our code with this submission, along with a ReadMe ﬁle detailing clear steps to reproduce our results, including a script to set-up a suitable environment.
We will open-source our code and release our trained models.
Our models were trained using standard hardware (Tesla V100 GPUs) and libraries (e.g. Pytorch, Cuda) for machine-learning research.
E THICAL CONSIDERATIONS
In this paper, we improve source code translation methods.
Our methods could facilitate codebase migrations and interoperability, encouraging companies to move away from ancient programming languages and making software developers more efﬁcient.
Although increased efﬁciency could reduce the number of developers needed to perform a task, its impact on the labor market is unclear as lower costs would also lower the bar for starting new projects and increase the demand for software engineers.
Today, the demand for software engineering skills is high despite (or thanks to) the development of software (e.g. git, IDEs), programming languages (e.g. python), libraries (e.g. pytorch) and methodologies (e.g. continuous deployment) improving the efﬁciency of software developers, and we believe that the development of automatic translation tools would not drastically affect the prospects of software developers either.
In the long term, the migration of codebases written in antiquated programming languages (e.g. COBOL) could negatively impact experts in those languages, who are in particularly high demand at the moment.
However, it would also beneﬁt society by facilitating debugging and updating software still used in most of our ﬁnancial transactions and for many government services, and by increasing the demand for other software engineering skills
A M ULTILINGUAL U NIT T ESTS C REATION A. G ENERATED UNIT TESTS
[Figure]
int pow ( int b, int e) { int r =
[Figure]
while (e > 0) { if ((e & 1) =
= 1) r = r * b; b = b * b; e = e >> 1; } return r; }
[Figure] [Figure]
As discussed in Section 3.1, we only generate unit tests for static functions with selected return and parameter types. It makes it easy to map the types of inputs and outputs in Java to Python or C++ types in the translated unit tests. While most of the unit tests are translated correctly, the translation sometimes fails due to EvoSuite generating test cases expecting exceptions.
Our analysis shows that it happens for about 5.6% of all tests and less than 2% of the tests with high mutation scores. In that case, the candidate translations cannot pass the translated tests and no parallel examples are created. A. M UTATION SCORE
In mutation testing, mutants are programs transformed from the original programs based on a series of syntactic transformation rules called mutation operators.
Mutation testing consists in introducing minor syntactic faults on the code and running the tests against the mutated code.
A strong test suite is expected to detect the code changes by having at least one test failing.
Table 5 shows the examples of mutation operators adopted in EvoSuite when generating mutants (Fraser & Arcuri, 2015).
A mutant is said to be killed by a test case if the output of this test case on the mutant is different from its output on the original program (i.e., the test fails the mutant).
Otherwise, the mutant is said to have survived.
Figure 7 shows an example of a mutant generated by changing the < in the return statement into > .
The test with input (-800, -800, -1), as shown by Figure 3, does not kill this generated mutant, because its outputs on the original program and the mutant are the same.
Mutation score is considered as the most effective criteria in accessing the fault-revealing ability of test suites.
Other criteria, such as code coverage, are weak: they check only whether the test Mutation operator Explanation [Table]
call [Table] [Table] Replace constant operator
[Table]
executes the code, but do not check whether the execution result is correct.
A test suite without any assertions can achieve 100% code coverage, but could not detect any faults.
B T RANSLATION EXAMPLES
[Figure]
[Figure]
) return n; return (n & ~(1<<(k - 1))); } def isNumber(s): for i in range(len(s)): if s[i].isdigit() !
=
True: return False return True static boolean isNumber(String s) { for ( int i = 0; i<s.length(); i++){ if (s.charAt(i) ! = ' ') { return false ; } } return true ; } static boolean isNumber(String s){ for ( int i = 0; i<s.length(); i++){ if (Character.isDigit(s.charAt(i)) ! = true ){ return false ; } } return true ; }
[Figure]
TransCoder Java translation TransCoder-ST Java translation int cassini( int n) { return (n & 1) ? - 1: 1;
[Figure] def cassini(n): return (n & 1) def cassini(n): return - 1 if (n & 1) ! = {+
+}
elseint power( int n) { if (n == 1) return 2; return 2 * power(n - 1); } def power(n): if n == 1: return 2 return 2 ** power(n - 1) def power(n): if n == 1: return 2 return 2 * power(n - 1) [Figure] [Table]
[Figure] [Table]
[Figure] [Table] [Figure] [Table]
[Figure]
new HashSet<Integer>(); for ( int i = 0; i < n; i++) { S.add(arr[i]); } int ans = 0; for ( int i = 0; i < n; i++) { if (S.contains(arr[i] - 1))
{ int j = arr[i]; while (S.contains(j)) { j++; }ans = Math.max(ans, j - arr[i]); } } return ans; } public static int findLongestConseqSubseq( int arr[], int n ){HashSet<Integer> S = new HashSet<>(); for ( int i = 0; i < n; i++) { S.add(arr[i]); } int ans = 0; for ( int i = 0; i < n; i++) { if (S.contains(arr[i] - 1)== false ){ int j = arr[i]; while (S.contains(j)) { j++; }ans = Math.max(ans, j - arr[i]); } } return ans; }
In the ﬁrst example computed the minimum XOR between two elements of an array, TransCoder erroneously translates INT_MAX into
Integer.
MIN_VALUE .
This value is used in similar contexts (i.e. to compute a maximum instead of a minimum) but is inappropriate here.
TransCoder-ST manages to correct this and outputs a function with the right semantics.
In the second example, where the function computes the size of the largest subset of elements of the list that could form a sequence of consecutive integers, TransCoderST manages to translate the semantics of S.find(arr[i]-1)==S.end() appropriately while
TransCoder translates it into its negation.
[Figure]
, manages to instantiate the right type of priority queue.
Input Java function
TransCoder Python Translation
TransCoder-ST translation static int divisorSum( int n) { int sum = 0; for ( int i = 1; i <= n; ++i) sum +=(n / i)
* i; return sum; } def divisor_sum(n): sum =for i in range(1 , n + 1): sum +=(n / i)
* i return sum def divisor_sum(n): sum
=for i in range(1 , n + 1): sum +=(n // i)
* i return sum staticboolean check( int degree[], int n){ int deg_sum = 0; for ( int i = 0; i < n; i ++) { deg_sum += degree[i]; } return (2*(n-1)==deg_sum); } def check(degree, n): deg_sum =for i in range(n): deg_sum += degree[i] return (2**(n-1)==deg_sum)
def check(degree, n): deg_sum =for i in range(n): deg_sum += degree[i] return (2*(n-1)==deg_sum) static int decimalToBinary( int N){ int B_Number = 0; int cnt = 0; while (N != 0) { int rem =
N % 2; double c = Math.pow(10, cnt);
B_Number += rem * c; N /= 2; cnt ++; } return B_Number; } def decimal_to_binary(N):
B_Number =cnt
= 0 while N !
=
: rem =
N % 2 c = pow(10, cnt) B_Number += rem * c N /=cnt += 1 return B_Number def decimal_to_binary(N): B_number =cnt = 0 while
[Figure] [Figure]
C E XTRA R ESULTS C. B EAM REORDERING
We also evaluate a simpler method where we create unit tests for the Java functions in the test dataset and use them to reorder the elements of the beam at test time.
We compute the results of the tests for every proposed C++ or Python translation and prioritize the elements that pass the unit tests.
As shown on Table
[Table]
reordering the elements of the beam at test time when translating from
Java leads only to small improvements compared to the best baseline (up to 1.7% CA@1 for
Java → Python) and the scores of this method are far from those obtained when requiring any of the 10 element of the beam to be correct (i
[Table]
e. CA@10).
It can be explained by the fact that the tests generated by EvoSuite on these functions can have low mutation scores and be insufﬁcient to thoroughly test the semantics of the functions.
Moreover, the tests we create are sometimes incompatible with those of our test set (see Figure 4 for an example)
[Table] [Table]