Data Augmentation for Code Translation with Comparable Corpora and Multiple References (2023)

Abstract: One major challenge of translating code between programming languages is that parallel training data is often limited. To overcome this challenge, we present two data augmentation techniques, one that builds comparable corpora (i.e., code pairs with similar functionality), and another that augments existing parallel data with multiple reference translations. Specifically, we build and analyze multiple types of comparable corpora, including programs generated from natural language documentation using a code generation model. Furthermore, to reduce overfitting to a single reference translation, we automatically generate additional translation references for available parallel data and filter the translations by unit tests, which increases variation in target translations. Experiments show that our data augmentation techniques significantly improve CodeT5 for translation between Java, Python, and C++ by an average of 7.5% Computational Accuracy (CA@1), which verifies the correctness of translations by execution. The code is available at https://github.com/Veronicium/CMTrans.

## introduction code translation is a special type of machine translation that translates between programming languages. it is widely applied in software engineering to migrate a codebase into another programming language. recent code translation models typically follow the pretrain-finetune pipeline, as shown in figure 1 . in pretraining, with denoising objectives such as masked span or identifier prediction @xcite @xcite , the model learns to produce sequences in both languages. when finetuned on parallel data @xcite , which are program pairs in the source and target language that are aligned lineby-line, the model learns functional equivalence: identifying programs with the same functionality, 1 code available at https://github.com/veronicium/cmtrans . either in the same language or between languages. we show an example of parallel data in figure 2 . one major challenge of code translation is that parallel data is typically limited. for instance, the transcoder dataset (roziere et al., 2020) only contains 466 python-c++ pairs. constructing parallel data requires substantial human effort and cannot be easily scaled. with limited fine-tuning examples, it is difficult for a model to learn functional equivalence across programming styles and domains. to mitigate the data scarcity issue, we hypothesize that the functional equivalence between languages can also be learned from comparable corpora, a term we borrow from natural language translation, where it refers to texts on similar topics in different languages @xcite . here, we use it to refer to programs with similar functionality in different languages @xcite . as shown in figure 2 , although the programs paired in a comparable example may have different algorithms and structures (e.g., global code vs. class functions), they are likely to have similar constructs and may even have some lines that can be matched. in this paper, we study what the model learns from comparable corpora by building three types of comparable examples: (1) naturally existing, where we leverage independently-written solutions board.append( str(input()) ) for x in board: if "bb" in x or "ww" in x: boolean @xmath0; scanner @xmath1.in ); for( int @xmath2; @xmath3; i++ ) board[i] = in.next(); for (int @xmath4; @xmath5; i++){ string @xmath6; if (x.contains("bb") || x.contains("ww")){ system.out.println( "no" ); @xmath7; break; } if (!x.equals("bw".repeat(4))) if (!x.equals("wb".repeat(4))){ system.out.println( "no" ); @xmath8; break; } } if (@xmath9.out.println( "yes" ); parallel (java) figure 2: an example of parallel and comparable data. parallel examples are line-by-line aligned. programs in a comparable example may have different algorithms and structures (e.g., global code vs. class in this case), but may still contain lines that can be matched, as highlighted in yellow, green, and blue. of the same coding problem; (2) generated, where we collect programs with docstrings in one language and apply a code generation model to generate programs in another language; and (3) retrieved, where we either retrieve a program's k nearest neighbors (knn) or simply choose a random program in another language. among them, (1) contains cleaner examples, which are guaranteed to be bug-free. (3) covers programs from a larger variety of sources, providing more diverse training signals. in addition to the functional equivalence between languages, the model should also learn the equivalence between different programs in the target language. this is challenging due to limited finetuning data. furthermore, the majority of finetuning data only have one reference translation (e.g., 82.5% in avatar, @xcite , which is likely to cause overfitting to a single translation without fully capturing its semantics. as a result, in this paper, we generate multiple translation references for the finetuning data. specifically, after training on comparable corpora, we finetune a model on the original parallel data, generate multiple translations for each example, and use automatically generated test cases to filter out incorrect translations. by training on different functionally equivalent programs, we reduce overfitting to a single output and improve the modeling of the target language. combining the two techniques, we name our full approach cmtrans, a code translation model trained with comparable corpora and multiple references. extensive experiments show that cm-trans significantly improves codet5 @xcite , which is initialized from the same pretrained checkpoint, for an average of 7.5% computational accuracy (ca@1) over translation between 6 language pairs. cmtrans also significantly outperforms the state-of-the-art method in 5 out of 6 language pairs, while reaching parity on the other one. analyses of our two techniques suggest that: (1) all three types of comparable corpora (including random program pairs) improve the syntax accuracy and perplexity of the translation outputs and lead to better final performance. (2) both naturally existing and generated comparable corpora help the model generate constructs that match the input. the combination of them gives the largest performance gain. ## methodology in this section, we introduce our data augmentation method that trains the model first on comparable corpora, which are either naturally existing, generated, or retrieved (section 3.2) and then on parallel data with additional references generated by our model (section 3.3). ## experiments in this section, we conduct experiments to answer four research questions: (rq1) how do cmtrans and its ablations perform compared against stateof-the-art approaches for code translation? (rq2) what can the model learn from comparable corpora? (rq3) what can the model learn from multiple references? (rq4) how is cmtrans affected by the size of comparable corpora and the number of references? ## conclusion and future work we present two data augmentation techniques for code translation: comparable corpora and multiple references. we study multiple ways of building comparable corpora, such as feeding natural language code documentation to code generation models. additionally, we generate multiple reference translations using our model and filter references using automatically generated unit tests. experiments on translation between 6 language pairs show that our two techniques improve codet5 substantially, by an average of 7.5% ca@1. our analyses further show that after training on comparable corpora, the model learns to generate more fluent code with the same types of constructs. with multiple references, the model learns to generate a larger number of unique correct translations. following our analyses of what the model learns from comparable corpora, future work may conduct a more comprehensive study on what the model learns in code translation. one may also explore combining our data augmentation techniques with data distillation from large language models (llms), where llm may generate data with higher quality, but our techniques are less expensive.