AVATAR : A Parallel Corpus for J ava-Python Program Translation (2023)

Abstract: Program translation refers to migrating source code from one programming language to another. It has tremendous practical value in software development, as porting software across languages is time-consuming and costly. Automating program translation is of paramount importance in software migration, and recently researchers explored unsupervised approaches due to the unavailability of parallel corpora. However, the availability of pre-trained language models for programming languages enables supervised fine-tuning with a small number of labeled examples. Therefore, we present AVATAR, a collection of 9,515 programming problems and their solutions written in two popular languages, Java and Python. AVATAR is collected from competitive programming sites, online platforms, and open-source repositories. Furthermore, AVATAR includes unit tests for 250 examples to facilitate functional correctness evaluation. We benchmark several pre-trained language models fine-tuned on AVATAR. Experiment results show that the models lack in generating functionally accurate code.

## introduction software developers and researchers often require to convert software codebases or research prototypes from one platform to another or rewrite them in the target programming languages. manually rewriting software is time-consuming, expensive, and requires expertise in both the source and target languages. for example, the commonwealth bank of australia spent around $750 million and 5 years translating its platform from cobol to java @xcite . a program translation system that converts the source code of a program written in a programming language to an equivalent program in a different programming language is known as a transcompiler, transpiler, or source-tosource compiler. transcompilers have a prodigious practical value; they could help to reduce the translation efforts of developers and researchers by not requiring them to write code from scratch, instead, they can edit the translated code with less effort. the conventional transcompilers are based on rule-based approaches; they first convert source code into an abstract syntax tree (ast) and then apply handwritten rules to translate to the target language. development and adaptation of transcompilers need advanced knowledge and therefore are available in a handful of programming languages. undoubtedly, the automation of program translation would facilitate software development and research tremendously. with the recent advancements in data-driven neural machine translation (nmt) approaches between natural languages, researchers have started investigating them for programming language translation. @xcite trained an nmt system in an unsupervised fashion using large-scale monolingual source code from github that showed noteworthy success in source code translation between java, python, and c++ languages. pre-trained language models (plms) of code have been shown to work well on java-c# translation after fine-tuning on a small amount of parallel examples @xcite @xcite . motivated by these favorable results, in this work, we propose a new parallel corpus of java and python programs. we propose a corpus, avatar (java-python program translation) that consists of solutions written in java and python for 9,515 programming problems collected from competitive programming sites, online platforms, and open source repositories. avatar includes 250 examples with unit tests to facilitate functional correctness evaluation of program translation. we train several baselines, including models trained from scratch or pre-trained on large-scale source code collection and fine-tuned on avatar. the experiment results indicate that while the models perform considerably in terms of the lexical match, they lack furthermore, avatar offers 3,391 parallel functions that we use to train models or fine-tune pre-trained language models and perform function translation evaluation on the dataset released by @xcite . our code and data are released at https: //github.com/wasiahmad/avatar . ## avatar construction data collection we construct avatar based on solutions of computational problems written in java and python collected from open source programming contest sites: atcoder, aizu online judge, google code jam, codeforces, and online platforms: geeksforgeeks, leetcode, project euler. we crawl codeforces and geeksforgeeks sites to collect the problem statements and their solutions. we collect the atcoder and aizu data from puri et al. (2021), google code jam data from nafi et al. (2019) foot_0 , and leetcode and project euler problem solutions from open source github repositories. foot_1 , foot_2 we collect [1 -20] accepted solutions for a single problem written in java and python. preprocessing & filtering at first, we tokenize the solution code and remove docstrings and comments from them. we use the javalang foot_3 tokenizer for java and the tokenizer foot_4 of the standard library for python. after tokenization, we filter out solutions that are longer than a specified length threshold (= 464). in the initial data collection, there are [1 -20] accepted solutions for each problem. we filter out solutions and only keep at most 5 solutions per problem. our goal is to keep the solutions that are maximally different from others in order to increase diversity among solutions of the same problem. we use the open source library difflib foot_5 to compare all the solutions pairwise (individually in java and python) and select five solutions that differ most from others. ## related works several works in the past have contributed to building a parallel corpus for source code translation. @xcite curated the first parallel corpus of java and c# functions by developing a semiautomatic tool to search for similar class names and method signatures from two open source projects, lucene and db4o. similarly, @xcite built a mining tool that uses the java and c# antlr grammar to search for similar methods from five open source projects -db4o, lucene, hibernate, quartz, and spring. subsequent works used libraries and transcompilers to construct parallel corpus. for example, @xcite used 2to3, a python library 9 and @xcite used a transcompiler to create a parallel corpus between python 2 -python 3 and coffeescript -javascript, respectively. recently, @xcite collected programming problem solutions in java, python, and c++ (∼850 func-9 https://docs.python.org/2/library/ 2to3 tions in each language) from geeksforgeeks to evaluate their proposed translation model. concurrent works @xcite present unit tests-based benchmarks to evaluate zero-shot translation capabilities of large language models. different from these works, we propose a sizeable parallel corpus of java and python programs by collecting programming problem solutions from competitive programming sites, online platforms, and open-source repositories. ## conclusion this work proposes a parallel corpus of java and python programs to contribute to the development of translation systems for programming languages that have a sizeable impact on software development. we evaluate several neural machine translation systems on the proposed dataset and perform analysis to reveal crucial factors that affect program translation accuracy. in our future work, we want to increase the size of the parallel corpus and support more programming languages.