C ode T rans O cean: A Comprehensive Multilingual Benchmark for Code Translation (2023)

Abstract: Recent code translation techniques exploit neural machine translation models to translate source code from one programming language to another to satisfy production compatibility or to improve efficiency of codebase maintenance. Most existing code translation datasets only focus on a single pair of popular programming languages. To advance research on code translation and meet diverse requirements of real-world applications, we construct **CodeTransOcean**, a large-scale comprehensive benchmark that supports the largest variety of programming languages for code translation. CodeTransOcean consists of three novel multilingual datasets, namely, **MultilingualTrans** supporting translations between multiple popular programming languages, **NicheTrans** for translating between niche programming languages and popular ones, and **LLMTrans** for evaluating executability of translated code by large language models (LLMs). CodeTransOcean also includes a novel cross-framework dataset, **DLTrans**, for translating deep learning code across different frameworks. We develop multilingual modeling approaches for code translation and demonstrate their great potential in improving the translation quality of both low-resource and high-resource language pairs and boosting the training efficiency. We also propose a novel evaluation metric **Debugging Success Rate@K** for program-level code translation. Last but not least, we evaluate LLM ChatGPT on our datasets and investigate its potential for fuzzy execution predictions. We build baselines for CodeTransOcean and analyze challenges of code translation for guiding future research. The CodeTransOcean datasets and code are publicly available at https://github.com/WeixiangYAN/CodeTransOcean.

## introduction early software systems are developed using programming languages such as fortran and cobol, which have a significantly smaller user base compared to modern mainstream programming languages (e.g., python and java). hence maintaining and modernizing early software systems are expensive @xcite . moreover, the readability and compatibility of the mixed multitude of programming languages are challenging when migrating existing software systems to new technology ecosystems or integrating software systems using different programming languages. the code translation task aims to convert source code from one programming language to another and is of great value in industry. code translation methods evolve from the inefficient, costly, and error-prone manual rewriting method to automatic methods. automatic code translation methods can be categorized into compilers and transpilers, rule-based methods, and neural network based methods. neural models @xcite @xcite have become dominant in code translation. details of code translation methods are presented in appendix a.1. the performance of neural models relies heavily on large-scale high-quality parallel data. however, existing code translation datasets are limited by insufficient coverage of programming languages and mostly focusing on a single pair of popular programming languages, limited scale, and uneven data distribution. the widely used codetrans @xcite is a small dataset containing only java-c# parallel data for quite short code samples. other datasets @xcite @xcite @xcite suffer from the same limitations. consequently, existing code translation models @xcite @xcite are confined to a narrow range of one-to-one code to address these limitations and advance neural code translation models, we construct a largescale comprehensive multilingual code translation benchmark codetransocean, summarized in dltrans includes 408 samples covering four mainstream deep learning frameworks. multilingual modeling shows great potential in neural machine translation @xcite @xcite , but it has not been systematically explored for code translation. we investigate multilingual modeling for code translation using our multilingualtrans, nichetrans, and dltrans datasets. experimental results demonstrate that multilingual modeling significantly improves translation quality for both high-resource and low-resource language pairs and improves the model training efficiency. recent research indicates that the proficiency of the llm chatgpt in natural language translation is on par with commercial-grade translation systems @xcite . to the best of our knowledge, our work is the first to systematically investigate the potential of chatgpt in code translation. we develop a fully automated translationexecution-evaluation pipeline autotransexecuter to support this study. note that match-based metrics and execution-based metrics have been used for evaluating code translation methods, with details in appendix a.1. in order to accurately evaluate the usability of translated code from chatgpt, we propose a novel execution-based evaluation metric debugging success rate @k (dsr@k), which is the percentage of samples with translation results that successfully execute and produce the expected functionality after k debugging rounds. on our llmtrans dataset, the baseline chatgpt setting achieves 48.57% dsr@0. we find that self-debugging and one-shot improve the performance while chain-of-thought strategies degrade the translation accuracy. since our autotransex-ecuter still cannot cover arbitrary programming languages, we also propose a novel metric fuzzy execution, attempting to address the limitations of existing evaluation metrics for code translation. our preliminary study using chatgpt shows that chat-gpt is still inadequate to predict fuzzy execution for any arbitrary programming language, which demands future research. our contributions can be summarized as follows: we are the first to systematically evaluate multilingual modeling on code translation for both high-resource and low-resource language pairs. experimental results demonstrate that multilingual modeling significantly improves translation quality for both high-resource and low-resource language pairs and improves training efficiency. code translation datasets the success of neural models for code translation relies heavily on large-scale high-quality parallel data. however, existing code translation datasets are plagued by issues such as insufficient coverage of programming languages, limited scale, and imbalanced data distribution. the widely used code translation dataset codetrans (lu et al., 2021) in the codexglue benchmark consists of java-c# function pairs. the small parallel corpus avatar (ahmad et al., 2023) is constructed for java-python code translation. nguyen et al. (2013) construct a java-c# dataset to explore statistical machine translation on code translation tasks foot_0 . chen et al. (2018) explore this dataset from nguyen et al. and also construct a coffeescript-javascript parallel dataset for investigating tree-to-tree neural models for code translation. rozière et al. (2020) create a dataset containing 852 programs to evaluate unsupervised methods. recently, zhu et al. (2022b) construct a new translation dataset cost from the geeksforgeeks website foot_1 . subsequently, they release the translation dataset xlcost (zhu et al., 2022a), which expands the cost dataset by 7.3 times. however, the limited language coverage of these datasets and their imbalanced data distribution hinder their practical applications. rozière et al. (2022) construct the transcoder-st dataset to perform unsupervised code translation using automated unit tests. details of these datasets are summarized in ## the codetransocean benchmark in this section, we provide detailed descriptions and analyses of our codetransocean benchmark, including the code translation tasks, their associated datasets, and dataset statistics. details of data collection methods and licensing information as well as quality control and quality assessment are presented in appendix a.2. note that the vast dataset source programming languages #samples avg. #tokens/sample avg. length codetrans (lu et al., 2021) java, c# 11,800 59 / 63 / 58 205 / 218 / 202 avatar (ahmad et al., 2023) java, python 9,517 239 / 235 / 234 691 / 687 / 688 nguyen et al.(nguyen et al., 2013) java, c# 16,966 --lachaux et al.(rozière et al., 2020) c++, java, python 852 -/ 119 / 120 -/ 313 / 311 cost (zhu et al., 2022b) c++, java, python, c#, javascript, php, c 16,738 272 / 180 / 199 770 / 458 / 511 transcoder-st (rozière et al., 2022) java, c++, python 437,030 --xlcost (zhu et al., 2022a) c++, java, python, c#, javascript, php, c 122,151 234 / 232 / 222 644 / 634 / 606 there is no overlap between codetransocean datasets and existing code translation datasets. ## experiments we present experiments of multilingual training for code translation (section 4.1). we then introduce a novel evaluation metric debugging success rate@k for program-level code translation (section 4.2) and the first comprehensive exploration of chatgpt for code translation (section 4.3). ## conclusion we construct codetransocean, a comprehensive code translation benchmark that includes multilingual and cross-framework datasets. we demonstrate that multilingual modeling has remarkable potential in enhancing code translation quality. we also reveal the superior code translation capability of chatgpt and advanced strategies lead to significant performance gains. moreover, we introduce fuzzy execution that may overcome limitations of existing metrics but requires future research. in summary, we provide a comprehensive suite of resources, tools, and baselines for code translation.