Program Translation via Code Distillation (2023)

Abstract: Software version migration and program translation are an important and costly part of the lifecycle of large codebases. Traditional machine translation relies on parallel corpora for supervised translation, which is not feasible for program translation due to a dearth of aligned data. Recent unsupervised neural machine translation techniques have overcome data limitations by included techniques such as back translation and low level compiler intermediate representations (IR). These methods face significant challenges due to the noise in code snippet alignment and the diversity of IRs respectively. In this paper we propose a novel model called Code Distillation (CoDist) whereby we capture the semantic and structural equivalence of code in a language agnostic intermediate representation. Distilled code serves as a translation pivot for any programming language, leading by construction to parallel corpora which scale to all available source code by simply applying the distillation compiler. We demonstrate that our approach achieves state-of-the-art performance on CodeXGLUE and TransCoder GeeksForGeeks translation benchmarks, with an average absolute increase of 12.7% on the TransCoder GeeksforGeeks translation benchmark compare to TransCoder-ST.

## introduction program translation is the process of converting code written in one programming language to another. unlike compilers that only convert highlevel languages to low-level machine code, program translation can generate translations between complex high-level programming languages. the industry demand for high-quality program translation is ever-present, as many companies require the migration of project codes from older programming languages to new ones with better support, features, and a modern workforce. for example, cobol * joint first author code libraries that are widely used in banks and other financial institutions need to be migrated to updated languages for easier maintenance and development. additionally, adaptation to different hardware platforms and operating systems often requires projects to be implemented in different high-level languages or different versions of the same language. an automatic program translation system, such as java to swift, is a blessing for android developers who want to adapt their applications to the ios operating system. natural language translation typically relies heavily on high-quality parallel corpora. however, creating parallel data for program translation tasks can be prohibitively expensive as it requires the generated code to be as aligned as possible. unlike natural language, even minor variations or missing tokens can result in significant errors in code languages. additionally, perfect alignment of tokens does not guarantee accuracy due to unique syntax rules and restrictions present in each code language. for example, when translating java code to c++, the variable name "xor" in java cannot be directly mapped as a variable name in c++ because it is a reserved word in c++. therefore, program translation is a challenging task due to the difficulty in obtaining high-quality parallel code data. recent research has proposed unsupervised methods for neural machine translation in program translation tasks. @xcite randomly adds various types of noise (token deletion, token shuffling, etc.) to multilingual code snippets and forces the model to recover them. this strategy not only helps close the gap between different programming languages but also improves the alignment ability of the model. transcoder @xcite , introduced an unsupervised approach for translating source code by constructing a pseudo-parallel corpus using back-translation to augment translation data pairs. another system, transcoder-st @xcite , employs open-source automatic test generation tools to develop unit tests for commonly used programming languages. this system eliminates potentially invalid data pairs in back-translation and refines the unsupervised model using highconfidence data examples that have passed the tests, thus enhancing the overall level of confidence in the machine translation process. these methods, however, face significant challenges, which currently limit their ability to serve as a perfect method of translation. due to the random noise, plbart may end up paying more attention to irrelevant information in the input, overlooking the crucial information needed for alignment, resulting in slower convergence speed and reduced performance. transcoder models can be biased towards certain data patterns, which can result in poor performance on new or unseen data. besides, the implementation of back translation in language alignment can introduce randomness and ambiguities between program languages, which ultimately slows the convergence of the model and presents challenges when aligning low-frequency words. to alleviate this problem, transcoder-st filter out these errors during back translation though unit tests, but the ensuing challenge is the cost of constructing unit tests, which can be particularly high in industrial projects where the code requires multiple dependencies. this poses a significant obstacle in the capturing of appropriate test data for code fragments in complex scenarios and at times, make it infeasible. in this paper, we introduce code distillation, which overcomes the limitations of parallel data and alignment in different languages by leveraging a custom compiler to a simplified intermediate representation which contains only the essential logical aspects and object names. we train a model to decompile the distilled code, so that the process of translation is to compile the source snippet to the distilled representation, and decompile to the target language using the trained model. we systematically discovered which elements of each source language are necessary for code understanding, eliminating these language-specific elements from the distilled code representation. further, our distillation unifies basic morphemes, blurring grammatical differences between languages, and uses a more abstract form to merge high-level semantic expressions between different languages. to demonstrate the effectiveness of using distilled code as translation pivot, we introduce a novel multilingual program translation model called codist that unifies multiple translation pairs into a single distilled code decompiler task. through the specific language token switch, the distilled code derived from a source language code snippet can be seamlessly converted into any target language supported by the model. our experiments demonstrate that our method surpasses state-of-the-art techniques on the codexglue @xcite and transcoder geeksforgeeks benchmark datasets. furthermore, our model exhibits zero-shot and few-shot abilities that are competitive in low-resource code languages. to summarize, our main contributions are: distilled code unifies all basic morphemes across languages. for high-level semantics (e.g., complex api calls), code distillation extracts the core part of program logic, and decomposes concatenated object names into bag-of-words representations. ## related work the lack of parallel translation pairs poses a significant challenge for automated program translation. to overcome this challenge, many recent works have focused on designing pre-training tasks or using various unsupervised methods. in this section, we will discuss the benefits and limitations of these approaches. ## distilled code compiler since these approaches mentioned in the "related work" section still have their own limitations, we choose an alternative approach that involves building a translation pivot. instead of one-to-one translation processes, we first compile any source language program to a distilled code representation and then decompile it into target program using a trained model. we provide a unified front-end for compilation, which improves information density and narrows the language gap by distilling core semantics from the original code. in this section, we will briefly introduce translation pivots and explain the construction process of our compiler. ## distilled code decompiler after source code is transformed into the translation pivot through the distilled code compiler, next we need to build a decompiler that can output the target code. we introduce our multilingual distilled code decompiler model, which is a encoderdecoder transformer based model pre-trained on large-scale multilingual code corpora. under the framework of multilingual models, all languages share the same encoder and decoder and map to the same latent space. in this section, we introduce several self-supervised pre-training tasks that we implemented. the first two tasks provide good representations for our distilled code and original code. the last pre-training task is similar to the final decompiler task, which further aligns the distilled code and target code. ## conclusion in this paper, we propose a novel method for program translation with distilled code. our distilled code has the characteristics of high information density and general applicability, making it an ideal candidate for serving as a translation pivot. we have developed distilled code compilers for c#, java, python, and c++, and a new multilingual program translation model. this model unifies multiple translation pairs into the distilled code decompiler task. our results demonstrate the competitiveness of the proposed approach on the codexglue and geeksforgeeks datasets, and highlights the potential for building good translation pivots in program translation tasks. we argue that our distillation idea has universality and is a highly efficient information compression method for structured texts like high-level programming languages, documents and tables with minimal data loss.