Summarize and Generate to Back-translate: Unsupervised Translation of Programming Languages (2023)

Abstract: Back-translation is widely known for its effectiveness in neural machine translation when there is little to no parallel data. In this approach, a source-to-target model is coupled with a target-to-source model trained in parallel. The target-to-source model generates noisy sources, while the source-to-target model is trained to reconstruct the targets and vice versa. Recent developments of multilingual pre-trained sequence-to-sequence models for programming languages have been very effective for a broad spectrum of downstream software engineering tasks. Hence, training them to build programming language translation systems via back-translation is compelling. However, these models cannot be further trained via back-translation since they learn to output sequences in the same language as the inputs during pre-training. As an alternative, we propose performing back-translation via code summarization and generation. In code summarization, a model learns to generate natural language (NL) summaries given code snippets. In code generation, the model learns to do the opposite. Therefore, target-to-source generation in back-translation can be viewed as a target-to-NL-to-source generation. We show that our proposed approach performs competitively with state-of-the-art methods. We have made the code publicly available.

## introduction the choice of programming language (pl) in software development depends on the requirement of the software and the available features of a particular pl. in modern api-driven software development, language choice often depends on the availability of libraries and apis. the advent of newer and richer programming languages often requires legacy software to be translated into modernized pls. in theory, modern programming languages' (2) java code to nl summary guard is used to determine if a generator is already defined . aligning libraries and apis across different pls is a non-trivial task. recent progress in neural machine translation (nmt) @xcite (a) plbart (b) plbart + s&g figure 2 : t-sne plot of function embeddings of java and python functions. figure 2a shows the embedding generated by the plbart model. figure 2b are the generated embedding when the plbart is finetuned to jointly summarize code to nl and generate code from nl (plbart + s&g). while plbart clusters programs from each pls, parallel programs in different pls are brought closer to each other by plbart + s&g. @xcite leveraging pre-trained models @xcite @xcite @xcite could be a possible way to learn the alignment between pls and translate source code across languages. a significant challenge in supervised learning for nmt is the need for large-scale parallel corpora. for instance, if we are planning to train a translator for java to python translation, we need a considerable number of the same program (i.e., exhibiting the same semantic behavior) in both languages. availability of such parallel datasets is a vital challenge in programming language translation @xcite . back-translation (bt) @xcite is a clever way to learn alignments across different languages. while bt demonstrates success in nmt, those require either (i.) small (perhaps noisy) parallel datasets or (ii.) a model with some capacity of cross-lingual generation -to kickstart the bt-based learning process. in this work, we investigate the suitability of the multilingual pre-trained sequence-to-sequence model (psm) for unsupervised programming language translation via bt. in particular, we assume a use-case scenario where no parallel data is available. without much of a surprise, we empirically found that, while these psms are good at generating code in each language, they exhibit very little to no knowledge about cross-lingual generation since such psms are typically trained to reconstruct code sequences from noisy inputs. for example, when we provide the input code in figure 1 to plbart @xcite and ask to generate python code without training, it generates a slight variation of the input java code, showing its lack of knowledge about cross-lingual generation. to endow such psms with knowledge about cross-lingual generation, we propose using a third language (i.e., english). since a large quantity of monolingual code corpora comes with documentation, which supposedly describes what the source code is doing, we train a summarize-and-generate (s&g) model that can generate pseudo-parallel code sequences. figure 1 shows plbart's behavior when it is further trained via s&g. first, given the java code, it generates an nl summary (figure 1 -2) and subsequently generates python code (figure 1 2 3 ). we empirically show that, even if such s&g model generates noisy parallel sequences, it allows us to employ psms in the bt-based training to learn programming language translation. in summary, we present a summarize-and-generate (s&g) based approach to enable unsupervised program translation training of plbart via back-translation (bt). experiment results show that our proposed approach makes plbart trainable via bt and performs competitively with stateof-the-art program translation models. ## motivation recent years saw several pre-trained sequenceto-sequence models (psm) @xcite . these models are pre-trained on hundreds of gigabytes of source code. thus, in this work, we are motivated to investigate their adoption in learning program translation via back-translation. to understand such feasibility, we investigate the program representations generated by the psm. as a case study, we chose plbart @xcite and evaluated its multilingual embeddings as suggested in @xcite . we in the first m training steps (out of total n steps), plbart generates natural language (nl) summary of the code in target language (python in this example) and generates the code in source language (java in this example) from the nl summary. in the remaining nm steps, plbart directly generates the code in source language from the code in target language. find the parallel java function for each of the 948 python functions using the parallel dataset proposed in @xcite . we find the nearest neighbor using cosine similarity between function embeddings and calculate the error rate. unsurprisingly, plbart performs poorly in function retrieval with an 87.5% error rate. in comparison, we fine-tune plbart jointly on code summarization and generation in java and python. repeating the experiment of function retrieval, we find that fine-tuned plbart's error rate drops to 23.7%. to visually illustrate the embeddings produced by plbart and its fine-tuned variant, we provide a t-sne plot of 8 sample functions' embedding in figure 2 . we see the functions that belong to the same language are clustered together while the same functions in two different languages are far apart from each other (see figure 2a ). in contrast, the fine-tuned plbart breaks up the intra-language clusters and brings functions in different languages close to each other in the embedding space (see figure 2b ). these results motivate us to initialize the translation models with fine-tuned plbart on code summarization and generation for back-translation as it learned some alignment across programming languages. ## approach sequence-to-sequence models, such as plbart @xcite , codet5 @xcite , spt-code @xcite map source code sequences into a shared multilingual space by pre-training on multiple programming languages jointly using unlabeled data (e.g., source code from github). the pre-training objective of these models is either denoising autoencoding (dae) or fill-inthe-blank, where the models reconstruct the original code snippet or predict the missing code tokens given a corrupted code snippet. although pretrained jointly on many languages, these models only learn to generate in the same language as input. as a result, these models are not trainable via backtranslation (bt) to learn programming language translation in an unsupervised fashion. as an alternative, we propose translating to and from natural language to perform back-translation between two programming languages. we refer to translating to and from natural language as code summarization and code generation, respectively. our proposal is motivated based on the availability of bimodal data, source code, and their summaries that are used to train code summarization and generation models. ## related work programming language translation translating programs or source code across different pro-gramming languages (pl) requires a profound understanding of the pls. having strictly defined syntax and semantics, pls are suitable for phrasebased statistical machine translation @xcite @xcite . @xcite introduced a tree-to-tree machine translation to translate programs and to learn the syntactic alignment between source and target pl. recently proposed pre-trained programming language models showed promising results in translating programs across pls @xcite @xcite . however, these approaches require a set of parallel programs to train the encoder-decoder model. recently proposed transcoder @xcite shows initial success results in unsupervised program translation, eliminating the requirement of bi-modal data. they achieve such jointly training a model using xlm @xcite , denoising auto encoding (dae) @xcite , and back-translation(bt) @xcite . this work empirically investigated the suitability of adopting bt to train existing pretrained encoder-decoder models and proposed an alternative via summarization and generation. ## conclusion in this research, we show that pre-trained sequenceto-sequence models (e.g., plbart) are not suitable for direct adaptation via back-translation to learn to translate. to address the issue, we propose to use code summarization and generation as an alternative to performing back-translation. we show that our proposed approach turns plbart into a translation model that performs competitively to existing unsupervised translation models.