Explain-then-translate: an analysis on improving program translation with self-generated explanations (2023)

Abstract: This work explores the use of self-generated natural language explanations as an intermediate step for code-to-code translation with language models. Across three types of explanations and 19 programming languages constructed from the MultiPL-E dataset, we find the explanations to be particularly effective in the zero-shot case, improving performance by 12% on average. Improvements with natural language explanations are particularly pronounced on difficult programs. We release our dataset, code, and canonical solutions in all 19 languages.

## introduction program translation (i.e., translating code from one language to another) has significant value in reallife applications, including in legacy software modernization and enabling programmers to quickly adapt to new languages. within prompt-based approaches to code translation, @xcite recently found that simply prompting an llm to generate explanations of the source program before generating the target program can improve performance. however, this conclusion is drawn on a single translation direction from c++ to python @xcite , and lacks evaluation on a broader set of programming languages including low-resource languages-a key component of code-to-code translation tasks in a software modernization setting. this paper systemtically evaluates this "explainthen-translate" approach to code translation through the multipl-e dataset @xcite . as the original dataset was constructed for the nl-to-code setting, we repurpose this dataset into a code-to-code, "multipl-c2c" dataset. we analyze our results in 36 different translation directions over different types of explanations. we find that explain-then-translate improves zero-shot performance consistently in 18 python-to-x translation directions, but much less so in the few-shot setting. we observe detailed explanations to be more useful when translating into high-resource pls and from low-resource into other low-resource pls. in contrast, translating from high-to lowresource pl's benefits from more abstract explanations. to aid future research in code-to-code translation across diverse language, we release our evaluation system, as well as canonical solutions in all languages, providing a 19-way parallel program translation evaluation set. ## explain-then-translate for code translation in code translation, we are given code x in a source language and must generate a program y in a target language that is functionally equivalent to x. in this paper we are interested in whether a selfgenerated natural language explanation z can be used to improve this translation process. foot_1 figure 1: compared to direct code translation prompt, exp (ours) prompts models to explain the code before translating. blue highlights are model completions, and red highlights point out the crucial difference between the two translations. example prompts and explanations for exp-lbl and exp-lbl-d in apx c, e.19 ## experiments and discussion in our study we focus on two main sets of translation directions: python-to-x, where we translate from python to 18 other target languages ranging from high to extremely-low-resource ( §3.1), and xto-x, where we target a representative set of translation directions varying source and target language resource levels and typing characteristics ( §3.2). we analyze translation improvements across models of 4 different sizes ( §3.3) and discuss improving individual explanations through heuristics ( §3.4). finally we show our method improves more on difficult-to-translate examples ( §3.5) and provide ablations to understand what nl explanations improves performance and whether alternative selfgenerated contexts could help ( §3.6). ## conclusion this work conducts a thorough analysis of the performance of large language models in program translation by using different types of selfgenerated explanations as an intermediate step. models generate higher quality detailed explanations for high-resource languages, while still generating good enough abstract explanations for lowresource languages. with simple heuristics, we have also demonstrated the potential to improve explanation quality and consequently translation quality. we identify key requirements for explanation and find that on average, mistakes in nl are less detrimental to performance, and do not require verification to perform well compared to using pl as self-generated contexts.