SUT : Active Defects Probing for Transcompiler Models (2023)

Abstract: Automatic Program translation has enormous application value and hence has been attracting significant interest from AI researchers. However, we observe that current program translation models still make elementary syntax errors, particularly, when the target language does not have syntax elements in the source language. Metrics like BLUE, CodeBLUE and computation accuracy may not expose these issues. In this paper we introduce a new metrics for programming language translation and these metrics address these basic syntax errors. We develop a novel active defects probing suite called Syntactic Unit Tests (SUT) which includes a highly interpretable evaluation harness for accuracy and test scoring. Experiments have shown that even powerful models like ChatGPT still make mistakes on these basic unit tests. Specifically, compared to previous program translation task evaluation dataset, its pass rate on our unit tests has decreased by 26.15%. Further our evaluation harness reveal syntactic element errors in which these models exhibit deficiencies.

## introduction program translation or transcompilation aims to automatically convert source code from one programming language (e.g., c++) into another (e.g., java), meanwhile the results should preserve the program function and ideally follow the target language conventions. the recent proposal of transcompilation datasets such as codexglue @xcite benchmark, geeksforgeeks dataset for computation accuracy @xcite and xlcost @xcite benchmark has propelled related research in the program translation domain @xcite @xcite @xcite * joint first author 2021; @xcite @xcite . these innovations have resulted in impressive improvements in neural transcompilation, but most models still make elementary syntax errors, especially when the source and target languages have different sets of syntactic structures. existing datasets are often criticized for their lack of granularity, which can mask deficiencies in basic syntactic elements that humans care. their data are derived from randomly sampled code stores or programming contest websites, and the code they contain may not even be executable. as a result, researchers have had to rely on sequence similarity metrics like bleu, rouge, or code-bleu to evaluate the quality of code translation. however, these metrics have been shown to be inadequate for objectively assessing the quality of code translation. in this paper, we propose syntactic unit tests(sut), which leverage basic syntax structures of each language along with unit tests to verify the correctness of the translations. table1 presents a comparative analysis of sut against other code translation evaluation datasets. our experiments show that sut performance of state-ofthe-art transcompilers like chatgpt has definite room for improvement, demonstrating the value of the metric for judging future model improvements. further, we propose the syntax unit test accuracy (sut acc) and syntax element test score (sets), using statistical inference to analysis individual syntax element performance, which is more in line with translation quality that humans concern about. in summary, we make the following contributions: ## conclusion in this paper, we introduce an interpretable benchmark via syntactic unit test and associated syntax element test score, which can diagnose the specific weakness of a given translation model. it fills the gap in the field of a fine-grained evaluation dataset for program translation. by our designed evaluation suites, researchers can actively probe deficiencies in their models about specific basic syntactic elements and improve their performance in a targeted manner.