Revisiting Code Similarity Evaluation with Abstract Syntax Tree Edit Distance (2024)

Abstract: This paper revisits recent code similarity evaluation metrics, particularly focusing on the application of Abstract Syntax Tree (AST) editing distance in diverse programming languages. In particular, we explore the usefulness of these metrics and compare them to traditional sequence similarity metrics. Our experiments showcase the effectiveness of AST editing distance in capturing intricate code structures, revealing a high correlation with established metrics. Furthermore, we explore the strengths and weaknesses of AST editing distance and prompt-based GPT similarity scores in comparison to BLEU score, execution match, and Jaccard Similarity. We propose, optimize, and publish an adaptable metric that demonstrates effectiveness across all tested languages, representing an enhanced version of Tree Similarity of Edit Distance (TSED).

## introduction and related work in the fields of natural language processing and software engineering, code generation tasks are gaining more and more attention. assessing the quality of generated code is now critically important, but we still lack evaluation methods other than traditional statistical sequence evaluation methods. widely used semantic evaluation metrics like bleu score and jaccard similarity rely on statistical characteristics, overlooking the intricate grammatical structures and logical relationships inherent in complex programming languages. however, recent developments in the nlp field paved the way for novel evaluation metrics which we explore in this study. for one, the staggering number of powerful large language models (llms) such as gpt-3.5/4 @xcite revolutionized the nlp landscape and led to noteworthy advancements in the realm of code review and evaluation @xcite . another recent study introduced the novel tsed metric and used it to evaluate text-to-sql tasks @xcite . for this study, we take advantage of these developments to (1) prompt the gpt-4 model to generate similarity scores for code, and (2) expand on the tsed metric. we utilize these two different metrics (gpt and tsed) to evaluate the structural similarity of different programming languages and how they relate to execution matches. furthermore, we address how these metrics are correlated to semantic similarity metrics like the bleu score. finally, we investigate some limitations of these metrics by delving into the impact of tsed's penalty weight of tree operations on evaluation accuracy and exploring the stability of outputs from the gpt llms. as a result, we have these 3 contributions from this research: (a) we propose and publish a new tool for 48 programming languages foot_0 , (b) we discuss 2 recent evaluation metrics and 2 traditional metrics and compare them via correlation coefficient, recall to execution match, (c) we discuss the unstable nature of gpt similarity scoring and the ways to optimize tsed. ## conclusion in this paper, we applied tsed to more programming languages, compared gpt similarity and tsed to semantic metrics, and checked representation to execution match. then we discuss limitations about the stability of gpt scoring and the penalty parameters of tsed.