Improved Evaluation of Automatic Source Code Summarisation (2022)

Abstract: Source code summaries are a vital tool for the understanding and maintenance of source code as they can be used to explain code in simple terms. However, source code with missing, incorrect, or outdated summaries is a common occurrence in production code. Automatic source code summarisation seeks to solve these issues by generating up-to-date summaries of source code methods. Recent work in automatically generating source code summaries uses neural networks for generating summaries; commonly Sequence-to-Sequence or Transformer models, pretrained on method-summary pairs. The most common method of evaluating the quality of these summaries is comparing the machine-generated summaries against human-written summaries. Summaries can be evaluated using n-gram-based translation metrics such as BLEU, METEOR, or ROUGE-L. However, these metrics alone can be unreliable and new Natural Language Generation metrics based on large pretrained language models provide an alternative. In this paper, we propose a method of improving the evaluation of a model by improving the preprocessing of the data used to train it, as well as proposing evaluating the model with a metric based off a language model, pretrained on a Natural Language (English) alongside traditional metrics. Our evaluation suggests our model has been improved by cleaning and preprocessing the data used in model training. The addition of a pretrained language model metric alongside traditional metrics shows that both produce results which can be used to evaluate neural source code summarisation.

## introduction research producing models for neural source code summarisation frequently uses metrics designed for translation and natural language generation (nlg) tasks, such as bleu @xcite , meteor @xcite , @xcite , and rouge-l @xcite . these metrics are oriented to tasks such as translation and summarisation. @xcite used these metrics and a shared dataset to compare state-of-the-art models for neural source code summarisation. these metrics are based on n-gram matching, which compares the lexical similarity of texts, but lacks a comparison of the meaning of the texts -the semantic similarity. a lack of semantic comparison means that machine-generated texts which share sequences of n-grams with reference texts score well regardless of their ability to convey a similar meaning; whereas texts which convey a similar meaning but don't share many sequences of n-grams with reference texts score poorly. for summarisation tasks, the ability to replicate the semantics of a text where the generated language may be abstractive is more significant than the ability to generate as many n-grams matching those in a human-written summary as possible because the purpose of a summary is to provide the reader with an overview of the meaning of a larger text (or in this case, a source code method). nlg metrics based on large language models (llm) aim to improve the reliability of evaluation scores by capturing semantics through reliance on contextual embeddings. however, these metrics require large amounts of computational resources -making them expensive to run in comparison to traditional n-gram matching metrics. new efforts in making these llm-based metrics more accessible attempt to reduce the number of parameters compared to previous llm-based models whilst retaining similar accuracy. this allows for faster calculation of llm-based metrics. one such new effort is the frugalscore metric @xcite for evaluating nlg tasks. frugalscore uses previous llm-based metrics to train a miniature language model which learns on the internal mapping of the expensive metric. it is this model which is used for generating scores for pairs of sequences of text. in this paper, our aim is to train a state-of-the-art neural network model (we use the neuralcodesum model @xcite ) designed for source code summarisation, using a dataset of source code -summary pairs as described in section 2. the dataset we use is a modified version of the funcom dataset @xcite . we selected this dataset to allow us to compare our model to previous research @xcite which trains neuralcodesum on the same dataset and evaluates the results using traditional n-gram matching metrics. we then evaluate our trained model using the frugalscore metric @xcite to show the ability of such a metric to evaluate neural source code summarisation. ## research methodology we began by building our dataset, as described in the section 2. we used the funcom dataset @xcite , as this will allow us to compare our results to @xcite evaluation of neuralcodesum, and added our own preprocessing steps in order to improve our model. the model was trained and evaluated on a machine using an intel i9-12900kf cpu, 128gb ddr4 ram, and an nvidia geforce rtx 3090 gpu, using the official implementation of neuralcodesum, pytorch 1.3, and python 3.6. the result data from the evaluation process is collected in json format. the code used to process the dataset is available at github.com/phillijm/javadatasetcleaner. ## result analysis comparing our results to previous experiments (rq1.2), we compared the performance of our model to previous experiments using the smoothed bleu-4, rouge-l, and meteor metrics -as these metrics were presented by both @xcite and @xcite . a possible future work would be to test these models against a wider range of n-gram-based metrics and llm-based metrics. in answer to rq2: when comparing traditional n-gram-based natural language generation metrics to frugalscore @xcite for the task of neural source code summarisation, our model scored an average frugalscore value of 65.76, as shown in it is notable that -when compared to traditional metrics -frugalscore gives far fewer "bad" results (for example, 31.308% of smoothed bleu-4 results were above 30, compared to 99.998% of fru-galscore results). frugalscore also gives fewer "perfect" or "near-perfect" results than traditional metrics (for example, 6.571% of smoothed bleu-4 results were above 99, compared to 4.215% of frugalscore results). the difference between these results suggests that frugalscore gives more credit to summaries which other metrics rank poorly and less credit to summaries which other metrics rank highly, with higher median and mean values. this should be taken into account when directly comparing frugalscore to traditional metrics. our results show that -on average -frugalscore ranks summaries more highly than traditional metrics. this is likely either due to the ability of llmbased metrics to take the semantics of a sentence into account, where n-gram-based metrics do not, or due to a possible overestimation of results. to determine this would require further human evaluation, as mentioned in the limitations section. ## conclusion in testing a neuralcodesum model trained on a dataset of source code which has been parsed and had developer comments accurately removed, with accurate tokenisation of both source code and summaries, we have demonstrated the effect of ensuring a higher quality of training data has on improving the quality of a model -with our model outperforming that of @xcite . in evaluating our model with frugalscore alongside traditional metrics, we have shown how the two can be used alongside each other to provide an improved method of evaluating a model for neural source code summarisation.