Code to Comment Translation: A Comparative Study on Model Effectiveness & Errors (2021)

Abstract: Automated source code summarization is a popular software engineering research topic wherein machine translation models are employed to “translate” code snippets into relevant natural language descriptions. Most evaluations of such models are conducted using automatic reference-based metrics. However, given the relatively large semantic gap between programming languages and natural language, we argue that this line of research would benefit from a qualitative investigation into the various error modes of current state-of-the-art models. Therefore, in this work, we perform both a quantitative and qualitative comparison of three recently proposed source code summarization models. In our quantitative evaluation, we compare the models based on the smoothed BLEU-4, METEOR, and ROUGE-L machine translation metrics, and in our qualitative evaluation, we perform a manual open-coding of the most common errors committed by the models when compared to ground truth captions. Our investigation reveals new insights into the relationship between metric-based performance and model prediction errors grounded in an error taxonomy that can be used to drive future research efforts.

## introduction and motivation proper documentation is an important component of modern software development, and previous studies have illustrated its advantages for tasks ranging from program comprehension @xcite to software maintenance @xcite . however, manually documenting software is a tedious task @xcite and modern agile development practices tend to champion working code over extensive documentation @xcite . as such, a range of important documentation activities are often neglected @xcite leading to deficiencies in carrying out development activities and contributing to technical debt. because of this, researchers have worked to develop automated code summarization techniques wherein machine translation models are employed to generate precise, semantically accurate natural language descriptions of source code @xcite . due to the promise and potential benefits of effective automated source code summarization techniques, this area of work has seen constant and growing attention at the intersection of the software engineering and natural language processing research communities @xcite . various techniques for automated source code summarization have been explored extensively over the past decade. some of the earliest approaches made use of a combination of structural code information and text retrieval techniques for determining the most relevant terms @xcite , with follow up work investigating the use of topic modeling @xcite . techniques then evolved from using information retrieval to canonical machine learning techniques, with ying and robillard (2013) using supervised naive bayes and support vector machine classifiers to identify code fragment lines that could be used as suitable summaries. one of the first appearances of language modeling came from @xcite who proposed an approach combining a software word usage model, natural language generation systems, and the pagerank algorithm @xcite to generate summaries. driven by the advent of deep learning, current state-of-the-art techniques generally make use of large-scale neural models and have significantly improved the performance of code summa-rization tasks. for instance, @xcite used long short term memory @xcite with attention @xcite to generate summaries from a code snippet. following this work, researchers have applied several deep learning-based approaches to the task of source code summarization @xcite @xcite . in most works on automated code summarization, the performance of the generated natural language descriptions is evaluated using referencebased metrics adapted from machine translation, e.g., bleu @xcite and me-teor @xcite , or text summarization, e.g., rouge @xcite . as such, most researchers make conclusions based on the results obtained using these metrics. however, the code summarization task is a difficult one -due in large part to the sizeable semantic gap between the modalities of source code and natural language. as such, while these metrics provide a general illustration of model efficacy, it can be difficult to determine the specific shortcomings of neural code summarization techniques without a more extensive qualitative investigation into their errors. few past studies have examined the failure modes of neural code summarization models as we outline in §6. therefore, to further explore this topic, in this paper we perform both a qualitative and quantitative empirical comparison of three neural code summarization models. our quantitative evaluation offers a comparison of three recently proposed models (codebert @xcite , neuralcodesum @xcite , and code2seq @xcite ) on the funcom dataset (leclair and mcmillan, 2019) using the smoothed bleu-4 @xcite , me-teor @xcite , and rouge-l @xcite metrics whereas our qualitative evaluation consists of a rigorous manual categorization of model errors (compared to ground truth captions) based on a procedure adapted from the practice of open coding @xcite . in summary, this paper makes the following contributions: ## background: deep learning for code summarization this section outlines necessary background regarding our chosen evaluation dataset as well as the three neural code summarization models upon which we focus our empirical investigation. ## design of the empirical evaluation to evaluate the performance of our three models applied to the task of code summarization, we perform both a quantitative and qualitative evaluation centered upon the following research questions: rq 1 : how effective is each model in terms of predicting natural language summaries from java methods? rq 2 : what types of errors do our studied models make when compared to ground truth captions? rq 3 : what differences (if any) are there between the errors made by different models? ## evaluation results in this section, we will discuss the quantitative and qualitative results from our empirical study in order to answer our research questions. ## discussion & learned lessons takeaway 1: the codebert model illustrates improved performance on the funcom dataset as compared to codexglue, likely due to the filtering steps undertaken in its construction. previously, the codebert model was finetuned on the codexglue dataset and the smoothed bleu-4 score obtained on the java dataset was 17.65 @xcite . however, we fine-tuned the model on the funcom dataset and obtained a smoothed bleu-4 score of 24.15. we believe there are two primary contributing factors to this observation: 1) a higher volume of data, and 2) filtering strategies. codexglue only provides 164923 training examples, whereas we used 400000 java methods and javadoc pairs during he fine-tuning process. moreover, the codexglue dataset is obtained from codesearchnet and the documents that contain special tokens (e.g., <@xmath0:) are filtered. in our preprocessing, we did not completely remove such data in the preprocessing; we only remove the html and special characters from the javadoc captions. we hypothesize that such characters may contain important information and as such lead to more effective predicted summaries. takeaway 2: models that rely on statically parsing source code can lead to high numbers of missing/incomplete predictions. the preprocessing for the code2seq model includes generating strings from the ast node representation of each method. unfortunately, it is difficult (or impossible) to construct a suitable ast representation for methods that fall under a certain token length threshold. as a result, about 19.98% of the original dataset could not be fed into the code2seq testing module, and for which we could not generate any prediction for these examples. takeaway 3: some of the generated summaries provide a semantic meaning similar to the ground truth, despite exhibiting fewer n-gram matches. our studied models can generate summaries that contain relevant semantic information which can be useful for code comprehension despite not perfectly matching the ground truth. for instance, let's consider the following example ground truth for a java method, "this method sets the text for the heading on the component". the generated summary from the codebert model is "sets the heading caption". comparing these two descriptions will not necessarily result in a high bleu-4 score. this suggests that a modification to the evaluation procedure for these models may provide a more realistic characterization of model performance in practice. for instance, measuring bertscore in addition to other metrics for evaluation @xcite 12 may help to better capture semantic similarities compared to purely symbolic similarities. takeaway 4: future techniques for neural code summarization should carefully consider techniques for mitigating potential errors related to missing information, and incorrect construction as these are the most prevalent error types observed in our taxonomy. our error taxonomy provides concrete indicators on where different types of models stand to gain performance in order to make them useful for downstream deployment. in particular, we suggest that future research focuses on rectifying missing critical information and missing non-critical information rather than grammatical errors or unnecessary file information. takeway 5: future studies should explore the 12 https://github.com/tiiiger/bert_score combination of ast traversal based and selfattention mechanism-based approaches to perform robust comment generation. ast-based approach is useful to provide syntax level information and it follows the structural tree traversal method to capture the global information. at the same time, we can see this approach is prone to errors like repetition and semantically unrelated to code. on the other hand, a selfattention mechanism is useful to capture the local information. so a multi-modal approach where standard encoders can be utilized to combine both ast-based and attention-based approaches can be a viable direction to explore further. takeway 6: robust evaluation metric(s) should be developed that specifically focus on source code -natural language translation. source code is fundamentally different from the natural language from a number of perspectives. for instance, it exhibits less significant word order dependency, the significance of appropriate syntax naming and mentioning, etc. so a robust code to natural language translation evaluation metric should consider assessment from both local and global levels. standard machine translation metrics like bleu, me-teor, rouge do not fully cover these factors. as such, we encourage future work to study and develop new forms of automated metrics for assessing this special case of machine translation. ## conclusion & future work in this work we perform both quantitative and qualitative evaluations of three popular neural code summarization techniques. based on our quantitative analysis, we find that the codebert model performs statistically significantly better than two other popular models (neuralcodesu, and code2seq) achieving a smoothed-bleu-4 score of 24.15, a meteor score of 30.34, and a rouge-l score of 35.65. our qualitative analysis highlights some the most common errors made by our studied models and motivates follow-up work on improving specific model attributes. in the future, we aim to expand our analysis to additional retrieval-augmented summarization techniques and to expand the scope and depth of our neural code summarization model error taxonomy.