Robust Code Summarization (2023)

Abstract: This paper delves into the intricacies of code summarization using advanced transformer-based language models. Through empirical studies, we evaluate the efficacy of code summarization by altering function and variable names to explore whether models truly understand code semantics or merely rely on textual cues. We have also introduced adversaries like dead code and commented code across three programming languages (Python, Javascript, and Java) to further scrutinize the model’s understanding. Ultimately, our research aims to offer valuable insights into the inner workings of transformer-based LMs, enhancing their ability to understand code and contributing to more efficient software development practices and maintenance workflows.

## introduction code summarization is a task that involves generating coherent and semantically relevant summaries that effectively describe the intended function of the software. in the dynamic realm of software development and maintenance, an adept grasp of program functionalities is of paramount importance. in this context, the integration of natural language summaries derived from source code emerges as a potent instrument, streamlining developers' efforts and augmenting program comprehension. while current state-of-the-art code summarization models are developed and evaluated on clean and curated datasets, the real-world coding environment is far from standardized. developers often deviate from standard coding practices, leading to inconsistent naming conventions. additionally, actual codebases often feature commented sections, serving as legacy code or reserved for future use cases. our research aims to simulate these realworld scenarios and assess whether models truly * equal contribution. the prevailing approaches to code summarization typically employ an encoder-decoder framework, encompassing the conversion of code into a hidden space and its subsequent transformation into natural language. for instance, codet5 @xcite , a unified pretrained encoder-decoder transformer model, leverages the semantics encoded in identifiers. in this research, we investgate the effectiveness of these models by tweaking the function and variable names in the existing code summarization datasets. furthermore, we introduce additional challenges, such as commented code and dead code, to elevate the complexity of data samples and scrutinize the models' summarization processes. dead code refers to unreachable code segments, devoid of functional importance, which language interpreters (e.g., python and javascript) ignore. we seek to evaluate whether models effectively disregard such code segments. all our experiments are reproducible and we will release our code and data upon publication. the driving motivation behind this research lies in enhancing code comprehension and reducing the efforts entailed in software development and maintenance. by unraveling how language models comprehend code, we aim to contribute insights that pave the way for more effective software development practices. our study, through experimentation and analysis, strives to provide valuable directions for improving the capabilities of language models in understanding and summarizing code, despite the challenges posed by real-world coding scenarios. * ## related work automated code summarization is a useful tool for software developers and has been an active re-serach field for quite some time. recently large language models have shown significant improvements in natural language tasks. inspired by this, several pretrained language models have been developed for the programming language tasks. the encoder-decoder models have been found to be more successful in programming language (pl) tasks, whereas fully decoder models perform significantly better in natural language (nl) domain. models like codebert @xcite , plbart @xcite , @xcite , codet5 @xcite , cotext @xcite have shown impressive performances in the codexglue @xcite benchmark. unlike natural language, it's necessary to capture the rich code semantics in the programming language. most programming language models (plms) in this domain are pretrained on a large corpus of nl-pl pair in several programming languages with a masked token prediction objective. to capture the code semantics, various models have used different approaches. for example, codet5 uses an additional masked identifier prediction and graphcodebert incorporates the data flow extracted from the code. these plms have shown impressive results in downstream tasks like code summarization. ahmed and devanbu (2022a) explored the code summarization in project-specific domain. @xcite used an extractive and abstractive framework from source code summarization. @xcite showed that multilingual training can amply performance for low resource languages in different downstream tasks including code summarization. @xcite provided further insights for low resource languages like ruby. as indicated by @xcite in graphcode-bert, indicators play a key role in code summarization. however, it's more desirable that our model relies more on code semantics and syntax rather than method names and identifiers. developers follow their own naming conventions which can affect the model performance. in a closely related work, @xcite have shown that semantic preserving transformations like removing code comments, replacing function names and local variable names to generic names significantly affects the bleu score of the models like plbart. we want to extend this exploration to other models like codet5, codebert etc. we also aim to increase the scope of semantic preserving transfor-mations by including dead code and commented code to check the model's understanding of the code. ## dataset there are several different code summarization datasets available. but we prefered codexglue @xcite over others for these reasons -• codexglue has been meticulously deduplicated, as demonstrated in @xcite . this ensures that any duplication within the dataset does not artificially inflate performance metrics. we can categorize the six languages available in codexglue into three groups based on the size of their combined datasets (including train, validation, and test sets). please note that the following information pertains to the combined dataset size: to analyze the impact of data transformation across resource categories, we selected one language from each category. therefore, for our experiments, we utilized the languages python, java, and javascript. the detailed statistics about train, validation and test splits is presented in ## experiments for all our experiments, we use the nvidia tesla m40 gpus. while finetuning for all types of data (clean, corrupted and combined), we used the clean data finetuning hyperparameters that were available in the codet5 repository. however, for codet5 base model, we had to reduce the batch size from 48 to 16 to fit in our gpu. all other hyperparameters remain the same. with this setup, we perform the following experiments. ## discussion our experiments try to answer the following essential questions on models' understandability. for all languages and models, the performance of a model trained on clean data tends to diminish when faced with corrupted test data, as compared to its performance on clean test data. the drop in performance for all models and languages is at least 4 points in terms of bleu score. this phenomenon may be attributed to the model's reliance on textual hints found within function and variable names during training, rather than grasping the true essence of the code's functionality and achieving generalization. the comparison is visually shown in figure 2 . 6.2 research question 2: does the model trained on identifier corrupted data perform well on clean data? a surprising observation arises when examining the performance of the model trained only on corrupted data. it demonstrates a commendable level of proficiency not only on corrupted test data (which is expected) but also on clean test data. this is visually explained in figure 3 . for all languages and models, the performance on clean test data between the model trained on clean data and corrupted data is less than 1 point in terms of bleu score, except for codebert in javascript. we hypothesize that when we train on the corrupted data, the model is forced to understand the code functionality in a generalized manner, thereby enabling it to perform well even in the clean dataset. 6.3 research question 3: how is the performance of the model trained on the combined data? our combined dataset contained both clean data and identifier corrupted data. the model trained on a combined dataset exhibits impressive performance not only on clean data but also on identifier corrupted data. its performance on the clean test data is very similar to and sometimes even surpasses the performance of the model only trained on the clean data. similar observations are seen for the corrupted test data. notably, this pattern is consistent across all the models and languages. this shows that if we curate our dataset correctly, the model can generalize across clean and corrupted datasets. the bleu score performance comparisons are visually explained in figures 4 and 5 . we observe that the model trained on a dataset consisting of commented code showcases comparable and impressive performance on both clean code data and commented code data. however, a model trained exclusively on clean data displays satisfactory performance on the clean test set, albeit experiencing a notable decline in performance when evaluated on the commented code test set (presented in figure 6 ). a potential explanation is that the absence of comments in the clean code training data prevents the model from learning the syntax associated with comments. however, when the model is trained on code that includes comments, it captures the code syntax information despite the comments not directly influencing the code's functionality. upon analysis, we determine that a model trained on a dataset that includes non-functional dead code demonstrates impressive and similar performance when applied to both clean code and the aforementioned dead code. we make the same observation for a model trained on clean data and evaluated on both clean and non-functional code test datasets. we thus conclude that the addition of dead code doesn't have any significant impact on the generated summaries. ## human evaluation in order to gain deeper insights into specific errors, a manual evaluation was conducted on a randomly selected subset of 200 examples from the identifier corrupted dataset. the evaluation compared the performance of two codet5 base models: the model trained on the combined dataset (including identifier corrupted codes) and the baseline model trained on clean data. the evaluation involved analyzing code, gold truth, baseline model summaries, and combined data trained model summaries. we prepare one such set for both python and javascript (refer language clean data model combined data model ties total the evaluation process included two individuals independently annotating the same set of data and marking the annotations as prediction 1, prediction 2, or tie (both models). the chosen option implies that the selected annotation is more closely aligned with the gold truth and code. the inter-annotator agreement can be observed in the in cases where there was a disagreement between the annotators, ties were resolved using the following strategies: the observations revealed that the summaries generated by the model trained on the combined data were more relevant to the code and closer to the desired outcome (gold truth) compared to the baseline model. there were several notable issues with the summaries generated by the baseline model, which are discussed in the figure 7 . the computed bleu scores for the two models, where one was trained on clean data and the other on combined data, are as follows: for python, the bleu scores on the identifier corrupted dataset are 12.92 and 16.17, while for javascript, the scores are 9.04 and 12.86, respectively. these scores align with the manual evaluation results, indicating that the model trained on combined data outperforms the model trained solely on clean data in code summarization. ## conclusion by studying advanced code summarization models, we discover how making changes that preserve the meaning of the code affects the quality of the summaries they generate. additionally, we provide evidence that if we train the large language models like codet5 properly, making changes that disrupt the meaning of function and variable names have little impact on the resulting summaries. our observations remain consistent across three distinct programming languages: java, python, and javascript. these findings raise important concerns about how well these models truly understand code, highlighting the need for better training methods and carefully curated datasets that improve their understanding. we propose using different types of code transformations, such as introducing renamed identifiers, adding comments, or dead code, as ways to enhance the training of these models. furthermore, there is an exciting opportunity to apply these findings to different programming languages, so that we can learn more about their general applicability.