Impact of Evaluation Methodologies on Code Summarization (2022)

Abstract: There has been a growing interest in developing machine learning (ML) models for code summarization tasks, e.g., comment generation and method naming. Despite substantial increase in the effectiveness of ML models, the evaluation methodologies, i.e., the way people split datasets into training, validation, and test sets, were not well studied. Specifically, no prior work on code summarization considered the timestamps of code and comments during evaluation. This may lead to evaluations that are inconsistent with the intended use cases. In this paper, we introduce the time-segmented evaluation methodology, which is novel to the code summarization research community, and compare it with the mixed-project and cross-project methodologies that have been commonly used. Each methodology can be mapped to some use cases, and the time-segmented methodology should be adopted in the evaluation of ML models for code summarization. To assess the impact of methodologies, we collect a dataset of (code, comment) pairs with timestamps to train and evaluate several recent ML models for code summarization. Our experiments show that different methodologies lead to conflicting evaluation results. We invite the community to expand the set of methodologies used in evaluations.

## introduction over the last several years, there has been a growing interest in applying machine learning (ml) models to code summarization tasks, such as comment generation @xcite @xcite @xcite @xcite @xcite @xcite @xcite and method naming @xcite @xcite . substantial progress has been reported over years, usually measured in terms of automatic metrics @xcite . despite a solid progress in generating more accurate summaries, the evaluation methodology, i.e., the way we obtain training, validation, and test sets, is solely based on conventional ml practices in natural language summarization, without taking into account the domain knowledge of software engineering and software evolution. for example, temporal relations among samples in the dataset are important because the style of newer code summaries can be affected by older code summaries; however, they are not explicitly modeled in the evaluation of code summarization in prior work, which assumed the samples in the dataset are independent and identically distributed. this gap could lead to inflated values for automatic metrics reported in papers and misunderstanding if a model might actually be useful once adopted. the key missing piece in prior work is the description of the targeted use cases for their ml models. prior work has implicitly targeted only the batch-mode use case: applying the model to existing code regardless of when the code is written. however, a more realistic scenario could be the continuous-mode use case: training the model with code available at a timestamp, and using the model on new code after that timestamp (as illustrated in figure 1 ). considering that programming languages evolve and coding styles are constantly revised, results obtained in batch-mode could be very different from those obtained in continuousmode. thus, it is insufficient to only report the task being targeted in a paper, and it is necessary to explain intended use cases for the ml models. once the task and use cases are clearly defined, an appropriate evaluation methodology (or potentially several methodologies) should be used. in this paper, we study recent literature on ml models for code summarization. by reasoning about their evaluation methodologies (which we call mixed-project and cross-project), we define two use cases that could be evaluated by these methodologies. next, we define a more practical use case when a developer uses a fixed model continuously over some period of time. we describe an appropriate evaluation methodology for this use case: time-segmented. finally, we evaluate several existing ml models using the three methodologies. we highlight two key findings. first, depending on the employed methodology we end up with conflicting conclusions, i.e., using one methodology, model a is better than model b, and using another methodology, model b is better than model a. second, our results show that the absolute values for automatic metrics vary widely across the three methodologies, which indicates that models might be useful only for some use cases but not others. thus, it is imperative that future work describes what use case is being targeted and use the appropriate evaluation methodology. in summary, this paper argues that we need to more diligently choose evaluation methodology and report results of ml models. regardless of whether or not the conclusions of prior work hold across methodologies, we should always choose the methodology appropriate for the targeted task and use case. we hope the community will join us in the effort to define the most realistic use cases and the evaluation methodology for each use case. we hope that our work will inspire others to design and formalize use cases and methodologies for other tasks. only a few research studies on defect prediction @xcite @xcite , program repair @xcite , and bug localization @xcite took into consideration software evolution when evaluating ml models. taking software evolution into account in those tasks appears more natural, but is not more important than in code summarization. moreover, for the first time, we present an extensive list of potential use cases and evaluation methodologies side-by- training validation test project 1 project 2 project 3 project n-1 project n . . . training validation test project 1 project 2 project 3 project m project n . . . . . . side, as well as the impact of choosing various methodologies on the performance of ml models. our code and data are available at https: //github.com/engineeringsoftware/ time-segmented-evaluation. ## methodologies we first summarize two commonly used methodologies: mixed-project ( §2.1) and cross-project ( §2.2). then, we introduce a novel time-segmented methodology ( §2.3). we will use τ -2 < τ -1 < τ to denote specific points in time (i.e., timestamps). ## use cases methodologies are used to set up experiments and obtain an appropriate dataset split for the evaluation. however, they do not describe the envisioned usage of an ml model. prior work picked a methodology in order to set up experiments, but we argue that ml models should be described with respect to use cases, i.e., how will the developers use the models eventually. once a use case is chosen, an appropriate methodology can be selected to evaluate the model. in this section, we define three use cases via examples of the comment generation task. the first two use cases are "extracted" from prior work. namely, we reason about the mixed-project and the cross-project methodologies used in prior work and try to link each to a (somewhat) realistic use case. the third use case is inspired by our own development and can be evaluated using the time-segmented methodology. note that we do not try to provide an exhaustive list of use cases, but rather to start off this important discussion on the distinction between a use case and an evaluation methodology. for the simplicity of our discussion, we only focus on the training and test sets (since the validation set can be regarded as the "open" test set for tuning). ## application of methodologies we describe the steps to apply the methodologies following their definitions ( §2) with a given dataset, as illustrated in figure 4 . the input dataset contains samples with timestamps, and the outputs include: a training and validation set for each methodology to train models; a standard test set for each methodology to evaluate the models for this methodology only; and a common test set for each pair of methodologies to compare the same models on the two methodologies. appendix a presents the formulas of each step. step 1: time-segment. see figure 4 top left part. a project is horizontally segmented into three parts by timestamps τ -2 and τ - foot_0 . step 2: in-project split. see figure 4 top right part. a project is further vertically segmented into three parts randomly, which is orthogonal to the time segments in step 1. step 3: cross-project split. see figure 4 middle part. projects are assigned to training, validation, and test sets randomly, which is orthogonal to the time segments and in-project splits in step 1 and 2. step 4: grouping. now that the dataset is broken down to small segments across three dimensions (time, in-project, and cross-project), this step groups the appropriate segments to obtain the training (train), validation (val), and standard test (tests) sets for each methodology. this is visualized in figure 4 bottom left part. step 5: intersection. the common test (testc) set of two methodologies is the intersection of their tests sets. this is visualized in figure 4 bottom right part. in theory, we could compare all three method-ologies on the intersection of the three tests sets, but in practice, this set is too small (far less than 4% of all samples when we assign 20% projects and 20% samples in each project into test set). ## experiments we run several existing ml models using different methodologies to understand their impact on automatic metrics, which are commonly used to judge the performance of models. ## conclusion we highlighted the importance of specifying targeted use cases and adopting the correct evaluation methodologies during the development of ml models for code summarization tasks (and for other software engineering tasks). we revealed the importance of the realistic continuous-mode use case, and introduced the time-segmented methodology which is novel to code summarization. our experiments of comparing ml models using the time-segmented methodology and using the mixedproject and cross-project methodologies (which are prevalent in the literature) showed that the choice of methodology impacts the results and findings of the evaluation. we found that mixed-project tends to over-estimate the effectiveness of ml models, while the cross-project may under-estimate it. we hope that future work on ml models for software engineering will dedicate extra space to document intended use cases and report findings using various methodologies. our dataset has been collected in a manner that is consistent with the licenses provided from the sources (i.e., github repositories). the evaluation methodologies described in our study is expected to assist researchers in evaluating and reporting ml models for code summarization, and assist software developers (i.e., users of those models) in understanding the reported metrics and choosing the correct model that fits their use case. our work can be directly deployed in code summarization research, and can potentially generalize to other software engineering areas using ml models ( §6.1). we expect our work to help researchers build ml models for code summarization (and other se areas) that are more applicable to their intended use cases. we do not claim that the methodologies and use cases described in our study are the most realistic ones, nor do we try to provide an exhaustive list of them. in particular, the continuous-mode use case ( §3.3) is inspired by our own observations during using and developing ml models for code summarization. we try our best to design this use case to reflect the most common and realistic scenarios, but other use cases may be more valid in certain scenarios ( §6.2). we conducted experiments involving computation time/power, but we have carefully chosen the number of times to repeat the experiments to both ensure reproducibility of our research and avoid consuming excessive energy. we provided details of our computing platform and running time in appendix d.