Automatic Metrics in Natural Language Generation: A survey of Current Evaluation Practices (2024)

Abstract: Automatic metrics are extensively used to evaluate Natural Language Processing systems. However, there has been increasing focus on how the are used and reported by practitioners within the field. In this paper, we have conducted a survey on the use of automatic metrics, focusing particularly on natural language generation tasks. We inspect which metrics are used as well as why they are chosen and how their use is reported. Our findings from this survey reveal significant shortcomings, including inappropriate metric usage, lack of implementation details and missing correlations with human judgements. We conclude with recommendations that we believe authors should follow to enable more rigour within the field.

## introduction evaluation practices in the field of natural language processing (nlp) are increasingly coming under a microscope by researchers. there is now a significant body of contributions presenting experimental research, meta-analyses and/or best practice guidelines, on issues ranging from statistical significance testing @xcite , to human evaluation methods @xcite @xcite , error analysis @xcite and replicability of evaluations @xcite . automatic metrics and their usage for evaluation have also been under extensive examination by researchers. similarity-based metrics are sometimes taken as proxies for human quality ratings, whereas findings suggest the two should not be conflated. this has lead to concerns about metric validity @xcite . for example, the validity of metrics such as bleu @xcite and @xcite has been put into question regarding their poor correlation with human judgements @xcite @xcite . in addition, automatic metrics do not capture factuality or faithfulness issues in text @xcite , such as incorrect names and numbers @xcite . interpreting the meaning of scores generated by automatic metrics can also be challenging. for example, what researchers often report as a "bleu score" actually consists of several metrics, depending on multiple parameters and varying across different implementations, which are not compatible with each other @xcite . there are also questions on whether it is possible to encapsulate the performance of a given system with a single number or whether the use of a single metric to demonstrate improvements over prior systems provides sufficient dimensionality in reporting the performance characteristics of a given system @xcite . given the well-documented shortcomings of automatic metrics, our goal in this paper is to survey the current state of play in metric-based evaluations of natural language generation (nlg). as with the above-mentioned studies focusing on other facets of evaluation, we aim to both understand how metrics are currently used in nlg, and to identify gaps and possible ways forward in an effort to improve the scientific quality of nlg research. specifically, we conduct an analysis of published work in the field, annotating which metrics are used, for what purposes, and how their usage is reported. in section 2, we describe past survey efforts within the field of nlg to frame our contribution. in section 3, we describe our paper selection procedure, the annotation procedure, the challenges we encountered, and the process and results of comput-ing inter-annotator agreement between the annotators. the analysis and results from the annotation process are presented in section 4, and we offer our insights into these results in section 5. finally, we wrap up with recommendations (section 6) and concluding thoughts (section 7) from the observations based on our results. ## evaluation surveys in nlg there have been several surveys inspecting the different aspects of evaluation practices within nlg over the last several years. some surveys focused on quantifying the types of evaluations, the proportion of intrinsic and extrinsic evaluations over a defined period of time either for the field as a whole @xcite , or for a specific domain such as question generation @xcite . in addition, there has been an effort to understand the different types of metrics and evaluation approaches employed and to categorise the challenges faced by researchers @xcite . in addition to survey work covering shortcomings of automatic metrics @xcite , a significant amount of work has focused on human evaluation practices within nlg. past work has revealed a large variation in practices among researchers @xcite . this was followed up by an extensive survey which has shown that in addition to the large variety of practices, there are fundamental gaps in reported details by authors @xcite . these issues have led to proposals for best practices for carrying out and reporting human evaluations in nlg (van der @xcite . however, the concern about human evaluation practices has also led researchers to consider whether human evaluations in nlg -and in nlp as a whole -are both reproducible and repeatable @xcite given the inconsistencies and gaps in reporting practices. one area where reporting practices have received attention is the way in which errors made by nlg systems are documented. @xcite found that there is severe under-reporting of the different kinds of errors a given nlg system can make, which leaves the broader community "in the dark" due to this missing information. beyond evaluations and reporting practices, there have been attempts to better understand the motivations of researchers and their reporting practices by directly surveying them. @xcite found that there is pressure towards a "kitchen sink" approach for evaluation. even though researchers recognise the limitations of existing metrics, lack of clarity about their evaluation goals and quality criteria can lead to over-reporting potentially uninformative metrics @xcite . other work explored the barriers that researchers face to conducting error analyses @xcite : while respondents were generally positive about error analyses, there are multiple barriers such as page limits, lack of tools or resources, and a lack of time and/or money. ## survey method although past surveys looked at the deficiencies of automatic metrics, none of them go beyond quantifying and aggregating their usage. this is necessary, considering that the use of automatic metrics increased by almost 25% in the 2016-2019 period, with some surveys reporting that almost half of the papers surveyed only use automatic metrics @xcite . to obtain a comprehensive and up-to-date view of current practices in automatic evaluation for nlg, we have focused on recently published articles in prominent, peerreviewed venues. paper selection our analysis is based on a snapshot of a total of 110 papers presented in 2023 in two relevant venues: the international conference on natural language generation (inlg) and the annual meeting of the association for computational linguistics (acl). all papers (@xmath0, of which 26 are long papers) at the main conference track of inlg 2023 were included. for acl, we used all the papers presented under the generation track (@xmath1, 63 are long papers). in addition to regular acl papers, this included three papers originally accepted for publication in the transactions of the acl (tacl) journal and one nlg paper from the journal computational linguistics. annotation procedure papers were randomly distributed among all the authors in a set of annotation batches, and independently annotated for the features summarised in crucially, we also consider whether a metric is newly introduced in a paper or was previously published. in either case, we are interested in whether authors describe the rationale for their use of a metric. in case a paper included a human evaluation, we also annotate whether metric-based evaluations were quantitatively correlated with the outcomes of the human evaluation, or whether there was any qualitative discussion of the relationship between the two. iterative refinement and inter-annotator agreement annotation proceeded in multiple rounds. during an initial round, we independently annotated a subset of papers and discussed the outcomes to fine-tune the annotation scheme. subsequently, a random sample of 4 papers (2 from inlg; 2 from acl) was selected and independently annotated by 6 of the authors. inter-annotator agreement (iaa) for the features outlined in ## analysis and results we present the results of our annotation of 110 papers in this section. out of the 110 papers annotated, a total of 102 papers included an evaluation of a generation system. the excluded 8 papers did not propose any systems to be evaluated. for example, they either presented a corpus or methods to detect the decoding algorithm of a closed-source model. after the removal of these papers, a total of 69 acl papers and 33 inlg papers were analysed. a total of 59 papers (56.73%) of papers use human evaluations; in contrast, 98 papers (94.23%) used automatic metrics, a result similar to what was found by van der @xcite , who reported 95% of papers using automatic metrics in both acl tracks and inlg. there were only 53 papers (50.96%) containing both automatic and human evaluation. another aspect explored was whether authors provide any implementation details, such as link to the specific implementation used for the evaluation. we found that for 66.2% of inlg and 57.3% of acl papers, these details were not mentioned either in the main body of the paper or within the appendices. given the high percentage of papers not giving specific implementation details, this can make it difficult to conduct reproduction studies under the same conditions, especially, considering how challenging it is to reproduce the original scores of nlp evaluations @xcite . in the subsequent sections, we will explore in more detail how specific metrics are used (sec- ## discussion our survey reveals both positive and negative aspects of current trends in nlg evaluation. undoubtedly positive is the fact that the vast majority of researchers do make their code and resources available after publication, despite no obligation to do so. additionally, it is encouraging to see that types of metrics used differ given the task, suggesting an effort to use metrics which are relevant to the research goals. overlap metrics are mostly complemented by metrics from other categories (cf. figure 9 in the appendix). on the other hand, the predominance of overlap metric types is concerning given their well-known caveats, such as their inability to measure faithfulness and poor correlation with human judgements @xcite . this is also compounded by the tendency to not state the rationale for the use of a metric. without any rationale of why a given metric or set of metrics are being used, there is uncertainty on what researchers are looking to measure and whether they chose the right metrics. our survey also reveals an over-reliance on reference-based metrics. this might be a hold-over from when generation tasks were more highly constrained and focused on more closed-domain problems such as weather forecast generation, with a defined set of reference "gold-standard" corpora. however, most generation problems are increasingly open-ended and require accepting a wider range of outputs that are not possible to cover in a given reference set. therefore, it is possible that an attitudinal or structural change is needed within the research community to ask deeper questions on the use of inappropriate metrics. another observation is the relationship between automatic and human evaluations. out of the metrics that had no rationale provided, around half performed human evaluations, yet did not investigate any link between the automatic and human evaluation results. this suggest that the majority of researchers treat their evaluations as separate entities. however, given the overall lack of rationales provided for the use of automatic metrics, we cannot be certain that authors are looking to measure different aspects with their automatic and human evaluations or whether the evaluations are in fact intended to be complementary. ultimately, this creates uncertainty for researchers reading papers and makes the reproduction of evaluations challenging. ## recommendations 6.1 evaluation quality rationalize your selection of metrics authors should consider the appropriateness of the metrics they are using and whether adding more automatic metrics will in fact yield interesting insights. in particular, we advise authors to state clearly what they expect to evaluate with each given metric so that there is clarity for those trying to interpret reported results. in our investigation, we found that less than 13% of metric occurrences are supported by a rationale other than following previous work. rationales are also important due to the number of metrics used -283 unique metrics were used at the surveyed venues last year. we cannot reasonably expect readers to be familiar with all of them, which strengthens the need for justification. do not copy-paste widely used metrics we found that around 10% of metric usages (and an unknown portion of the 77% with no rationale provided) are justified on the grounds that they follow evaluations done in previous work. authors should question whether these metrics truly measure the intended qualities in the evaluation, and if they do, the authors should share their reasoning in the paper. however, if the metrics fail to show a correlation with human judgment or a specific quality, we strongly advise authors to omit them, or at least relegate them into the appendix to clearly show their decreased priority. comment on metric combinations given that automatic metrics frequently have blind spots, we also recommend commenting on the chosen combination of metrics: how do the metrics complement each other to provide a more objective evaluation of a system? respect the intended use of metrics generally, when a new metric is proposed, its authors demonstrate its suitability for a given setting or task. however, we frequently see metrics used for purposes that they were not intended for. in such a case, the authors should justify their use of the metric from first principles or empirically. ## conclusion we have presented our analyses and a new dataset of 102 papers annotated with nine attributes to ascertain the different metrics, used currently by authors in nlg across publications in 2023 in both inlg and acl venues. the process of creating and validating the annotation schema, the analyses that we have conducted, and the results we have obtained are described in this paper. from the results that we have obtained, we have shown that there are outstanding issues related to the type and number of metrics used, the lack of comparison and linkage between automatic and human evaluation results, and missing justifications for the selection of metrics. we have proposed several recommendations in the hope to offer possible solutions to these structural problems. however, while many papers have or will make recommendations on improving evaluation practices, it is only when these solutions are adopted that we as a research field can make progress on these issues. our main conclusion is, that as a field, we need to provide more information on the usage of automatic metrics and the motivations behind their usage. only by doing this can we start to bring more clarity to how evaluations are being conducted and help to alleviate adjacent challenges such as the reproduction and repeatability of evaluations.