A global analysis of metrics used for measuring performance in natural language processing (2022)

Abstract: Measuring the performance of natural language processing models is challenging. Traditionally used metrics, such as BLEU and ROUGE, originally devised for machine translation and summarization, have been shown to suffer from low correlation with human judgment and a lack of transferability to other tasks and languages. In the past 15 years, a wide range of alternative metrics have been proposed. However, it is unclear to what extent this has had an impact on NLP benchmarking efforts. Here we provide the first large-scale cross-sectional analysis of metrics used for measuring performance in natural language processing. We curated, mapped and systematized more than 3500 machine learning model performance results from the open repository ‘Papers with Code’ to enable a global and comprehensive analysis. Our results suggest that the large majority of natural language processing metrics currently used have properties that may result in an inadequate reflection of a models’ performance. Furthermore, we found that ambiguities and inconsistencies in the reporting of metrics may lead to difficulties in interpreting and comparing model performances, impairing transparency and reproducibility in NLP research.

## introduction benchmarking, i.e., the process of measuring and comparing model performance on a specific task or set of tasks, is an important driver of progress in natural language processing (nlp). benchmark datasets are conceptualized as fixed sets of data that are manually, semi-automatically or automatically generated to form a representative sample for these specific tasks to be solved by a model. a model's performance on such a benchmark is then assessed based on a single or a small set of performance metrics. while this enables quick comparisons, it may entail the risk of conveying an incomplete picture of model performance since metrics inherently condense performance to a single number, omitting certain performance aspects completely or balancing trade-offs between different aspects (e.g. accuracy vs. fluency). additionally the capacity of metrics to capture performance may differ strongly between tasks and languages. capturing model performance in a single metric is an inherently difficult task, and this is further aggravated in the nlp domain by the structural and semantic complexity of human language. traditionally used nlp metrics such as bleu or rouge, originally devised for machine translation and summarization, were shown to suffer from low correlation with human judgment and poor transferability to other tasks @xcite @xcite @xcite . these fundamental problems are increasingly recognized by the nlp community-e.g., metric evaluation was even introduced as an independent task at the annual machine translation conference @xcite . in the past 15 years, a wide variety of superior metrics for evaluating models on nlp tasks have been proposed, including task-agnostic, ai-based metrics such as bertscore @xcite @xcite . however, it is unknown to what extent this had an impact on metrics used in nlp research. we aim to address this question by providing a global analysis of performance measures used in nlp benchmarking. our contributions are threefold: (1) we curated, mapped and systematized performance metrics covering more than 3500 performance results from the open repository 'papers with code' to enable a global and comprehensive analysis. (2) based on this dataset, we provide a cross-sectional analysis of the prevalence of performance measures in the subset of natural language processing benchmarks. (3) we describe inconsistencies and ambiguities in the reporting and usage of metrics, which may lead to difficulties in interpreting and comparing model performances. ## discussion nlp covers a wide range of different tasks and thus shows a large diversity of utilized metrics. we limited our analysis to more complex nlp tasks beyond simple classification, such as machine translation, question answering, and summarization. metrics designed for these tasks generally aim to assess the similarity between a machine-generated text and a reference text or set of reference texts that are human-generated. we found that, despite their known shortcomings, the bleu score and rouge metrics continue to be the most frequently used metrics for such tasks. several weaknesses of bleu have been pointed out by the research community, such as its sole focus on n-gram precision without considering recall and its reliance on exact n-gram matchings. zhang et al. have discussed properties of the bleu score and nist, a variant of the bleu score that gives more weight to rarer n-grams than to more frequent ones, and came to the conclusion that neither of the two metrics necessarily show high correlation with human judgments of machine translation quality @xcite . the rouge (recall-oriented understudy for gisting evaluation) metrics family was the second most used nlp-specific metric in our dataset after the bleu score. while originally proposed for summarization tasks, a subset of the rouge metrics (i.e., rouge-l, rouge-w and rouge-s) has also been shown to perform well in machine translation evaluation tasks @xcite . however, the rouge metrics set has also been shown to not adequately cover multi-document summarization, tasks that rely on extensive paraphrasing, such as abstractive summarization, and extractive summarization of multi-logue text types (i.e., transcripts with many different speakers), such as meeting transcripts @xcite @xcite . several new variants have been proposed in recent years, which make use of the incorporation of word embeddings (rouge-we), graph-based approaches (rouge-g), or the extension with additional lexical features (rouge 2.0) @xcite @xcite . rouge-1, rouge-2 and rouge-l were the most common rouge metrics in our analyzed dataset, while newer proposed rouge variants were not represented. meteor (metric for evaluation of translation with explicit ordering) was proposed in 2005 to address weaknesses of previous metrics @xcite . meteor is an f-measure derived metric that has repeatedly been shown to yield higher correlation with human judgment across several tasks as compared to bleu and nist @xcite @xcite . matchings are scored based on their unigram precision, unigram recall (given higher weight than precision), and a comparison of the word ordering of the translation compared to the reference text. this is in contrast to the bleu score, which does not take into account n-gram recall. furthermore, while bleu only considers exact word matches in its scoring, meteor also takes into account words that are morphologically related or synonymous to each other by using stemming, lexical resources and a paraphrase table. additionally, me-teor was designed to provide informative scores at sentence-level and not only at corpus-level. an adapted version of meteor, called meteor++ 2.0, was proposed in 2019 @xcite . this variant extends meteor's paraphrasing table with a large external paraphrase database and has been shown to correlate better with human judgement across many machine translation tasks. compared to bleu and rouge, meteor was rarely used as a performance metric (8%) across the nlp benchmark datasets included in our dataset. the gleu score was proposed as an evaluation metric for nlp applications, such as machine translation, summarization and natural language generation, in 2007 @xcite . it is a support vector machine-based metric that uses a combination of individual parser-derived metrics as features. gleu aims to assess how well the generated text conforms to 'normal' use of human language, i.e., its 'fluency'. this is in contrast to other commonly used metrics that focus on how well a generated text reflects a reference text or vice versa. gleu was reported only in 1.8% of nlp benchmark datasets. additional alternative metrics that have been proposed by the nlp research community but do not appear as performance metrics in the analyzed dataset include translation error rate (ter), ter-plus, "length penalty, precision, n-gram position difference penalty and recall" (lepor), sentence mover's similarity, and bertscore. figure 1 depicts the timeline of introduction of nlp metrics and their original application. ter was proposed as a metric for evaluating machine translation quality. ter measures quality by the number of edits that are needed to change the machine-generated text into the reference text(s), with lower ter scores indicating higher translation quality @xcite . ter considers five edit operations to change the output into the reference text: matches, insertions, deletions, substitutions and shifts. an adaptation of ter, ter-plus, was proposed in 2009. ter-plus extends ter with three additional edit operations, i.e., stem matches, synonym matches and phrase substitution @xcite . ter-plus was shown to have higher correlations with human judgements in machine translation tasks than bleu, meteor and terp @xcite . lepor and its variants hlepor and nlepor were proposed as a language-independent model that aims to address the issue that several previous metrics tend to perform worse on languages other than those it was originally designed for. it has been shown to yield higher correlations with human judgement than meteor, bleu, or ter @xcite . sentence mover's similarity (sms) is a metric based on elmo word embeddings and earth mover's distance, which measures the minimum cost of turning a set of machine generated sentences into a reference text's sentences @xcite . it was proposed in 2019 and was shown to yield better results as compared to rouge-l in terms of correlation with human judgment in summarization tasks. bertscore was proposed as a task-agnostic performance metric in 2019 @xcite . it computes the similarity of two sentences based on the sum of cosine similarities between their token's contextual embeddings (bert), and optionally weighs them by inverse document frequency scores @xcite . bertscore was shown to outperform established metrics, such as bleu, me-teor and rouge-l in machine translation and image captioning tasks. it was also more robust than other metrics when applied to an adversarial paraphrase detection task. however, the authors also state that bertscore's configuration should be adapted to task-specific needs since no single configuration consistently outperforms all others across tasks. difficulties associated with automatic evaluation of machine generated texts include poor correlation with human judgement, language bias (i.e., the metric shows better correlation with human judgment for certain languages than others), and worse suitability for language generation tasks other than the one it was proposed for @xcite . in fact, most nlp metrics have originally been con-ceptualized for a very specific application, such as bleu and meteor for machine translation, or rouge for the evaluation of machine generated text summaries, but have since then been introduced as metrics for several other nlp tasks, such as question-answering, where all three of the above mentioned scores are regularly used. non-transferability to other tasks has recently been shown by chen et al. who have compared several metrics (i.e., rouge-l, meteor, bertscore, bleu-1, bleu-4, conditional bertscore and sentence mover's similarity) for evaluating generative question-answering (qa) tasks based on three qa datasets. they recommend that from the evaluated metrics, meteor should preferably be used and point out that metrics originally introduced for evaluating machine translation and summarization do not necessarily perform well in the evaluation of question answering tasks @xcite . many nlp metrics use very specific sets of features, such as specific word embeddings or linguistic elements, which may complicate comparability and replicability. to address the issue of replicability, reference open source implementations have been published for some metrics, such as, rouge, sentbleu-moses as part of the moses toolkit and sacrebleu @xcite . in summary, we found that the large majority of metrics currently used to report nlp research results have properties that may result in an inadequate reflection of a models' performance. while several alternative metrics that address problematic properties have been proposed, they are currently rarely used in nlp benchmarking. our findings are in line with a recent, focused meta-analysis on machine translation conducted by @xcite found that 82.1% of papers report bleu as the only performance metric despite its well-known shortcomings @xcite . our analysis extends these findings by providing a global overview of metrics used in the entire nlp domain. ## conclusions the reporting of metrics was partly inconsistent and partly unspecific, which may lead to ambiguities when comparing model performances, thus negatively impacting the transparency and reproducibility of nlp research. large comparative evaluation studies of different nlp-specific metrics across multiple benchmarking tasks are needed.