Evaluating Evaluation Metrics: A Framework for Analyzing NLG Evaluation Metrics using Measurement Theory (2023)

Abstract: We address a fundamental challenge in Natural Language Generation (NLG) model evaluation—the design and evaluation of evaluation metrics. Recognizing the limitations of existing automatic metrics and noises from how current human evaluation was conducted, we propose MetricEval, a framework informed by measurement theory, the foundation of educational test design, for conceptualizing and evaluating the reliability and validity of NLG evaluation metrics. The framework formalizes the source of measurement error and offers statistical tools for evaluating evaluation metrics based on empirical data. With our framework, one can quantify the uncertainty of the metrics to better interpret the result. To exemplify the use of our framework in practice, we analyzed a set of evaluation metrics for summarization and identified issues related to conflated validity structure in human-eval and reliability in LLM-based metrics. Through MetricEval, we aim to promote the design, evaluation, and interpretation of valid and reliable metrics to advance robust and effective NLG models.

## introduction evaluation metrics provide quantitative assessments to guide model development, benchmark scientific progress, and inform generalizability across tasks and domains @xcite . effective evaluation metrics can extract valuable signals and robust evidence from model outputs that describe model capability, diagnose model failures, and compare the strengths and weaknesses of different models, allowing for more informed decision-making in real-world deployment @xcite . conversely, problematic evaluation metrics can mislead model diagnoses, development, and deployment, resulting in downstream harms figure 1 : metriceval: a framework that conceptualizes and operationalizes four main components of metric evaluation, in terms of reliability and validity to individuals and society @xcite . designing effective evaluation metrics for natural language generation (nlg) tasks has long been challenging due to the complex nature of language, open-endedness of tasks, multifaceted and context-dependent definition of language quality @xcite @xcite . most recently, the nlg evaluation challenge has been further exacerbated by the emergence of "generalpurpose" large language models (llms), further demanding evaluation methods to capture model utility for diverse downstream use cases. to address these challenges, researchers and practitioners have developed various types of nlg evaluation metrics, including word-based metrics (e.g., rouge, bleu), embedding-based metrics (e.g., bertscore, moverscore) and to end-to-end metrics (e.g., @xcite . with this increasingly rich set of evaluation metrics being pursued, we must understand how good each metric is. while researchers pointed out shortcomings of popular metrics (e.g., rouge), such as their inability to capture semantic meanings, insensitivity to perturbations, and failure to reflect real-world performance @xcite @xcite @xcite , there is a lack of principled approaches to evaluate nlg evaluation metrics, and to begin with, a lack of clear definition on what makes a metric good. some prior works have attempted to evaluate the quality of nlg evaluation metrics by their correlations with human judgments @xcite @xcite , which are deemed the gold standard for quality assessment. however, correlation with human preferences gives limited quality signals. more problematically, human evaluation data collection itself currently suffers from validation, standardization, consistency, and reproducibility issues @xcite @xcite . these issues subsequently undermine their validity as the foundation for evaluating automatic metrics. in this paper, we introduce metriceval, a theoretical framework to define the desiderata of and assess evaluation metrics by drawing from measurement theory in educational and psychological testing. based on two core concepts in measurement theory that define a "good" metric in testing individual capabilities-reliability and validity, our framework conceptualizes and operationalizes four key desiderata: metric stability, metric consistency, metric construct validity, and metric concurrent validity. we further propose a set of statistical tools to quantify these desiderata to systematically evaluate evaluation metrics. metriceval enables quantifying the standard error for each metric on a specific task which allows meaningful interpretations of the evaluation results. we demonstrate the utility of metriceval with a case study of evaluating 16 metrics, including llm-based metrics, on a summarization task. ## measurement theory originating from educational and psychological testing, measurement theory aims to inform evaluation processes that devise a coherent numerical representation of individual capabilities-for instance, evaluating a person's language proficiency through essay responses to questions. scores on these tests have direct consequences for high-stakes decisions, such as school admissions. key to measurement theory is the distinction between the observed score on a test, e.g., the score of an examinee's essay in a language proficiency exam, and the true score on the general construct @xcite ) that the test is theorized to measure, e.g., language proficiency. the gap between the observed and true scores is referred to as measurement error @xcite . measurement theory defines two sources of measurement error, random and systematic. random measurement errors are fluctuations specific to a time, place, examinee, and exam question that are transient and balance out to 0 over repeated measures, and they have direct consequences on the reliability of the evaluation process. systematic errors, on the other hand, are persistent shifts across one or more time, place, examinee, or exam questions, and they have direct consequences on test validity by producing observed scores with systematic deviations from the true score that the test purports to measure (e.g., a downward bias if the rubric on a language proficiency exam looks for specialized knowledge about a certain subject). by evaluating and identifying the source of measurement errors, the test designer could iteratively improve their test design by adding or removing test items or changing their rubric. the results can also help the evaluator to interpret a test score with caution. for example, we can quantify the uncertainty as the standard error for meaningful comparison: when the difference between two models' metric scores on a benchmark does not afford conclusions about significant differences, the evaluator may consider narrowing the confidence interval by averaging scores from repeated measurements. in short, as a safeguard to the trustworthiness of tests, measurement theory offers a conceptual framework for how the validity and reliability of a test should be formalized, evaluated, and optimized to reduce measurement error with the aid of statistical methods and tools. ## metric evaluation framework in this section, we introduce metriceval and its components that define and assess different aspects of the "goodness" of nlg evaluation metrics, inspired by the core concept of reliability and validity in measurement theory (see fig. 1 ). metriceval aims to evaluate and compare the reliability and validity of the metrics. for example, to evaluate a summarization model, one can apply referencebased metrics (e.g., rouge, bertscore), reference-free metrics (e.g., supert), or human ratings on specific output quality aspects (e.g., coherence or relevance) on the same benchmark to draw inferences about model capability. for the remainder of this section, we will illustrate our framework with this running example of evaluating summarization models with diverse metrics. it is important to note that the quality of evaluation results is also dependent on the chosen dataset and reference (for reference-based metrics), which, in nlg evaluation, are concerned with benchmark designs. measurement errors may cascade from those components to the observed score. in this work, we focus on the metric aspect and answer questions such as "giving a cnn/daily mail benchmark, whether using rouge or bertscore or human ratings offer reliable and valid evaluation results.". this is an important question given the far-reaching impact that prevalent benchmarks can have on the output of the research community. ## case study to illustrate how to apply metriceval to evaluate nlg evaluation metrics, in this section, we ran a case study on evaluating summarization metrics. as noted earlier, our evaluation focuses on the metrics and the results should be interpreted as dependent on the benchmark dataset used. we leave it for future research to explore the generalizability of the results across different benchmark datasets.