Analyzing and Evaluating Correlation Measures in NLG Meta-Evaluation (2025)

Abstract: The correlation between NLG automatic evaluation metrics and human evaluation is often regarded as a critical criterion for assessing the capability of an evaluation metric. However, different grouping methods and correlation coefficients result in various types of correlation measures used in meta-evaluation. In specific evaluation scenarios, prior work often directly follows conventional measure settings, but the characteristics and differences between these measures have not gotten sufficient attention. Therefore, this paper analyzes 12 common correlation measures using a large amount of real-world data from six widely-used NLG evaluation datasets and 32 evaluation metrics, revealing that different measures indeed impact the meta-evaluation results. Furthermore, we propose three perspectives that reflect the capability of meta-evaluation: discriminative power, ranking consistency, and sensitivity to score granularity. We find that the measure using global grouping and Pearson correlation coefficient exhibits the best performance in both discriminative power and ranking consistency. Besides, the measures using system-level grouping or Kendall correlation are the least sensitive to score granularity.

## introduction automatic evaluation metrics (e.g., bleu @xcite and @xcite ) are widely used in natural language generation (nlg) evaluation to assess the quality of content generated by a system or model for a specific task. furthermore, the evaluation of the performance of these evaluation metrics is referred to as meta-evaluation, which typically uses the correlation between the metrics and human evaluation as a crucial criterion, as human evaluation is generally considered the gold standard. however, the implementation of the correlation measure is not uniform * equal contribution. our relevant preliminary experiments show that different correlation measures can indeed lead to different meta-evaluation results. however, prior studies have rarely paid attention to the relationships and differences between different measures; instead, they often simply follow the conventional practices of related work or authoritative competitions on evaluation. some studies even do not clearly describe the correlation measure they used, particularly in terms of grouping methods, let alone explain the reasons for selecting that measure. moreover, the correlation measures used in some authoritative competitions are constantly changing. for instance, wmt22 @xcite used segment-level correlations that include three different grouping methods, whereas wmt21 @xcite and wmt23 @xcite only used one. these issues of non-transparency and inconsistency indicate that the correlation measure and meta-evaluation in nlg require more indepth understanding. on the other hand, large language models (llms) have been increasingly used in automatic evaluation, including both prompting proprietary llms for nlg evaluation @xcite @xcite and fine-tuned llm evaluators @xcite @xcite . unlike traditional continuous evaluation metrics, llm-based evaluators typically output discrete scores and can assess on different evaluation scales based on flexible requirements (e.g., 1-5, 0-100). this introduces more evaluation ties and varying degrees of the granularity or discretization of evaluation scores, which may affect the fairness of com-parisons in certain correlation measures @xcite , making the already confusing selection of correlation measures even more complex. however, it is by no means easy to strictly determine whether a particular correlation measure is reasonable, as it depends on the specific scenario and research objectives. for example, @xcite believe that when using multidimensional quality metric (mqm) @xcite for machine translation evaluation, fine-grained tied scores should be trusted and used for metaevaluation, while traditional kendall correlation coefficients cannot handle these situations and are therefore unsuitable. however, in coarser-grained evaluations (such as likert scale ratings from 1 to 5), ties may not be as reliable, rendering the above conclusion invalid. therefore, this paper focuses on the comparison and analysis of different correlation measures and their meta-evaluation capabilities, primarily around the following three and characteristic perspectives and research questions: to achieve more comprehensive and realistic analyses, we collect six commonly-used nlg evaluation datasets, including 30 different subsets, and calculate and annotate the results of 32 different automatic evaluation metrics (including evaluators using llms such as gpt-4). based on such large amounts of real data, we design specific testing algorithms to analyze the above three questions and summarize the corresponding conclusions. our contributions are summed up as follows: ## background in nlg evaluation, we usually focus on the quality of the output generated by a system or model for a given task and input. for example, in automatic news summarization, the input is a news article, and the output is a summary. there are two ways to evaluate the output: human evaluation and automatic evaluation, both typically expressed as scores. human evaluation scores are considered the gold standard, and their consistency with automatic evaluation metrics is used to assess and compare the performance of different metrics, which is caculated by the specific correlation measure. this process is referred to as meta-evaluation and can be formalized as follows: assume that there are n systems, {s i } n @xmath0, {d j } m @xmath1. each system s i generates an output h ij for each input d j and the human evaluation score for each output h ij is denoted as z ij . these form a meta-evaluation dataset in most meta-evaluation datasets, n ≪ m , and generally, the range of n is a few to dozens, while the range of m is tens to thousands. on the other hand, an automatic evaluation metric m typically requires the input, output, and other related optional content, such as the reference for evaluation. for each output h ij , the score given by the automatic evaluation metric is denoted as x ij . if there are k metrics to be evaluated in metaevaluation, they are denoted as {m k } k @xmath2, and their output scores are denoted as {x k ij } k @xmath3. the correlation c(x, z) between {x ij } n,m @xmath4,@xmath5.e., x) and {z ij } n,m @xmath6,@xmath7.e., z) is used to evaluate the quality of a certain evaluation metric m. and there are multiple ways to measure this correlation, which can be divided into four categories based on the grouping method, where c denotes a specific correlation coefficient, commonly pearson's r, spearman's ρ, and kendall's τ : it can be seen that the correlation measure includes two parts: the grouping method and the correlation coefficient. considering the common four grouping methods and three correlation coefficients 4 mentioned above, there are 4 × 3 = 12 different correlation measures in total. for two evaluation metrics m 1 and m 2 and a specific correlation measure c, it is generally considered that m 1 outperforms m 2 if c(x 1 , z) > c(x 2 , z). furthermore, the ranking of k metrics {m k } k @xmath8, z) k @xmath9. to verify whether different correlation measures would affect the results of meta-evaluation, 2 wmt22 @xcite used this grouping as one of segment-level grouping methods. and we rename it "item level" to avoid confusion. ## meta-evaluation data for more comprehensive and realistic analyses on correlation measures, we collect six widely used evaluation datasets, as well as the results of 32 common automatic evaluation metrics on them. our subsequent experiments on three research questions are based on this large-scale real evaluation data. ## analyses of correlation measures to analyze and compare the meta-evaluation capabilities of different correlation measures, we consider three important perspectives: discriminative power, ranking consistency, and sensitivity to score granularity. in practice, correlation measures are mainly used to assess the performance of automatic evaluation metrics, with two core applications: comparing two metrics and ranking a set of metrics. for the former, the key lies in the discriminative power of the correlation measure, i.e., whether it can distinguish various metric pairs as effectively as possible. for the latter, the consistency of the correlation measure in ranking the same set of metrics is crucial, i.e., whether the ranking remains stable. in addition, recently emerging llm-based evaluators feature a flexible and discrete scoring pattern similar to human evaluation, so the sensibility of correlation measures to varying levels of score granularity of the same metric should meet certain expectations. in the following subsections, we will design specific tests and conduct experimental analyses based on these three capability aspects to identify the characteristics of different correlation measures. ## conclusions we analyze and compare the characteristics and capabilities of 12 typical correlation measures in nlg meta-evaluation through three proposed perspectives. based on various experiments with largescale real-word data from six nlg datasets and 32 evaluation metrics, we find measures using globallevel grouping and pearson correlation have better overall meta-evaluation capability, while those using system-level and kendall correlation show the opposite. and they have all been used on some common evaluation datasets. we hope that our work can deepen the understanding of correlation measures and draw more attention and emphasis on related research in the future.