NLG Evaluation Metrics Beyond Correlation Analysis: An Empirical Metric Preference Checklist (2023)

Abstract: In this study, we analyze automatic evaluation metrics for Natural Language Generation (NLG), specifically task-agnostic metrics and human-aligned metrics. Task-agnostic metrics, such as Perplexity, BLEU, BERTScore, are cost-effective and highly adaptable to diverse NLG tasks, yet they have a weak correlation with human. Human-aligned metrics (CTC, CtrlEval, UniEval) improves correlation level by incorporating desirable human-like qualities as training objective. However, their effectiveness at discerning system-level performance and quality of system outputs remain unclear. We present metric preference checklist as a framework to assess the effectiveness of automatic metrics in three NLG tasks: Text Summarization, Dialogue Response Generation, and Controlled Generation. Our proposed framework provides access: (i) for verifying whether automatic metrics are faithful to human preference, regardless of their correlation level to human; and (ii) for inspecting the strengths and limitations of NLG systems via pairwise evaluation. We show that automatic metrics provide a better guidance than human on discriminating system-level performance in Text Summarization and Controlled Generation tasks. We also show that multi-aspect human-aligned metric (UniEval) is not necessarily dominant over single-aspect human-aligned metrics (CTC, CtrlEval) and task-agnostic metrics (BLEU, BERTScore), particularly in Controlled Generation tasks.

## introduction natural language generation (nlg) refers to an automatic process to generate texts in one or more language categories that satisfy multiple desirable human-like qualities. for example, in text summarization @xcite @xcite , nlg systems are expected to produce coherent, consistent, fluent, and relevant summarization outputs. in dialogue response generation @xcite , the system outputs are mainly assessed based on aspects that are important in a typical human conversation, such as naturalness and engagingness. in controlled generation @xcite , the generation outputs are evaluated based on its relevance to the predefined topic category or sentiment category as control attributes. a standard evaluation protocol in nlg for assessing the above human-like qualities involves conducting a human evaluation study or running an automatic evaluation, or both ways. a human evaluation study improves the reliability of evaluation process, particularly when the assessment is done by experts. it is also often infeasible to translate human evaluation aspects into an automatic statistical metric formulation due to its multi-dimensional abstractive properties @xcite . however, human evaluation is known to be more costly and does not scale well @xcite . utilizing automatic metrics, on the other hand, is cost-effective and more feasible for large-scale evaluation data. recent works on automatic nlg evaluation metrics, such as ctrleval @xcite , ctc @xcite , and unieval @xcite , have made progress in improving the correlation between automatic metrics and human up to 43% by developing human-aligned automatic metrics. despite the advancements, there is a need for a standardized framework to assess the utility of these metrics in the context of discerning systemlevel performance. the reason is that an overall correlation score to human does not necessarily represents the metric effectiveness as an evaluation tool, as demonstrated by previous analysis studies on nlg automatic metrics @xcite @xcite . however, none of these works connect the correla-assessment type ## related work existing automatic metrics in nlg are mainly dominated by task-agnostic metrics -metrics that assess the quality of generation outputs without considering human evaluation aspects as context or objective of the evaluation task @xcite . task-agnostic metrics are highly adaptable across nlg tasks because the adaptation does not require task-specific design. for example, bleu @xcite and rouge @xcite , which represent string-based metrics, are largely adopted in neural machine translation (nmt) and text summarization. perplexity @xcite ) -a reference-less metric, is a standard evaluation metric in a language modeling-based nlg tasks, including controlled generation @xcite . bertscore @xcite has been largely adopted in diverse nlg tasks, including @xcite , text summarization (deutsch and roth, 2021), and dialogue system @xcite . nevertheless, currently available task-agnostic metrics are weakly correlated to human judgment @xcite @xcite . a low correlation score introduces a criticism on the capability of automatic metrics at identifying the different quality of system outputs and their potential usage to substitute a costly human evaluation study. recent works @xcite @xcite have demonstrated that incorporating desirable human-like qualities as a training objective or contextual knowledge is the best-fit solution for improving the correlation level between automatic metrics and human. however, verifying whether a higher correlation represents a higher human preference for ranking the quality of system outputs and ranking system performance, and vice versa, remains an underexplored query. compared to the recent analysis studies that focus on validating the robustness @xcite @xcite , explainability @xcite , reproducibility @xcite , and fairness @xcite of the nlg evaluation metrics, our study focuses more on a systematic assessment by connecting the link between correlation score to the practical use of the metrics in nlg evaluation. that is, (i) for discriminating the system outputs based on desirable human-like qualities; and (ii) for ranking system performance. ## metric preference checklist a standard evaluation protocol in nlg involves validating automatic metrics based on their correlation to human. intuitively, a high correlation suggests a high agreement on discerning the quality of system outputs because low-quality outputs are penalized with lower scores, while high-quality outputs are rewarded with higher scores. however, currently available metrics are known to have a poor correlation to human. so, it is unclear to what extend current automatic metrics are capable of (i) identifying human-like quality of system outputs and (ii) discriminating performance between independent nlg systems. to further investigate the above questions, we pose several relevant research questions as a metric preference checklist, as presented in ## conclusion we introduce the metric preference checklist as a framework for analyzing the effectiveness of currently available nlg automatic metrics. we show the importance of verifying the preference similarity between automatic metrics and human, regardless of their correlation scores. we also find that automatic metrics are more discriminating than human for discerning system-level performance, except for human evaluation aspect with a binary voting protocol. lastly, we show the implication of current work on guiding the selection of the best-performed system based on pairwise system ranking.