Is Reference Necessary in the Evaluation of NLG Systems? When and Where? (2024)

Abstract: The majority of automatic metrics for evaluating NLG systems are reference-based. However, the challenge of collecting human annotation results in a lack of reliable references in numerous application scenarios. Despite recent advancements in reference-free metrics, it has not been well understood when and where they can be used as an alternative to reference-based metrics. In this study, by employing diverse analytical approaches, we comprehensively assess the performance of both metrics across a wide range of NLG tasks, encompassing eight datasets and eight evaluation models. Based on solid experiments, the results show that reference-free metrics exhibit a higher correlation with human judgment and greater sensitivity to deficiencies in language quality. However, their effectiveness varies across tasks and is influenced by the quality of candidate texts. Therefore, it’s important to assess the performance of reference-free metrics before applying them to a new task, especially when inputs are in uncommon form or when the answer space is highly variable. Our study can provide insight into the appropriate application of automatic metrics and the impact of metric choice on evaluation performance.

## introduction automatic evaluation metrics for generated texts play a crucial role in the development of natural language generation(nlg) techniques. most commonly used metrics are reference-based @xcite @xcite . such metrics provide evaluation results by measuring the similarity between text and human-written references @xcite , which are widely applied in various evaluation tasks. however, in the era of large language models (llms), we are witnessing the emergence of llms with varying parameters and domain-specific variations. although their performance can be tested using standard benchmarks, due to the lack of ground truth reference texts, evaluating the generated texts of language models in specific user-oriented scenarios with reference-based metrics is challenging. as a result, the assessment of coherence, consistency, fluency, and other criteria of the language model's output demands substantial time and cognitive resources. in recent years, many reference-free metrics have been proposed as a potential solution to the aforementioned challenges @xcite @xcite . the evaluation procedure of reference-free metrics can be viewed as a generative process, using an underlying generation model to assess other models @xcite , without any reliance on human annotations. the evaluation processes of both reference-based and reference-free metrics are illustrated in figure 1 , and we will give a formal definition in section 2.2. although in some tasks the evaluation results of reference-free metrics have shown a higher correlation with human assessment @xcite , when and where can they be used as a substitute for reference-based metrics is still not well understood. in order to figure out the answer, in this study, we mainly focus on the following questions: specifically, we employ three task-independent criteria: coherence, consistency, and fluency, to comprehensively evaluate the differences between the two types of metrics across various tasks. three reference-free and five reference-based metrics are included, which are tested on eight datasets spanning three tasks: summarization, data-to-text, and dialogue. our experiments reveal that, (i) regarding the first question, reference-free metrics exhibit a stronger correlation with human judgment on all three criteria and almost all tasks, and they are more sensitive to fluency and coherence deficiencies. (ii) as for the second question, the performance of reference-free metrics is constrained by underlying models. the evaluation effectiveness of reference-free metrics could vary across tasks and is influenced by the quality of candidate texts. (iii) addressing the final question, it is crucial to assess the performance of reference-free metrics before applying them to a new task. they are capable of recognizing texts with poor quality, but may not be able to evaluate high-quality candidates. our contribution includes: ## experiments in this section, we first evaluate the performance of metrics on different datasets and criteria. then, we conduct perturbation experiments to examine metrics' sensitivity concerning sentence defects and employ the ks score for further criterion-level analysis. finally, we use the meta-correlation index to explore the stability of metric performance in relation to candidate quality. ## related work the rapid growth of nlg techniques and the emergence of llms have highlighted the importance of automatic evaluation. numerous metrics have been developed and are widely used in a great variety of tasks. apart from metrics used in this study, @xcite presents a thorough survey of common evaluation metrics for nlg systems. assessing the effectiveness of automatic metrics therefore becomes an important task, and various meta-eval approaches are proposed. correlation with human judgment is widely applied, however, as it only provides an evaluation of metrics' overall performance, more fine-grained analyses are developed. for example, @xcite and @xcite present meta-eval approaches beyond correlation with human judgment. @xcite presents a thorough perturbation template for deeper investigating metrics' ability to detect quality defections. openmeva @xcite focuses on story generation, providing a test suite for meta-evaluation from multiple dimensions, pointing out that many metrics have a poor ability to perceive discourse-level incoherence. comprehensive comparison and analysis of automatic metrics are also of importance, which is also the focus of this work. @xcite shows that bleu is not sufficient for the quality evaluation in the translation task. true @xcite focuses on the evaluation of consistency, explicitly defines the meaning, and provides a standard benchmark. @xcite select three reference-free metrics for evaluating machine translation and summarization, indicating that reference-free metrics tend to give texts similar to the output of the underlying model higher scores, instead of human-written sentences, and recommending that reference-free metrics should be used as diagnostic tools instead of evaluation metrics. compared with these researchs, we broaden the types of metrics, criteria and application scenarios, verify the pros and cons of each automatic metric by experiments, and provide possible solutions. ## conclusion in this study, we aim to provide insights into the appropriate usage of automatic evaluation metrics. to achieve this goal, we thoroughly examine the performance of reference-based and reference-free metrics with various meta-analysis methods. our experiments show that, compared with referencebased metrics, the evaluation results provided by reference-free metrics have a closer correlation with human judgment. also, reference-free metrics are more sensitive to the semantic deficiency in texts. however, the performance of referencefree metrics is task-dependent and is not stable as the quality of candidate texts increases. therefore, we recommend assessing metrics before applying them to new tasks and new criteria, especially when metrics are not explicitly designed to be used in the specific scenario.