Not All Metrics Are Guilty: Improving NLG Evaluation by Diversifying References (2024)

Abstract: Most research about natural language generation (NLG) relies on evaluation benchmarks with limited references for a sample, which may result in poor correlations with human judgements. The underlying reason is that one semantic meaning can actually be expressed in different forms, and the evaluation with a single or few references may not accurately reflect the quality of the model’s hypotheses. To address this issue, this paper presents a simple and effective method, named **Div-Ref**, to enhance existing evaluation benchmarks by enriching the number of references. We leverage large language models (LLMs) to diversify the expression of a single reference into multiple high-quality ones to cover the semantic space of the reference sentence as much as possible. We conduct comprehensive experiments to empirically demonstrate that diversifying the expression of reference can significantly enhance the correlation between automatic evaluation and human evaluation. This idea is compatible with recent LLM-based evaluation which can similarly derive advantages from incorporating multiple references. *We strongly encourage future generation benchmarks to include more references, even if they are generated by LLMs, which is once for all.* We release all the code and data at https://github.com/RUCAIBox/Div-Ref to facilitate research.

## introduction evaluation plays a pivotal role in advancing the research on natural language generation (nlg) @xcite . it aims to measure the quality of the generated hypotheses in nlg tasks (e.g., machine translation, text summarization, and image caption) from multiple to reduce the human cost, researchers have proposed various automatic evaluation metrics. yet, due to their rigid analytic forms, they often suffer from an inaccurate approximation of the task goal, even having significant discrepancies with human evaluation @xcite . despite the widespread concerns about evaluation metrics @xcite , another seldom discussed yet important factor is the number of reference texts in the evaluation benchmarks. there always exist diverse hypotheses that would satisfy the goal of an nlg task, however, the number of ground-truth references provided by human annotators is often limited in scale. for example, there is only one english ground-truth reference written for a chinese input sentence in the wmt22 news translation task @xcite . this potentially leads to unreliable evaluation results when using limited ground-truth references, as illustrated in considering the above-mentioned issue, this paper attempts to improve the nlg evaluation benchmarks and make existing automatic metrics better reflect the actual quality of the hypotheses. we focus on increasing the number of reference texts to narrow the gap between automatic and human evaluation. the key idea is to leverage the excellent ability of existing llms to provide more highquality references for a single sample. by enriching the diversity of the references while maintaining semantic consistency, we expand the coverage of the semantic expressions for evaluating the generated texts from a single or few standard references to a more diverse set of semantically equivalent references. in this way, our evaluation method can better approximate human evaluation criteria, as the improved scores shown in to demonstrate the effectiveness of diversifying references, we conduct extensive experiments on the benchmarks from multiple nlg tasks. the experimental results demonstrate that incorporating multiple references can significantly improve the consistency between traditional evaluation metrics and human evaluation results. surprisingly, it is even applicable in multilingual and multimodal text generation scenarios. importantly, our approach is orthogonal with automatic metrics, enabling even the recent llm-based evaluations @xcite to benefit from our diversified references and achieve sota correlation with human judges. therefore, incorporating more references for the nlg benchmark proves advantageous, requiring a one-time effort, and future researchers can reap its benefits. ## methodology this section first provides a formal definition by introducing several crucial aspects of nlg evaluation. we then describe our approach that leverages llms to enrich the semantic coverage of references, bridging the gap between automatic evaluation and human evaluation. ## experiments in this section, we deliberately select three different types of natural language generation tasks to verify the effectiveness of multiple references. ## conclusion in this paper, we have investigated the effect of enriching the number of references in nlg benchmarks and verified its effectiveness. our diversifying method, div-ref, can effectively cover the semantic space of the golden reference, which can largely extend the limited references in existing benchmarks. with extensive experiments, our approach yields substantial improvements in the consistencies between evaluation metrics and human evaluation. in future work, we will explore the current evaluation method on more nlg tasks, and also consider extending it to evaluate generation tasks in other modalities. it is also valuable to investigate whether paraphrasing can improve llms' training and utilization.