Towards Multiple References Era – Addressing Data Leakage and Limited Reference Diversity in Machine Translation Evaluation (2024)

Abstract: Recent research has shown a weak correlation between n-gram-based metrics and human evaluations in machine translation task, particularly when evaluating large language models (LLMs). Additionally, the data leakage risk in LLMs may cause an overestimation problem when evaluating LLMs on downstream tasks. In this work, we identify the limited diversity of references as the primary cause for the inferior performance of n-gram-based metrics and the overestimation problem. To address this issue, we propose to utilize multiple references generated by LLMs, coupled with an effective selection strategy focused on accuracy and diversity, to improve the alignment between automatic metrics and human evaluations. We validate our approach on the WMT22 Metrics benchmark with 4 languages and observe a maximum accuracy gain of 9.5% in F200spBLEU, which makes it on par with computationally expensive neural-based metrics. We also show that using multi-reference with n-gram-based metrics significantly alleviates the overestimation problem when evaluating LLMs with data leakage. Further analysis explores the factors that affect the quality of generated references, offering insights into data synthesis by LLMs.

## introduction due to the inherent diversity and complexity of natural language, human evaluation serves as the gold standard for assessing the quality of machine translation. however, conducting human evaluation is time-consuming and prohibitively expensive in real-world scenarios. consequently, developing reliable automatic evaluation metrics is crucial for advancing machine translation research and optimizing machine translation systems @xcite . the ideal automatic evaluation metrics need to assess the accuracy, fluency, fidelity, and diversity of the generated candidates by the model. it also requires high consistency with human assessment to demonstrate its reliability. automatic evaluation metrics for machine translation can be generally classified into two categories: n-gram-based metrics and neural-based metrics. n-gram-based metrics, e.g., bleu @xcite primarily calculate the lexical overlap between the model's outputs and the ground truth. on the other hand, neural-based metrics e.g., bleurt @xcite are trained on a huge amount of text data and score the candidates with a black-box model. due to the huge amount of training data and parameters, neuralbased metrics generally achieve better robustness and generalization. recent studies @xcite have demonstrated that neural-based metrics exhibit better agreement with human evaluation when compared with n-grambased metrics. the emergence of llms has further deepened these concerns @xcite due to the diversity of output results from llms. recent studies @xcite have highlighted the challenges in using n-gram-based metrics, such as bleu, to evaluate llm-generated hypotheses. some latest reports @xcite exclusively rely on neural-based metrics to evaluate the generation quality. another challenge in evaluating llms is the data leakage risk due to their reliance on huge corpora obtained from the web. recent research @xcite has emphasized that data leakage in llms is a prevalent and pressing issue, which underlines the importance of conducting more in-depth research to develop effective methods for addressing this challenge. in this paper, we examine the performance bottleneck of n-gram-based metrics and identify the limited diversity of references as the primary cause for the inferior performance of n-gram-based metrics and the overestimation problem. to address this issue, we propose llm-ref, a framework that utilizes llms to generate multiple references, coupled with an effective selection strategy focused on diversity, to improve the alignment between automatic metrics and human evaluations. experimental results show that our framework improves the consistency between human evaluation and both kinds of automatic evaluation metrics. further analysis reveals that multiple references with n-grambased metrics can effectively mitigate the potential data leakage risk of llms while neural-based metrics still face difficulty in overcoming this issue. to our knowledge, this is the first attempt to address the data leakage risk issue in evaluating llms. ## preliminary experiments to verify bleu's evaluation capability on llmgenerated results and the data leakage issue of llms, we conduct preliminary experiments using bloomz-7b on the flores-200 test set @xcite . previous work @xcite has identified data leakage issues with bloomz-7b on flores-200 across multiple translation directions. we conduct supervised fine-tuning (sft) on the bloomz-7b @xcite with bilingual corpus in chinese and japanese from the ccmatrix dataset @xcite . to provide a comprehensive comparison, we also evaluate the performance of closed-source large models such as gpt-3.5 and claude. the baseline machine translation (mt) model is an in-house model with the conventional encoder-decoder architecture. the evaluation metrics utilized in our experiments include f200spbleu @xcite , bleurt @xcite , and human evaluation (human eval). f200spbleu is the bleu @xcite score with a multilingual tokenizer trained in 200+ languages. human eval involves human experts rating the given translations on a scale ranging from 0 to 5, based on how accurately it reflects the meaning of the original sentence. we acquire 3 scores per sentence from human experts and take the average as the final sentence score. the summarized results are presented in figure 1 . we observe two phenomena from the above experiments: the inadequacy of n-gram-based metrics for evaluating llms. we conjecture that the output distribution of the llms differs significantly from that of the mt model, and the n-gram-based nature of bleu is constrained by the provided reference, resulting in degraded performance when there is a substantial divergence between the output and the reference. to gain further evidence of the underlying factors contributing to the bias in bleu's evaluation, we conduct experiments on the chinese to english dataset from wmt22 @xcite to examine the output distribution of different models. we primarily use distinctn @xcite and the count of unique tokens to analyze the output distribution. we select system outputs and ground truth translations from the task for our analysis. the results are presented in we discover that both distinctn scores and the count of unique tokens of gpt-3.5-turbo are higher than all mt models, commercial systems, and human-translated references. these findings indicate that the outputs of llms exhibit a high degree of diversity, which renders it challenging for n-gram-based metrics to effectively estimate the quality of llm generations. llms' data leakage risk. the open-source llms have gradually become foundational models for most nlp tasks @xcite @xcite . however, due to their reliance on huge corpora obtained from the web, data leakage is an inevitable problem, which may potentially lead to an overestimation of the performance on downstream tasks. recent study @xcite about llms have confirmed that data leakage in some widely used benchmarks is a common issue. our hypothesis posits that utilizing multiple references can alleviate overestimation due to data leakage to a certain degree. the tendency of overestimation comes from the model generating almost identical characters as the ground truth, thereby resulting in the performance overestimated by the golden reference. with multiple references provided, the model output close to the golden reference tends to obtain suboptimal performance on other references, which can alleviate the overestimation issue. we will discuss it in detail in sec 5.4. ## methodology we employ a series of steps to obtain high-quality reference candidates for a more accurate evaluation of generation performance, as outlined below. ## experiments in this section, we describe the benchmarks and the evaluation we used for our experiments. ## analysis in this section, we will first analyze the effects of reference numbers and prompt variants, and then analyze whether multi-reference solves the data leakage issue. we conduct these analyses on the wmt22 metrics shared task. ## conclusion this work presents a study on the limited diversity of references for n-gram-based metrics and the overestimation issue when assessing llms that suffer from data leakage. we propose llm-ref, a framework that enhances alignments of machine translation evaluation metrics with human evaluation. by generating multiple reference candidates with llms and implementing the quality-aware selection mechanism, we effectively address these two challenges. experiments on the wmt22 metrics shared task including 4 languages demonstrate that llm-ref significantly improves the consistency of all automatic evaluation metrics with human assessments. additionally, our analyses highlight the efficacy of n-gram-based metrics with multiple references in mitigating overestimation issue due to data leakage in llms while neural-based metrics struggle to overcome. we explore the factors that affect the quality of generated references by llms and provide some insights for future data synthesis from llms.