Beyond Reference: Evaluating High Quality Translations Better than Human References (2024)

Abstract: In Machine Translation (MT) evaluations, the conventional approach is to compare a translated sentence against its human-created reference sentence. MT metrics provide an absolute score (e.g., from 0 to 1) to a candidate sentence based on the similarity with the reference sentence. Thus, existing MT metrics give the maximum score to the reference sentence. However, this approach overlooks the potential for a candidate sentence to exceed the reference sentence in terms of quality. In particular, recent advancements in Large Language Models (LLMs) have highlighted this issue, as LLM-generated sentences often exceed the quality of human-written sentences. To address the problem, we introduce the Residual score Metric (ResuMe), which evaluates the relative quality between reference and candidate sentences. ResuMe assigns a positive score to candidate sentences that outperform their reference sentences, and a negative score when they fall short. By adding the residual scores from ResuMe to the absolute scores from MT metrics, it can be possible to allocate higher scores to candidate sentences than what reference sentences are received from MT metrics. Experimental results demonstrate that ResuMe enhances the alignments between MT metrics and human judgments both at the segment-level and the system-level.

## introduction evaluation metrics for machine translation (mt) mainly rely on comparisons with human-crafted reference sentences. most frequently used metrics count the number of matching tokens between the reference and candidate sentences @xcite or estimate the similarity within the embedding space @xcite . these metrics impose penalties on candidate sentences that show less alignment with the reference † major in bio artificial intelligence sentence while rewarding those that closely match the reference. however, the current design of mt metrics leads to a limitation: they overlook the possibility that candidate sentences outperform reference sentences in terms of quality. candidate sentences can be better than their reference sentences in reality for two reasons. firstly, reference sentences, subject to the variability of human translators, may contain errors or inconsistencies. thus, reference sentences are not guaranteed to be of best quality and often include critical translation and grammatical errors @xcite . secondly, large language models (llms) exhibit remarkable generative capability. as recent advancements focus on training llms to follow human feedbacks @xcite , llms align with human preferences increasingly. consequently, llms show strong performance across various generative tasks and even surpass average human performance in academic exams originally designed for humans @xcite . notably, some llms have been shown to produce better translations than gold human references according to automatic evaluation metrics @xcite and can be rated higher than a human reference system for specific language pairs @xcite . nevertheless, conventional mt metrics inherently assign the highest scores to candidate sentences that are identical to the reference sentence, thereby suffering from the reference-bias problem @xcite . as a result, current mt metrics do not allocate a higher score to a candidate sentence than the scores assigned to the reference sentence even when the candidate sentence is superior to its reference sentence. this situation requires a metric design that recognizes the potential superiority of machine-generated content. to address the problem mentioned above, we present residual score metric (resume), a novel approach for mt evaluation. while traditional mt metrics allocate an absolute score to the candidate sentence based on the similarity with the reference sentence, our approach resume quantifies the relative quality of the candidate sentence compared to the reference sentence. the relative score is positive when the candidate sentence is better than the reference sentence, and negative when the reference sentence surpasses the candidate sentence. then, we add the relative score to the absolute score computed by an existing mt metric. by introducing resume, it has become possible to assign a higher score to the candidate sentence better than to the reference sentence itself. as a result, the proposed method can be utilized to appropriately evaluate llms that provide translation results superior to those created by humans. in the experimental section, we report an interesting observation that gpt-4 outperforms conventional mt systems with the proposed method resume. labeling a substantial amount of residual scores to train our model is both time-consuming and costly. for this reason, we propose a method to train our resume model using existing labeled data with standard absolute scores, eliminating the additional labeling costs. we conduct extensive experiments to validate the performance of resume both with the segment-level and the system-level evaluations. specifically, resume encourages the alignment between mt metrics and human expert ratings across various language pairs at the segment level evaluation even without residual score labels. in system-level evaluation, our approach affords more accurate assessments for mt systems especially for llms which are underestimated by conventional mt metrics. furthermore, resume can assess candidates outperforming references accurately compared to other mt metrics. the code is available via a github repository foot_0 . our main contributions are summarized as follows: embedding similarity metrics. word embeddings are employed in order to capture semantic similarity between words. specifically, since pre-trained language models (plms) @xcite are learned from huge amounts of text data, its contextual embedding enables a deeper understanding of words between reference and candidate sentences. zhang et al. (2020) extracts token embeddings corresponding to reference and candidate sentences using bert or roberta. then, the final score is derived by calculating the cosine similarity among these embeddings. trainable metrics. recent works @xcite attempt to train metrics using a dataset containing human rating. these methods involve adding a regression layer on the top of plms and training it to directly predict human rating between reference and candidate sentences. in the machine translation, several studies @xcite @xcite utilize the source sentence as semireference jointly during training since it holds the same meaning with reference sentence, playing an important role in translation quality assessment. this approach represents improved alignment with human ratings in machine translation evaluation. reference bias of mt metrics. the reference bias problem is raised by @xcite ; @xcite ; @xcite . previous studies highlight the bias in conventional mt evaluation, where subpar candidates sharing a similar style with references receive high scores. with advancements in llms, a new issue of mt metrics has emerged, which has not been considered by the previous studies. although llm-based models can generate high-quality candidates that surpass the references, these candidates do not receive higher scores than those given to the references. in this work, we identify the new issue and propose a novel method to overcome the problem. zouhar and bojar (2024) emphasizes the importance of reference quality in the automatic mt evaluation. low-quality references can significantly degrade the performance of mt metrics, leading to unreliable assessments of candidates. however, obtaining high-quality references is both expensive and time-consuming. in this work, we introduce a new mt metric that measures the relative quality between the reference and candidates, allowing for an accurate evaluation regardless of the reference quality. ## conventional learnable mt metrics the evaluation of mt commonly involves the use of source, reference, and candidate sentences, denoted as s, r, and c respectively. given a triplet of (s, r, c), an mt metric is formulated as a scoring function f : (s, r, c) → r. a dataset for a learnable mt metric comprises a set of tuples {(s i , r i , c i , y i )} n @xmath0, where y i represents the human rating that assesses the quality of the candidate sentence c i . the primary objective of a learnable metric is to generate scores that are correlated with human ratings. to achieve this, the training process for a learnable metric involves minimizing the mean squared error between the prediction of the metric and human ratings ## proposed method in this section, we first introduce the residual score metric (resume) for machine translation ( §4.1). then, we discuss how to train resume with residual scores ( §4.2). finally, we propose a method for training resume with existing data that includes only absolute scores ( §4.3). ## conclusion in this paper, we propose resume, a novel metric designed to assess the relative quality between reference and candidate sentences in mt. by adding residual scores from resume to the output of existing mt metrics, we address the reference-bias problem in current mt evaluation, which arises with significant advancements in mt systems: the candidate sentences, produced by mt systems, excel the quality of the reference sentences. the empirical results demonstrate that resume not only increases the alignment between mt metrics and human judgements but also enables to discriminate between top mt systems. furthermore, we disclose analyses that resume indeed evaluates higher quality translations over references more accurately against other mt metrics.