Lost in the Source Language: How Large Language Models Evaluate the Quality of Machine Translation (2024)

Abstract: This study investigates how Large Language Models (LLMs) leverage source and reference data in machine translation evaluation task, aiming to better understand the mechanisms behind their remarkable performance in this task.We design the controlled experiments across various input modes and model types, and employ both coarse-grained and fine-grained prompts to discern the utility of source versus reference information.We find that reference information significantly enhances the evaluation accuracy, while surprisingly, source information sometimes is counterproductive, indicating LLMs’ inability to fully leverage the cross-lingual capability when evaluating translations.Further analysis of the fine-grained evaluation and fine-tuning experiments show similar results.These findings also suggest a potential research direction for LLMs that fully exploits the cross-lingual capability of LLMs to achieve better performance in machine translation evaluation tasks.

## introduction the last decade has witnessed significant development in neural machine translation (nmt) @xcite @xcite . as the quality of machine translations has been improved, it becomes more challenging and critical for automatic translation evaluation. the recent study @xcite calls for stopping using bleu @xcite , a traditional metric, as it is not reliable for highquality translations and has a lower correlation with human judgements. neural metrics based on pre-trained language models @xcite @xcite , such as comet @xcite , show a higher correlation with human judgments. however, these neural metrics only provide a score lacking interpretability and still exhibit robustness issues hard to detect @xcite . recently, large language models (llms) @xcite @xcite have also been used as translation evaluators. gemba @xcite presents that gpt-4 can achieve state-of-the-art performance in system-level assessment. while llms show remarkable performance in translation evaluation tasks, the reasons underlying their success have not been thoroughly investigated. in this paper, we take the further step to investigate how llms leverage source and reference information in evaluating translation in both coarsegrained and fine-grained settings, bringing better understanding of the working mechanism of llms. four input modes are defined, each of which exposes different information to llms. these include translation-only (t), source-translation (s-t), reference-translation (r-t) and source-reference-translation (s-r-t) modes. we first instruct both open and closed llms to predict coarsegrained quality scores using gemba prompt, but given different information, namely sources and references. while references significantly improve the system-level accuracy and segment-level correlations, we surprisingly find that the source information is sometimes counterproductive. for example, as shown in in addition to the superficial score prediction, we further explore the fine-grained error detection to better understand the cross-lingual ability. our fine-grained experiments confirm the above observations, where we follow automqm @xcite ) method to predict error spans and categories for translation sentences and study the performance of llms with different input modes. we also conduct a comprehensive meta-evaluation for the error span and error category, alongside executing a critical error detection task. the findings suggest that llms struggle to fully utilize the source information for translation evaluation. lastly, we examine the effect of fine-tuning, which makes deeper modifications to the model, with multidimensional quality metrics (mqm) data @xcite . although fine-tuning can greatly improve the model's performance of translation evaluation, the negative impact of source sentence still exists. these experimental results reveal the limitation of current llms on machine translation evaluation tasks and suggest a potential research direction that fully exploits the cross-lingual capability of llms to achieve better performance. overall, our main contributions are as follows: neural metrics leverage the meaningful representations of pre-trained language models to evaluate translations. bertscore @xcite and @xcite calculate similarity using these representations, while generationbased methods like prism @xcite and bartscore @xcite ) assess text quality through generation tasks, conditioned on sources or references. learned metrics such as comet @xcite , bleurt @xcite , unite @xcite and xcomet @xcite apply neural networks to predict quality scores in a supervised manner. particularly, unite suggests that the interaction between source and hypothesis may have an adverse effect. although unite performs best with the s-r-t mode, our work shows this is not always the case for llms. llms that can follow the instructions of evaluation tasks are also used to evaluate translations. gemba @xcite shows that gpt-4, when asked to directly generate a quality score, is the state-of-the-art translation evaluator at that time in the system-level assessment. eaprompt @xcite , automqm @xcite , and gemba-mqm @xcite endeavor to instruct llms in detecting translation errors meticulously and labelling the error category and severity. despite their remarkable performance, the reasons underlying their success have not been thoroughly investigated. ## coarse-grained score prediction we first investigate how llms leverage the source and reference information in conducting coarsegrained evaluations of translation quality via score prediction. we adopt the gemba-sqm prompt template @xcite , as shown in figure 1 . the inclusion of source and reference information in the prompt varies based on the selected input mode. for instance, in the r-t mode, the source text in red is omitted from the prompt. we assess the efficacy of llms across four distinct input modes, examining their impact on the model's performance. ## fine-grained error detection while coarse-grained scoring methods have demonstrated their potential, recent innovations like au-tomqm and gemba-mqm are eliciting the capabilities of llms through the use of specialized prompts, leading to more refined and interpretable results. we further dive into studying how well llms leverage the different information with finegrained methods. we adopt the automqm prompt template (fernandes et al., 2023), as illustrated in figure 2 . the content in the yellow part varies depending on the input mode. our assessment encompasses three perspectives: mqm scores, error spans, and error categories, hence offering a comprehensive diagnosis of the model's predictions. ## fine-tuning llms with mqm data we further investigate the effect of fine-tuning an open llm with task-specific data to determine if it can eliminate the above limitation. ## conclusion we empirically analyze how well llms incorporate the source and reference information for translation evaluation, comparing the effectiveness of open and closed llms through prompting and finetuning. our results reveal their limitations in fully exploiting the cross-lingual capability for the task, with the inclusion of source information even occasionally proving detrimental to performance. furthermore, our work contributes a detailed metaevaluation of spans and categories with the finegrained evaluation method, along with the critical error detection task. these findings not only furnish insights into the current capabilities and limitations of llms in translation evaluation, but also establish a foundational basis for subsequent scholarly endeavors. in the future, we would like to extend these analyses to other nlg evaluation tasks.