Large Language Models Are State-of-the-Art Evaluators of Translation Quality (2023)

Abstract: We describe GEMBA, a GPT-based metric for assessment of translation quality, which works both with a reference translation and without. In our evaluation, we focus on zero-shot prompting, comparing four prompt variants in two modes, based on the availability of the reference. We investigate seven versions of GPT models, including ChatGPT. We show that our method for translation quality assessment only works with GPT 3.5 and larger models. Comparing to results from WMT22’s Metrics shared task, our method achieves state-of-the-art accuracy in both modes when compared to MQM-based human labels. Our results are valid on the system level for all three WMT22 Metrics shared task language pairs, namely English into German, English into Russian, and Chinese into English. This provides a first glimpse into the usefulness of pre-trained, generative large language models for quality assessment of translations. We publicly release all our code and prompt templates used for the experiments described in this work, as well as all corresponding scoring results, to allow for external validation and reproducibility.

## introduction one of the interesting properties of large language models (llms) such as gpt @xcite is their (implicit) support for multilingual q&a. prompting the model in the right way allows us to translate text between languages @xcite . this is surprising as gpt has not been fine-tuned for the translation task. @xcite show that gpt-enabled translation achieves high quality when applied for the translation of high-resource languages, but still lacks in terms of translation quality for underrepresented languages. building on this finding-if the model can translate, it may be able to differen-tiate good from bad translations-we apply gpt for the task of translation quality assessment. in the remainder of this paper, inspired by recent progress on generative, pre-trained large language models (llms), we explore how these models can be applied for automated assessment of translation quality. the primary query for this study centers around the question: can llms be used for effective quality assessment of translations? we propose gemba, which stands for gpt estimation metric based assessment. the metric evaluates each segment translation in isolation and then averages across all obtained scores for the final, system-level score. we define and evaluate several prompt variants for zero-shot assessment of translation quality in two modes, either with a human reference translation, as a quality metric, or without one, as a quality estimation task. we design the main prompts based on the da+sqm template used for human assessment of translation quality as implemented in the appraise framework @xcite for wmt22 @xcite , building on previous work conducted by @xcite . to the best of our knowledge, our research represents the pioneering effort in exploring the utilization of large language models (llms) for the purpose of quality assessment. subsequent to the publishing of our findings, @xcite independently published a related report, corroborating the high performance of llms. the main contributions of this paper are: to assess translation quality via prompting an llm, the following parameters are needed: we generate a gpt request for every segment, querying as individual zero-shot problems, and then aggregate results. for this initial proof of concept, we leave improvements such as few-shot queries or document-level context to future work. ## experiments to measure the performance of the proposed gemba metric, we follow the methodology and use test data provided by the wmt22 metrics shared task @xcite which hosts an annual evaluation of automatic metrics, benchmarking them against human gold labels. effectively, we compare gemba against the best-performing automatic metrics: comet @xcite , bleurt @xcite , or the non-public winner metricx xxl. ## results we investigate gemba's performance for two modes: with a reference translation and without reference translation (in a quality estimation setting). ## conclusion we have presented our work on gemba, a gptbased estimation metric-based assessment method. comparing our metrics to other automated metrics from the wmt22 metrics shared task we report state-of-the-art performance on the mqm 2022 test set across three language pairs: english to german, english to russian, and chinese to english. we intend to continue research on the application of gpt models for quality assessment. further research will focus on the switch to few-shot (as opposed to our current zero-shot methodology) as well as model fine-tuning. both of which promise to increase gemba accuracy. furthermore, modifying prompts to support mqm error-based evaluation or post-editing efforts may lead to further improvements. gpt-enhanced evaluation metrics may allow us to make progress with respect to document-level evaluation (due to their ability to use much larger context windows). this could be beneficial as there is little research into document-level metrics @xcite .