GEMBA - MQM : Detecting Translation Quality Error Spans with GPT -4 (2023)

Abstract: This paper introduces GEMBA-MQM, a GPT-based evaluation metric designed to detect translation quality errors, specifically for the quality estimation setting without the need for human reference translations. Based on the power of large language models (LLM), GEMBA-MQM employs a fixed three-shot prompting technique, querying the GPT-4 model to mark error quality spans. Compared to previous works, our method has language-agnostic prompts, thus avoiding the need for manual prompt preparation for new languages. While preliminary results indicate that GEMBA-MQM achieves state-of-the-art accuracy for system ranking, we advise caution when using it in academic works to demonstrate improvements over other methods due to its dependence on the proprietary, black-box GPT model.

## introduction gemba-mqm builds on the recent finding that large language models (llms) can be prompted to assess the quality of machine translation @xcite . we release the scoring script. 1 the earlier work kocmi and federmann (2023a) (gemba-da) adopted a straightforward methodology of assessing single score values for each segment without specifying the scale in detail. employing a zero-shot approach, their technique showed an unparalleled accuracy in assessment, surpassing all other non-llm metrics on the wmt22 metrics test set @xcite . next, @xcite (eaprompt) investigated prompting llms to assess individual error classes from a multidimensional quality metrics (mqm) framework @xcite , where each error can be classified into various error classes (such another notable effort by fernandes et al. (2023) paralleled the eaprompt approach, also marking mqm error spans. in contrast, their approach used a palm-2 model, pooling mqm annotations to sample a few shot examples for the prompt. their (system) you are an annotator for the quality of machine translation. your task is to identify errors and assess the quality of the translation. ## description our technique adopts few-shot learning with the gpt-4 model (openai, 2023), prompting the model to mark quality error spans using the mqm framework. the underlying prompt template is modeled on guidelines for human annotators and shown in figure 1 . in contrast to other methods, we use three predetermined examples (see appendix a), allowing the method to be used with any language pair, avoiding the need to create language pair specific mqm few-shot examples. this was the original limitation that prevented @xcite from evaluating automqm beyond two language pairs. our decision was not driven by a desire to enhance performance -since domain and languagespecific prompts typically boost it @xcite -but rather to ensure our method can be evaluated across any language pairs. ## experiments to measure the performance of the gemba-mqm metric, we follow the methodology and use test data provided by the wmt22 metrics shared task @xcite which hosts an annual evaluation of automatic metrics, benchmarking them against human gold labels. we compare our method against the bestperforming reference-based metrics of wmt22: metrixx_xxl (non-public metric), comet-22 @xcite , unite @xcite , bleurt-20 @xcite , and comet-20 @xcite . in addition, we also compare against "classic" string-based metrics bleu @xcite and chrf @xcite . lastly, we compare against reference-less metrics of wmt22: cometkiwi @xcite , unitesrc @xcite , comet-qe @xcite , ms-comet-qe-22 @xcite . we contrast our work with other llm-based evaluation methods such as gemba-da @xcite and eaprompt @xcite , conducting experiments using two gpt models: gpt-3.5-turbo and the more powerful gpt-4 (openai, 2023). ## results in this section, we discuss the results observed on three different test sets: 1) mqm test data from wmt, 2) internal test data from microsoft, and 3) a subset of the internal test data to measure the impact of the mqm locale convention. ## caution with "black box" llms although gemba-mqm is the state-of-the-art technique for system ranking, we would like to discuss in this section the inherent limitations of using "black box" llms (such as gpt-4) when conducting academic research. firstly, we would like to point out that gpt-4 is a proprietary model, which leads to several problems. one of them is that we do not know which training data it was trained on, therefore any published test data should be considered as part of their training data (and is, therefore, possibly tainted). secondly, we cannot guarantee that the model will be available in the future, or that it won't be updated in the future, meaning any results from such a model are relevant only for the specific sampling time. as @xcite showed, the model's performance fluctuated and decreased over the span of 2023. as this impacts all proprietary llms, we advocate for increased research using publicly available models, like llama 2 @xcite . this approach ensures future findings can be compared both to "black box" llms while also allowing comparison to "open" models. 3 ## conclusion in this paper, we have introduced and evaluated the gemba-mqm metric, a gpt-based metric for translation quality error marking. this technique takes advantage of the gpt-4 model with a fixed three-shot prompting strategy. preliminary results show that gemba-mqm achieves a new state of the art when used as a metric for system ranking, outperforming established metrics such as comet and bleurt-20. we would like to acknowledge the inherent limitations tied to using a proprietary model like gpt. our recommendation to the academic community is to be cautious with employing gemba-mqm on top of gpt models. for future research, we want to explore how our approach performs with other, more open llms such as llama 2 @xcite . confirming superior behaviour on publicly distributed models (at least their binaries) could open the path for broader usage of the technique in the academic environment.