The Devil Is in the Errors: Leveraging Large Language Models for Fine-grained Machine Translation Evaluation (2023)

Abstract: Automatic evaluation of machine translation (MT) is a critical tool driving the rapid iterative development of MT systems. While considerable progress has been made on estimating a single scalar quality score, current metrics lack the informativeness of more detailed schemes that annotate individual errors, such as Multidimensional Quality Metrics (MQM). In this paper, we help fill this gap by proposing AutoMQM, a prompting technique which leverages the reasoning and in-context learning capabilities of large language models (LLMs) and asks them to identify and categorize errors in translations. We start by evaluating recent LLMs, such as PaLM and PaLM-2, through simple score prediction prompting, and we study the impact of labeled data through in-context learning and finetuning. We then evaluate AutoMQM with PaLM-2 models, and we find that it improves performance compared to just prompting for scores (with particularly large gains for larger models) while providing interpretability through error spans that align with human annotations.

## introduction evaluating natural language generation systems has always been challenging, and as the output quality of these systems has improved, evaluation has become even more challenging and critical. for example, in machine translation (mt), a field where evaluation has garnered considerable attention, previous standard automatic surface-level metrics such as bleu @xcite are becoming less reliable as the quality of generation systems improves, with little remaining correlation with human judgments @xcite . to keep pace with the constantly improving quality of mt output, the next generation of automatic metrics is rapidly evolving. learned automatic metrics that leverage human-judgments to finetune language models @xcite currently represent the state-of-the-art in automatic evaluation benchmarks like the wmt metrics task @xcite , and show high correlation with human judgments. however, these metrics typically output a single, uninterpretable quality score, making it difficult to understand the type and extent of errors identified by them. the lack of insights makes it difficult for model developers to leverage these metrics to improve their systems. unlike automatic metrics that only provide a single scalar value as quality score, state-of-the-art human evaluation methodologies like multidimensional quality metrics (mqm; @xcite ask professional annotators to identify and label error spans with a category and severity. this much richer feedback can be used to gain a better understanding of the current limitations of the model under evaluation and improve it. in this paper, we ask whether large language models (llms) in combination with a few human annotations can be used to design an automatic metric that generates rich feedback similar to that generated by human experts in mqm. this work is motivated by recent papers that demonstrated that llms can be used as automatic metrics @xcite to generate a single quality score. in particular, @xcite showed that llms can be prompted to assess the quality of machine-generated translations, even achieving state-of-the-art performance on assessing systemlevel quality. however, previous work only provides a limited view of the capabilities of llms for machine translation evaluation: the focus has predominantly been on score prediction (i.e. predicting a numerical value for quality), without considering the use of any annotated data (either through in-context learning or finetuning), and only in highresource language pairs. we provide a large-scale study of the capabilities of llms (from the palm and palm-2 families; @xcite for machine translation evaluation (both with and without a reference translation), provide a novel comparison between prompting and finetuning, and investigate the performance in the low-resource scenario. inspired by findings that the performance of llms can be improved by prompting them for rationales of their predictions @xcite , we also propose automqm, a prompting technique for mt evaluation that asks llms to identify error spans in a translation and to classify these errors according to the mqm framework, with a quality score derived automatically from the identified errors. a key advantage of automqm is its interpretability, as users can inspect the errors responsible for a score (figure 1 ). our contributions can be summarized as follows: our findings might have significant implications for not only mt evaluation, but evaluation of machine-generated text in general, and further highlight the potential of using llms to provide ai feedback @xcite . the outputs of our models prompted with automqm are available at github.com/googleresearch/google-research ## background: mt evaluation machine translation evaluation is one of the most well-studied evaluation problems in nlp @xcite . in this task, given 1. a source sentence in a (source) language 2. a candidate translation in a (target) language an evaluation metric assesses the quality of the candidate translation by how well it conveys the meaning of the source sentence while considering other factors like fluency. like many other natural language generation evaluation problems, this task is difficult because the set of correct translations for a given source sentence is often very large and not entirely known in advance. to simplify the problem of machine translation evaluation, often (3) a reference translation (typically created by a professional human translator) is included as additional information when assessing the candidate translation. this sub-problem is known as reference-based evaluation (as opposed referenceless evaluation or quality estimation). up until recently, human evaluation of machine translation was carried out predominantly with the aim of assigning a single quality score to a candidate translation. consequently, learned metrics, which leverage collected human judgment data, are trained for and evaluated on the same task of score prediction (i.e., assigning a single quality score to a candidate translation), and can achieve high correlation with human-provided scores @xcite . however, framing machine translation evaluation as a score prediction task is problematic: any scoring or ranking of translations is implicitly based on an identification of errors in the candidate translations, and asking raters to solely provide a single score can lead to rushed and noisy judgments @xcite . this insight has led to the adoption of the multidimensional quality metrics (mqm) framework @xcite as the gold standard for evaluating machine translation. the mqm framework asks human evaluators to identify error spans in candidate translations and classify those errors according to various dimensions, e.g., fluency, accuracy, ... (see appendix a for a more detailed description of mqm). importantly, the mqm framework does not ask annotators to provide a quality score for each translation, and instead derives one automatically from the identified error spans and their classifications. however, despite its richness, most automatic metrics that leverage mqm data only use the final quality score produced by the framework and discard the error span information and classification. ## related work the success of learned machine translation metrics @xcite @xcite , which finetune neural network models pretrained on large amounts of (unsupervised) data, highlighted the importance of leveraging transfer learning to achieve metrics with better correlation with human judgments. more recently, generative llms @xcite have consistently demonstrated impressive results in natural language understanding and zeroand few-shot transfer and, naturally, interest in employing these models for (translation) evaluation has increased. @xcite first explored the use of gpt models for evaluating machine translation tasks, showing their potential as zero-shot evaluators, and others have since extended gpt-based evaluation to other generation problems @xcite . @xcite first highlighted that mqm annotations could be leveraged to allow pretrained models to predict major and minor errors and, sim-ilarly to automqm, used the identified errors to automatically score translations. however, their approach relied on weaker encoder-only or encoderdecoder language models, required supervised data to work, and overall underperformed other top metrics. we compare against their matase metric in our experiments. @xcite showed that doing error analysis, a prompting technique similar to automqm, could lead to better chatgpt-based evaluators. however, they still relied on the llm to provide a score once it identified errors (rather than do it automatically using something like the mqm framework). furthermore, they provided a very limited meta-evaluation using only 40 examples per language pair. concurrently with our work, @xcite proposed instructscore, a llama-based evaluator that asks models to identify and categorize errors in translation (as well as providing a natural language explanation for each error). however, the authors only explore a 7b parameter model and don't leverage zero-and fewshot capabilities of models as in this work. instead, they rely on a more complex approach of distilling the knowledge of a more capable gpt-4 llm. additionally, wmt word-level quality estimation shared tasks @xcite leverage mqm data by converting span-level annotations of errors (normally of major severity) to word-level tags and task 2 in the wmt19 quality estimation shared task evaluation explicitly evaluated submissions of span-level annotations (although most submissions still consisted of models that predicted word-level tags which were converted to spans). we also compare against state-of-the-art word-level quality estimation models. ## using llms to predict quality scores recent works have shown that large language models are versatile, general-purpose models that can be used to tackle many problems in nlp, including evaluation @xcite @xcite . we begin by exploring how llms can be used for machine translation evaluation through score prediction. ## using llms to predict error spans while producing quality scores that correlate with human judgments is an important part of translation quality assessment, metrics that solely do score prediction suffer from problems of interpretability: if a metric assigns a low score, the downstream users are left in the dark about which parts of the translation were responsible for the score and thus need to be corrected. this is especially problematic in cases where the metric assigns a wrong score to a translation, as it is much harder to diagnose why the evaluation model made a mistake, and identify and prevent similar mistakes in the future. in fact, reducing translation quality to a single score has proven problematic even for human annotators: asking raters to solely provide a single score can lead to rushed and noisy judgments @xcite and the current gold standard for translation quality evaluation involving human annotators is instead based on methodologies like the mqm framework (see §2) , which provide richer feedback by identifying error spans, categorizing them, and evaluating their severity. interestingly, another emergent phenomenon in llms is the success of chain-of-thought prompting @xcite : when defining a prompt for a particular task, if we instruct the model to produce a series of intermediate reasoning steps ("let's think step-by-step"), it tends to generate a free-text rationale before generating an output, and this often improves the performance on the based on the given source and reference, identify the major and minor errors in this translation. note that major errors refer to actual translation or grammatical errors, and minor errors refer to smaller imperfections, and purely subjective opinions about the translation. {src_lang} source: "{source}" {tgt_lang} human reference: "{reference}" {tgt_lang} translation: "{candidate}" errors: {error1:span} -{error1:severity}/{error1:category}; {error2:span} -... task at hand @xcite . furthermore, this chain-of-thought prompting can be used to obtain structured rationales from llms, and this can lead to better performance than with free-text rationales @xcite . motivated by these findings, we propose automqm, a prompting technique for translation quality assessment that instructs llms to identify errors in a translation, and categorize the type of error according to the mqm framework @xcite . furthermore, we don't ask the model to produce a score, as the mqm framework provides an algorithmic procedure to obtain one from identified errors: the total score is the sum of penalties for all errors identified, where (roughly) major errors get penalized with -5 and minors with -1 (see appendix a for a more detailed description of the scoring algorithm). 3 figure 2 shows the main automqm prompt used in this paper. importantly, obtaining meaningful automqm results in a zero-shot setting is a substantially more challenging task compared to score prediction: we found that, without any in-context examples, llms tend to produce outputs that are either uninformative or difficult to parse. thus we only consider the automqm task in the few-shot scenario. based on the findings from §6.2, we explore the impact of in-context learning by sampling from the example pool using stratified sampling extended with a set of rejection criteria (appendix d), which ensures that the example set has a balance between major and minor errors as well as diversity in the categories of errors. additionally, in our experiments, automqm required in-context examples with mqm annotations to work, so we restrict our evaluation of automqm to @xmath0. ## conclusion in this study, we have systematically investigated the capabilities of large language models for machine translation evaluation through score prediction, and proposed automqm, a novel prompting technique that leverages the multidimensional quality metrics (mqm) framework for interpretable mt evaluation using llms. we demonstrated that just prompting llms for score prediction leads to state-of-the-art systemlevel evaluators, but still falls short of the best learned metrics at the segment-level (with finetuning being necessary to close this gap). then we showed that automqm can further improve the performance of llms without finetuning while providing interpretability through error spans that align with human annotations. our findings surrounding finetuning llms for score prediction hint that llms' performance in machine translation evaluation could be further improved by finetuning these models on fine-grained human judgment data (like mqm) and is a direction we are actively pursuing. additionally, the general-purpose nature of llms may enable the application of similar prompting techniques (leveraging some fine-grained evaluation schemes) to other evaluation problems @xcite .