What do Large Language Models Need for Machine Translation Evaluation? (2024)

Abstract: Leveraging large language models (LLMs) for various natural language processing tasks has led to superlative claims about their performance. For the evaluation of machine translation (MT), existing research shows that LLMs are able to achieve results comparable to fine-tuned multilingual pre-trained language models. In this paper, we explore what translation information, such as the source, reference, translation errors and annotation guidelines, is needed for LLMs to evaluate MT quality. In addition, we investigate prompting techniques such as zero-shot, Chain of Thought (CoT) and few-shot prompting for eight language pairs covering high-, medium- and low-resource languages, leveraging varying LLM variants. Our findings indicate the importance of reference translations for an LLM-based evaluation. While larger models do not necessarily fare better, they tend to benefit more from CoT prompting, than smaller models. We also observe that LLMs do not always provide a numerical score when generating evaluations, which poses a question on their reliability for the task. Our work presents a comprehensive analysis for resource-constrained and training-less LLM-based evaluation of machine translation. We release the accrued prompt templates, code and data publicly for reproducibility.

## introduction recent surge in the use of large language models (llms) for natural language processing (nlp) tasks like question answering @xcite has taken strides, and significantly improved their applications to other downstream tasks such as machine translation (mt), text summarization, information retrieval and etc., due to advancements in natural language understanding capabilities, contextual awareness, and a versatile knowledge base @xcite @xcite . for automatic evaluation of mt quality, traditional approaches use metrics such as bleu @xcite , bleurt @xcite or bertscore @xcite to compare mt output with a reference translation. when references are not available, quality estimation (qe) methods such as fine-tuning multilingual pretrained language models (ptlms) on human evaluation data like direct assessment (da) scores @xcite are often used to predict estimated scores to approximate human evaluation @xcite . recent studies leverage prompting techniques and instruct llms to output a score for translation quality, claim to achieve promising results @xcite . however, there exists no systematic exploration of what translation information llms need for quality evaluation, and whether different prompting techniques, such as chain-of-thought (cot) @xcite or few-shot prompting, can help boost the performance of llms. to that end, we conduct this investigation to systematically explore the ability of llms in quality evaluation in a training-less scenario. our contributions can be summarized as: the rest of the paper is structured as follows: section 2 discusses relevant work in quality evaluation, while section 3 introduces the dataset we utilize for this work. section 4 describes the prompting methods and the baselines with the experimental setup. results and discussion are presented in section 5. section 6 concludes our study and outlines future directions. ## related work traditional automatic mt quality evaluation metrics such as bleu, bleurt and bertscore compare the mt output to one or several references, whilst metrics like translation error rate (ter) @xcite are based on the number of edits required for mt output to become reference, and neither takes semantic variations into account. training supervised machine learning systems on human-annotated data based on metrics such as da or multi-dimensional quality metrics (mqm) @xcite can help predict translation quality without any references @xcite . @xcite proposed the transquest framework to utilize the source text and mt output only and finetune xlm-roberta-large @xcite to predict a da score as an estimation of translation quality. comet @xcite @xcite was proposed initially to incorporate references along with the source and mt output to train multilingual ptlms for quality evaluation, but later it also supported reference-less evaluation. @xcite proposed a unified translation evaluation framework that could include source or reference or both as input for quality evaluation. various approaches achieved promising results in the qe shared task of the conference on machine translation (wmt) @xcite @xcite , however, most require supervision and training @xcite . the advent of llms prompted its application to translation quality evaluation. @xcite proposed a zero-shot prompting technique, called gemba for da score prediction using gpt-4 (openai et al., 2024), claiming llms can achieve performance comparable to state-of-the-art models fine-tuned on multilingual data. based on the gemba prompt, fernandes et al. (2023) proposed to use llms for both da score prediction and error categorization via fine-tuning to achieve more fine-grained evaluation. previous research focused on whether llms can be better translation evaluators than state-of-theart models. to the best of our knowledge, only huang et al. ( 2024 ) investigated how llms leverage the source and reference for quality evaluation. however, they only perform zero-shot prompting for three language pairs. our work comprehensively examines factors such as translation errors and annotation guidelines across eight language pairs, eight prompt templates, and three different prompting techniques. ## data we utilized the da score prediction data released with wmt22 qe shared task @xcite . this dataset includes the source (mainly from news articles), mt output (from different mt engines) and (post-edited) human references for eight language pairs, i.e., english-german (en-de), english-marathi (en-mr), english-chinese (en-zh), estonian-english (et-en), nepali-english (ne-en), romanian-english (ro-en), russian-english (ru-en) and sinhala-english (si-en). for each source-mt pair, the dataset contains a da score ranging from 0 to 100, rated by human annotators for quality assessment. to include annotated errors in the mt output into our prompts, we obtained word-level qe data from wmt22, where tokens of the mt output have sequence labels with either "ok" or "bad" indicating translation quality at word level. this dataset also involves the above 8 language pairs, which contains the source, mt output and the tags for translation quality. for each mt output, we extracted the tokens that were tagged as "bad" as error words. since source-mt segments from sentence-level qe data might differ from those of word-level, we compared each source-mt pair in the two datasets and used the overlapping as the main resource of our research. it includes source, mt output, reference translations and error words for the 8 language pairs, covering high-, medium-and low-resource languages. we present different prompt templates in section 4.2 to selectively include source, reference and error words to test what translation information llms need for quality evaluation. we split the data into training, validation, and test sets in proportions of 80%, 10%, and 10% respectively. we inferenced with llms on the test set to obtain evaluation results. training and validation sets were used to sample examples for few-shot learning (see section 4.4). the size of the test set for each language pair can be seen in ## methodology this section presents the baselines, and our zeroshot, cot and few-shot prompting methods. ## results and discussion this section displays our baseline and prompting results using existing qe frameworks and llms. ## conclusion and future work in this paper, we explored what translation information is needed for llms to evaluate mt quality. we conducted a comprehensive investigation into different prompting techniques such as zeroshot, cot and few-shot prompting using different translation information for 8 language pairs and 6 llms of different sizes and types. our findings suggest that the source, mt output and reference are essential compared to other information such as translation errors for quality evaluation. larger models may not necessarily perform better than smaller models, but cot prompting works better on larger than smaller model variants. we also observe that llms do not always provide a numerical score when generating evaluations, which makes their assessments less reliable. for future research, we plan to explore whether fine-tuning llms could improve their performance in quality evaluation. we also plan to thoroughly investigate error explainability of llms using mqm and other fine-grained error identification techniques. these future studies can inform downstream error correction through automatic post-editing, contributing to a more comprehensive evaluation and correction framework.