Error Analysis Prompting Enables Human-Like Translation Evaluation in Large Language Models (2024)

Abstract: Generative large language models (LLMs), e.g., ChatGPT, have demonstrated remarkable proficiency across several NLP tasks, such as machine translation, text summarization. Recent research (Kocmi and Federmann, 2023) has shown that utilizing LLMs for assessing the quality of machine translation (MT) achieves state-of-the-art performance at the system level but performs poorly at the segment level. To further improve the performance of LLMs on MT quality assessment, we conduct an investigation into several prompting designs, and propose a new prompting method called Error Analysis Prompting (EAPrompt) by combining Chain-of-Thoughts (Wei et al., 2022) and Error Analysis (Lu et al., 2023). This technique emulates the commonly accepted human evaluation framework - Multidimensional Quality Metrics (MQM, Freitag et al., (2021)) and produces explainable and reliable MT evaluations at both the system and segment level. Experimental Results from WMT22 metrics shared task validate the effectiveness of EAPrompt on various LLMs, with different structures. Further analysis confirms that EAPrompt effectively distinguishes major errors from minor ones, while also sharing a similar distribution of the number of errors with MQM. These findings highlight the potential of EAPrompt as a human-like evaluator prompting technique for MT evaluation. We will release our code and scripts to facilitate the community.

## introduction large language models (llms), especially generative pre-trained transformer (gpt) models @xcite such as chat-gpt @xcite , have shown remarkable performance in various natural language processing (nlp) tasks @xcite and complex reasoning and agent applications @xcite . llms are capable of integrating multiple nlp tasks and can generate detailed and comprehensive responses to human inquiries. additionally, they can respond appropriately to follow-up questions and maintain sensitivity throughout several turns of conversation. previous research has demonstrated that llms can perform as well as or even better than other llms in machine translation task @xcite @xcite . given the high cost and time-intensive nature of human evaluation, there is a growing demand for mt metrics that offer both explainability and reliability. therefore, llms hold promise in serving as ideal evaluators, capable of generating both judgments and explanations for the translations. concurrent to our research, gemba @xcite ) presents an encouraging finding that gpt models can surpass current best mt metrics at the system level quality assessment using straightforward zero-shot standard prompting, confirming the reliability and potential of this technique. however, such prompts exhibit unrealistic performance at the segment level, and cannot offer additional interpretable information regarding translation errors, thus detracting from the goal of achieving a "human-like" evaluation. to this end, we take the further step by carefully investigating advanced prompting strategies upon various llms for mt quality assessment and propose a novel prompting strategy -error analysis prompting (eaprompt), combining the chain-of-thought (cot, @xcite ) and error analysis (ea, @xcite ). we give an example of eaprompt in figure 1 . the idea is to prompt llms to emulate the human evaluation framework -mqm @xcite by ❶ identifying ma-jor&minor errors, and ❷ scoring the translations according to the severity of these errors. we conduct experiments using the test set from the wmt22 metrics shared task, compris-q: source : 迈克去书店。 reference : mike goes to the bookstore. translation : jerry went bookstore happily. based on the given source and reference, identify the major and minor errors in this translation. note that major errors refer to actual translation or grammatical errors, and minor errors refer to smaller imperfections, and purely subjective opinions about the translation. a: major errors: (1) "jerry" -mistranslation (2) "happily" -addition minor errors: (1) "went" -grammar (2) "the" -omission q: source : 猫在垫子上。 reference : the cat is on the mat. translation : the mat sat on the mat. based on the given source and reference, identify the major and minor errors in this translation. note that major errors refer to actual translation or grammatical errors, and minor errors refer to smaller imperfections, and purely subjective opinions about the translation. a: major errors: (1) "mat" -mistranslation (2) "sat" -verb tense and agreement error minor errors: none q: based on the above error information, output 2 numbers only with the format: "x, x", indicating the number of major and minor errors. do not add other information! a: 2, 0 q: source : 迈克去书店。 reference : mike goes to the bookstore. translation : jerry went to bookstore happily. score the following translation from en to de with respect to the human reference on a continuous scale from 0 to 100, where a score of zero means "no meaning preserved" and score of one hundred means "perfect meaning and grammar" a: 30. q: source : 猫在垫子上。 reference : the cat is on the mat. translation : the mat sat on the mat. score the following translation from en to de with respect to the human reference on a continuous scale from 0 to 100, where a score of zero means "no meaning preserved" and score of one hundred means "perfect meaning and grammar" a: 40. ## conclusion in this paper, we explore the potential of llms as a metric for evaluating translations. we design a novel one-shot prompting strategy eaprompt based on chain-of-thought and error analysis, and show that this strategy significantly improves the evaluation performance on both the system and segment levels. we compare different eaprompt variants and ultimately opt for a 2-step prompting approach with itemized error demonstrations. further analysis confirms eaprompt's proficiency in error identification and its alignment with the commonly accepted human evaluations mqm. in future work, we would like to experiment with a broader range of llms @xcite @xcite , to make our conclusion more convincing. lastly, it will be interesting to test the capabilities of llms for other mt-related tasks, such as grammatical error correction and automatic post-editing @xcite .