LLM s in Post-Translation Workflows: Comparing Performance in Post-Editing and Error Analysis (2024)

Abstract: This study conducts a comprehensive comparison of three leading LLMs—GPT-4, Claude 3, and Gemini—in two translation-related tasks: automatic post-editing and MQM error annotation, across four languages. Utilizing the pharmaceutical EMEA corpus to maintain domain specificity and minimize data contamination, the research examines the models’ performance in these two tasks. Our findings reveal the nuanced capabilities of LLMs in handling MTPE and MQM tasks, hinting at the potential of these models in streamlining and optimizing translation workflows. Future directions include fine-tuning LLMs for task-specific improvements and exploring the integration of style guides for enhanced translation quality.

## introduction large language models (llms) have been at the forefront of many recent advancements in natural language processing. these models show impressive capabilities in a range of tasks, including tasks that were unseen at training time. as modern llms are typically multilingual, machine translation is a natural application of these models. despite initial optimism, so far research has found that well-tuned encoder-decoder models trained specifically for the task tend to outperform llms in most content types in the task of machine translation @xcite . however, promising results have been obtained in the peripheral tasks of machine translation post-editing @xcite and machine translation quality evaluation @xcite . post-editing of machine translation is a common step in modern localization workflows, and can be a significant expense for global organizations producing content in multiple languages. quality evaluation can be used to obtain actionable insights into the sources of machine translation errors, and automated quality evaluation performed at translation time in the production workflow can inform decisions about whether a translation needs additional attention or can be used directly. with the great advancements in generative language models over the last year, the possibility of automating these tasks using large language models (llms) has received growing attention. thus, in this work, we set out to compare the performance of three state-of-the-art llms on these two tasks in four target languages: portuguese for brazil (ptbr), italian (it), german (de), and japanese (ja). ## related research with the advent of llms, several attempts have been made to apply them to different translation tasks. many works explore prompting llms to perform translation and compare their performance with the encoder-decoder based systems @xcite @xcite @xcite . @xcite proposed an adaptive translation workflow using llms. other scenarios include a human-in-the-loop pipeline to guide an llm to produce customized output @xcite or an ai-mediated post-editing process @xcite . there is evidence that llms can be successfully applied for mt quality evaluation. in the first attempt to use a gpt model for this purpose, @xcite demonstrated their potential as zero-shot evaluators. fernandes et al. @xcite then took it one step further and experimented with the automqm methodology: prompting an llm to produce mqm-style annotations of mt errors. this study is motivated by the recent finding that the evaluation methodologies that are based on mqm annotations @xcite ) demonstrate higher correlation with human judgments @xcite . automatic post-editing (ape) of mt is another area where the utilization of llms has been considered. ape consists in using automated techniques to improve the quality of black-box machine translation systems. it has been a popular research topic in the mt community since the times of statistical mt systems @xcite . with the evolution of deep learning, neural models have been increasingly applied to ape tasks @xcite @xcite . a detailed overview on the history of ape can be found in the excellent review article by do @xcite . to our knowledge, two experiments were published so far on utilizing llms for this task. @xcite used gpt-3 for the task and reported promising results while concluded that there was room for improvement in several areas. in a later publication, @xcite reported significant improvements over the initial mt output as well as over the wmt baseline. however, the authors did find it challenging to control the model hallucinations. despite the promising results reported in these studies, the latest wmt task on ape paints a different picture @xcite . out of the three participating systems, one used gpt-3.5-turbo (the other two used non-llm methods), however, it did not show any improvement over the baseline. this was the first time llms were used in this shared task. from a wider perspective, one can notice that the wmt competitions, which serve as an industry and research reference, present a consistent picture across different translation tasks. in the translation task, llms are not "quite there" compared to the encoder-decoder systems @xcite . the same is demonstrated for ape: the transformer systems trained specifically for this task performed better than the more generic llms. in our experiment, we carry out a varied study on llms for automqm evaluation and for automatic post-editing by utilizing several state-of-theart llms. we suggest that our methodology will help the community get a deeper understanding of how efficient llms are for these tasks. our contribution, compared to previous experiments, consists in comparing a variety of llms and their performance on two different translation-related tasks. this way, we can get insights on llms' behavior: do the models rank equally on both tasks? are there any specific models that significantly outperform the rest? ## materials and methods below we describe our process of data selection, the models used at different stages, and the llm prompt development process for each task. ## discussion after having carried out the automatic evaluation of the results obtained on the two proposed tasks (namely, ape and mqm analysis) we can conclude the following: ## future work among our future work plans we intend to explore the fine-tuning of llms for the task of post-edition and mqm and compare the performance and costs with the approach proposed on this paper. finetuning an llm for a certain task has been proven to be a successful technique for achieving better results in certain tasks while reducing costs due to the shorter prompts that need to be sent to the model. another item of interest would be studying the integration of a style guide either by introducing it into the prompt or during fine-tuning. this could be useful for correction of stylistic errors for client customization. moreover, taking into account that our baseline models already achieve state-of-the-art performance, it would be interesting to carry out the same experiments on mt output which is objectively of poorer quality and analyze whether llm post-edition and mqm analysis could significantly improve the translation in those cases. finally, human evaluation of the quality of llm-post-edited content could be performed in order to get a better understanding of the results that were achieved with the automatic metrics presented on this paper. ## appendix a -prompt examples in order to carry out a proper analysis, first think of the error category and once you have that clear, subcategorize the error using the subcategories from that error category.