Guiding Large Language Models to Post-Edit Machine Translation with Error Annotations (2024)

Abstract: Machine Translation (MT) remains one of the last NLP tasks where large language models (LLMs) have not yet replaced dedicated supervised systems. This work exploits the complementary strengths of LLMs and supervised MT by guiding LLMs to automatically post-edit MT with external feedback on its quality, derived from Multidimensional Quality Metric (MQM) annotations. Working with LLaMA-2 models, we consider prompting strategies varying the nature of feedback provided and then fine-tune the LLM to improve its ability to exploit the provided guidance. Through experiments on Chinese-English, English-German, and English-Russian MQM data, we demonstrate that prompting LLMs to post-edit MT improves TER, BLEU and COMET scores, although the benefits of fine-grained feedback are not clear. Fine-tuning helps integrate fine-grained feedback more effectively and further improves translation quality based on both automatic and human evaluation.

## introduction machine translation (mt) remains one of the last nlp tasks where large language models @xcite have not yet replaced dedicated supervised systems. llms such as chatgpt @xcite started outperforming commercial mt systems very recently @xcite @xcite . however, supervised models continue to outperform llms in numerous language pairs @xcite , and the performance of llms remains uneven, exhibiting significant variation across models, languages, and translation directions @xcite . this suggests that llms and supervised systems possess complementary strengths, and that combining them should offer some benefits. naturally tanned with sunflower oil only, no use of dye or chemicals. natürlich nur mit sonnenblumenöl verdünnt, keine farbstoffe oder chemikalien. ## method we consider two strategies for guiding language models to edit mt error annotations: prompting and fine-tuning with instructions. ## prompting results figure 2 shows the zero-and 10-shot prompting performance of llama-2 7b across three language pairs. the complete results in table form for both llama-2 7b and 13b can be found in appendix e. ## analysis by mt error categories our error analysis aims to pinpoint the types of errors that are most effectively resolved through the integration of external feedback. we evaluate 200 output translations generated by prompting llama-2 7b with each generic, score-based, and mqm feedback. we do not include instructscore or xcomet as instructscore annotates more than 1 error spans making it difficult for fair comparison and xcomet does not output error type information. we also compare the outputs from our custom fine-tuned models, both bilingual and multilingual version. all of the feedback is based on mqm, thus we categorize the error type as per "error category" from mqm detailed in appendix in figure 3 , we illustrate the extent to which each error type has been resolved by incorporating external feedback. first, we check whether a span annotated as an error in the original translation matches the output after post-editing with feedback. a match increments the count for the error type associated with the span. if there is no match found, the count for the "no match" category is incremented. we observe that using any form of feedback (generic, score, or mqm) increases the portion of "no match" category compared to the original translation. however, there is no distinct trend for any specific error type; all of the errors are addressed in a balanced manner. further, by incorporating the output translations from our fine-tuned model, we see a sudden leap in the "no match" category. this suggests that fine-tuning best fixes the targeted error span. this finding is also consistent with the conclusions from section 6, where we noted that fine-tuning help align llm behavior with the provided feedback. ## post-editing correct outputs the experiments we have presented so far are focused on post-editing mt hypotheses that are known to leave room for improvement. for completeness, we present in appendix as expected, the resulting edits lead to a small drop in automatic metrics, confirming the observation that the nature of edits goes beyond correcting errors to address more stylistic issues such as translationese. interestingly, the larger llama-2 model and the fine-grained feedback are the least prone to over-editing. we anticipate that different prompts and fine-tuning data are needed for models to jointly consider the task of editing or not, and of what edits to perform. ## conclusion we explore a range of strategies to guide llama-2 models to improve mt outputs using external feedback, varying in different granularity. we demonstrate that prompting llm to edit mt with feedback reliably enhances the overall translation quality and post-editing efforts. we further explore instruction fine-tuning llms with fine-grained feedback. through automatic and human evaluation, we demonstrate that fine-tuning shows stronger improvements on enhancing the translation quality, resolving errors in the initial translation, and most notably, generating translations that are more natural (less translationese) in the target language. taken together, these results clearly show that post-editing mt output does not require the largest proprietary llm models and can be done with smaller open-source models. this opens many questions for future work to further explore how to do this well in more diverse settings, while minimizing the reliance on human annotated mt outputs which are expensive to obtain at scale. building on llms fine-tuned for many translation related tasks @xcite is a promising direction for encouraging transfer learning from limited amounts of annotation.