Leveraging GPT -4 for Automatic Translation Post-Editing (2023)

Abstract: While Neural Machine Translation (NMT) represents the leading approach to Machine Translation (MT), the outputs of NMT models still require translation post-editing to rectify errors and enhance quality under critical settings. In this work, we formalize the task of direct translation post-editing with Large Language Models (LLMs) and explore the use of GPT-4 to automatically post-edit NMT outputs across several language pairs. Our results demonstrate that GPT-4 is adept at translation post-editing, producing meaningful and trustworthy edits to translations that help improve its general quality as well as remove different classes of major errors in translations. In particular, human evaluations on assessing edit trustworthiness show that GPT-4 exhibits a large improvement over the prior state-of-the-art LLM. Notably, we improve upon state-of-the-art performance on WMT-22 English-Chinese, English-German, Chinese-English and German-English language pairs using GPT-4 based post-editing, as evaluated by state-of-the-art MT quality metrics. However, we also show that GPT-4 could produce hallucinated edits, thereby urging caution in its use as an expert translation post-editor.

## introduction state-of-the-art neural machine translation (nmt) models, trained on web-mined parallel corpora suffer from reliability problems even for higher resource language pairs, despite high average case performance @xcite @xcite @xcite @xcite . a number of prior works have demonstrated that the parallel data and model training artifacts in nmt could manifest in terms of catastrophic outputs in rare cases, and the detection of such egregious model behaviors remains a challenging task @xcite @xcite @xcite . thereby, post-editing neural machine translations remains an important exercise for their use in critical settings across the translation and localization industry. as such, a relevant question to ask is whether large language models (llms) such as gpt-3, @xcite @xcite , which have demonstrated a wide-range of general purpose reasoning and knowledge-based capabilities could be leveraged for the task of translation post-editing. llm based automatic translation post-editing could aid in both detecting and fixing translation errors to ensure greater reliability of nmt outputs. besides alleviating reliability problems in nmt, leveraging llms for post-editing could be opportune owing to the advanced multi-lingual understanding capabilities of latest llms, in particular @xcite and potentially, their ability to apply knowledge-based or culture-specific customization to translations @xcite , which are hard to ascertain only through the source text. in this work, we explore the efficacy of state-ofthe-art llms such as gpt-4 on the task of translation post-editing in a direct setting, i.e., without any quality-estimation or error detection step applied to the translations prior to post-editing. our contributions are as follows: (i) we formalize the task of direct translation post-editing with llms and posit a set of research questions and measurements to quantify their utility for the goal of improving translations obtained from nmt models. (ii) we demonstrate that gpt-4 produces meaningful, human-judgement aligned edits to state-ofthe-art translations with a high degree of trustworthiness. (iii) we demonstrate gains on translation quality over best systems from wmt-22 across a number of language pairs on the wmt-22 benchmark @xcite , improving upon the state-of-the-art translation performance on english-example of translation post-editing using gpt-4 (verbatim outputs under the cot setting) source: they were addressed to her son, who has autism and lives in a private care facility, she said. but instead of her son's name inside when you opened them, the letters said dear maine's department of health and human services -in cincinnati, she told local media. translation: sie waren an ihren sohn gerichtet, der autismus hat und in einer privaten pflegeeinrichtung lebt, sagte sie. aber anstelle des namens ihres sohnes im inneren, als sie sie öffneten, hieß es in den briefen dear maine dear maine 's department of health and human servicesservices -in cincinnati, sagte sie den lokalen medien. ## the translation post-editing task task definition: we formalize the post-editing task in a generative setting as follows: given a source (s) and a translation (t ), propose improvements over the translation (e) and generate the translation with the proposed edits (t ′ ), i.e.: under this task setting, e represents the improvements or the edits that are verbalized by a llm. in the absence of e, the task is reduced to simply generating the improved translation without any intermediate reasoning chain or chain of thought (cot) @xcite . throughout this work, we refer to the postediting task in the above zero-shot cot setting as post-editing with cot and the setting without e as post-editing without cot. in the next sections, we explore the above four research questions using the state-of-the-art llms. we describe our experimental settings in section 3 & present the results in section 4. ## experimental settings datasets: we experiment with wmt-22 general mt translation task datasets @xcite as well as with wmt-20 and wmt-21 news translation task submissions annotated with mqm (mul-tidimensional quality metrics framework) errors 1 freitag et al. (2021). for the post-editing experiments pertaining to the mqm annotated wmt-20 and wmt-21 system outputs, we experiment with samples that have a major error as an annotation, whereas we experiment with the full wmt-22 datasets throughout. we use the latest wmt-22 test sets for the majority of our experiments, the curation of which falls beyond the training cut-off dates for gpt-4 and other llms under investigation foot_1 . llms and baselines: we experiment with gpt-4 and gpt-3.5-turbo in our experiments. these models represent the most capable publically available llms @xcite . we use a prompt that describes the system role as a translation posteditor and under the cot setting, instruct the llm to propose improvements to the provided translation (t ) of a given source (s), before producing the final post-edited translation (t ′ ). for postediting, we experiment under three settings: (i) post-editing with cot, (ii) post-editing without cot and (iii) post-editing with structured-cot (scot). the scot baseline uses the mqm annotation instructions from @xcite to produce the intermediate cot in the form of an mqm annotation over the source-translation pair. we describe the prompts used for the three baselines in appendix c. for producing the initial translations on wmt-22, we use microsoft-translator, one of the strongest publically available mt systems @xcite . for wmt-20 and wmt-21 systems, we take the translations provided by the different nmt systems and annotated in @xcite as the initial translations upon which post-editing is applied. ## further discussion post-editing across language pairs: we report the gpt-4 post-editing performance under the cot setting with ms-translator as the initial translation for several other language pairs in appendix b. in general, the results show that gpt-4 based post-editing leads to consistent gains in translation quality across language pairs, with larger gains for x-e translations. utility of the chain-of-thought: our results show that the inclusion of the edit proposals (cot) in the post-editing step is detrimental towards the quality of the post-edited translations, but is useful in constraining the post-edited outputs to the initial translations. therefore, the necessity of variable computation leveraged by the zero-shot chain-ofthought step is questionable for the post-editing task, even though the edit artifacts produced by the gpt-4 might themselves be valuable for making the automatic post-editing task more trustworthy. further, we also found that imposing a specialized structure on the edit proposals in the form of mqm error categories was not valuable towards improving the final post-edited translation quality. on the other hand, we also demonstrated that imposing structure upon the proposed edits in the form of mqm annotations doesn't hurt the general quality of the post-edited translations, suggesting that it might be possible to combine mqm based automatic quality assessment along with post-editing through gpt-4. however, evaluating whether postediting could be done in conjunction with mqm based quality assessment is beyond the scope of our work and we leave such a joint evaluation to future work. ## summary and conclusions we formalized the task of direct automatic postediting in a generative setting and posited a set of research questions and measurements to quantify the utility of the state-of-the-art llm, gpt-4 on the task. through this formalization, we demonstrated that zero-shot chain-of-thought is critical in constraining the post-edited translation to be close to the initial translation. we also demonstrated that gpt-4 produces meaningful human-judgement aligned edits to translations that also lead to general quality improvements, as evaluated on the wmt-22 test sets. further, we demonstrated that the edit generation process in gpt-4 is considerably more trustworthy than a previous generation of llm. overall, we demonstrated promising results on post-editing with gpt-4, improving upon the wmt-22 best translation performance on english-chinese, english-german, chinese-english and german-english language pairs. you are a native speaker of both english and german. you are an expert post editor of translations from english into german. you know that the german translation of a given english text must faithfully represent its meaning in german. the english input text itself might contain any number of different words, including typos and placeholder entities, but still the german translation must remain faithful to the english input text. faithfulness of a german translation means that every word in the translation can be reconstructed from the given english input and vice versa. therefore, you notice any deviations in the faithfulness of the german translations, including the below issues that make the given german translation not optimal: 1. words in the german translation that are not supported in the input 2. words in the english input that are not adequately translated 3. words in the german translation that do not convey the specific meaning of the corresponding word in the input 4. words in the german translation that are not in the correct language 5. punctuations in the german translation that are different from the input 6. symbols in the english input that are not correctly present in the german translation 7. casing in the german translation that does not conform to the english input 8. incorrect modifications in the german translations of names, organizations, entities 9. incorrect modifications in the german translations of any cardinal or ordinal numbers 10. incorrect translations of web terminologies such as urls, web addresses and hashtags in the input 11. incorrect translations of physical units or currencies in the input 12. unsupported expansions of the acronyms present in the input you identify and fix the above twelve issues one by one in the german translation if they are present, in a way that improves the translation fluency. further, as an expert translation post editor, for the improvements made to the german translation, you make sure that the following principles are followed: 1. no corrections are made that add any word or phrase in the translation which are unsupported in the input 2. the capitalizations in the translation strictly follow the input capitalizations, e.g., acronym capitalizations should not be changed 3. the translation contains the appropriate articles and determiners to follow the specifics in the input 4. do not leave any symbol, word or phrase in the input text untranslated in the final, improved translation 5. do not add any extraneous words, phrases, clauses or sentences in the translation that is not supported by the input 6. if the input starts with a non capitalized word, the translation starts with a non capitalized word 7. in the case that the translation is severely inadequate, you generate an improved translation from scratch 8. no end punctuations or full stops are added if such punctuations or full stops are not in the input 9. do not assume that an acronym is a typo, always err on the side of assuming that the presented input words are not typos 10. do not replace any entities or placeholders in the translation with fictitious (unsupported) entities 11. if the input contains offensive or lewd words, you still translate them faithfully 12. if the translation misses to convey the meaning of a large part of the input sentence, you include the translation for the missing part