Iterative Translation Refinement with Large Language Models (2024)

Abstract: We propose iteratively prompting a large language model to self-correct a translation, with inspiration from their strong language capability as well as a human-like translation approach. Interestingly, multi-turn querying reduces the output’s string-based metric scores, but neural metrics suggest comparable or improved quality after two or more iterations. Human evaluations indicate better fluency and naturalness compared to initial translations and even human references, all while maintaining quality. Ablation studies underscore the importance of anchoring the refinement to the source and a reasonable seed translation for quality considerations. We also discuss the challenges in evaluation and relation to human performance and translationese.

## introduction large language models (llms), e.g. generative pretrained transformers (gpt), have made notable advancements in natural language processing @xcite @xcite . in machine translation (mt), where the convention is to use an encoderdecoder architecture to deal with source and target sentences respectively @xcite , recent papers have examined the feasibility of llm prompting for translation @xcite @xcite . with autoregressive decoding being the convention, machine translation models yield output in a single attempt, and so do post-editing models. rather, a human translator can read and edit translations repeatedly, or even pass the outcome to another translator for a second opinion. we explore such an iterative refinement process with llms, where the proposed method simply feeds a sourcetranslation pair into an llm for an improved translation in multiple rounds. it is worth noting that this method can be applied to an initial translation from any model, not just llm outputs. we further conduct a qualitative evaluation of the outputs. our approach offers two insights from a fluency and naturalness perspective: 1) llms are pre-trained on natural texts that are orders of magnitude larger than traditional mt data, and 2) the method does not require complicated prompt engineering, yet allows for iterative and arbitrary rephrasing compared to automatic post-editing, which is limited to token-level error correction without style editing @xcite . empirical results show that the refinement procedure introduces significant textual changes reflected by the drop in bleu and chrf++, but attains similar or higher comet scores compared to initial translations. native speakers prefer refined outputs in terms of fluency and naturalness when compared with gpt translations and even human references. reference-based human evaluation confirms that such gains are made without sacrificing general quality. as corroborated by recent works, automatic metrics like bleu and comet are witnessed to move in opposite directions @xcite . our human-like llm prompting method contributes to translation naturalness which can enhance utility as perceived by the target language users. on a broader scope, this work touches on the concept of involving llms in a collaborative translation editing strategy. ## methodology having an input source sentence x and an optimizable model θ mt , the process to obtain a translation y can be modelled as @xmath0; θ mt ). next, an automatic post-editor θ ape creates a refined translation y ′ through modelling y ′ = argmax y ′ p (y ′ |x, y; θ ape ). conventional translation or automatic post-editing models are trained on (x, y) or (x, y, y ′ ) data pairs. extending prior work on llm prompting, our study uses zero-shot prompting by affixing a task description to form a prompt p and querying an llm θ llm to elicit a response @xcite . we introduce five prompts in our study: we propose to iteratively call the refinement prompts, where the source stays the same but the previous translation is updated each turn. to encourage a parsable model response, we ask the llm to not give any explanation. such prompting does not require model parameters θ llm to be accessible. through ablation prompts, refine random and paraphrase, we analyse to what degree the source input and seed translations are helpful. the exact prompt texts are displayed in ## human evaluation string-based and neural scores are observed to vary in opposite directions, which may suggest volatile changes in texts. since it is questionable to conclude a quality degradation in this case, we set up human evaluations to measure two characteristics in the refined translations: text naturalness and overall quality. human evaluators involved in this study are practitioners in the field of natural language processing but are unaware of the goal of this study. ## conclusion and future work we presented a simple way to leverage an llm for translation refinement, which greatly helps fluency and naturalness. it is shown that our method maintains translation quality and introduces lexical and structural changes, especially for high-resource into-english translation. we have also discussed the potential of using our work to obtain diverse, fluent translations that are less translationese, as well as the limitation in automatic metrics to measure this. on a broader note, this work connects to the concept of using llms to imitate collaborative translation refinement. yet, it is important to acknowledge the high cost of running a multi-round llm refinement. future work can explore sentence-level refinement decisions to reduce cost.