Translate-and-Revise: Boosting Large Language Models for Constrained Translation (2024)

Abstract: “Imposing constraints on machine translation systems presents a challenging issue because thesesystems are not trained to make use of constraints in generating adequate, fluent translations. Inthis paper, we leverage the capabilities of large language models (LLMs) for constrained trans-lation, given that LLMs can easily adapt to this task by taking translation instructions and con-straints as prompts. However, LLMs cannot always guarantee the adequacy of translation, and,in some cases, ignore the given constraints. This is in part because LLMs might be overly confi-dent in their predictions, overriding the influence of the constraints. To overcome this overidingbehaviour, we propose to add a revision process that encourages LLMs to correct the outputs byprompting them about the constraints that have not yet been met. We evaluate our approach onfour constrained translation tasks, encompassing both lexical and structural constraints in mul-tiple constraint domains. Experiments show 15% improvement in constraint-based translationaccuracy over standard LLMs and the approach also significantly outperforms neural machinetranslation (NMT) state-of-the-art methods.IntroductionConstrained translation seeks to generate translations that adhere to pre-specified constraints. Toachieve this, conventional approaches impose constraints on machine translation systems and force themto follow the constraints during inference (Hokamp and Liu, 2017; Hasler et al., 2018; Dinu et al., 2019;Bergmanis and Pinnis, 2021b; Wang et al., 2022b; Ailem et al., 2022). More recently, large languagemodels (LLMs) have been shown to be strong translation systems (Hendy et al., 2023; Moslem et al.,2023). They provide a general way to involve various instructions, demonstrations, and constraints intothe translation process (Mu et al., 2023; Bogoychev and Chen, 2023), enabling us to perform constrainedtranslation using off-the-shelf, well-trained LLMs.”

## introduction constrained translation seeks to generate translations that adhere to pre-specified constraints. to achieve this, conventional approaches impose constraints on machine translation systems and force them to follow the constraints during inference @xcite @xcite @xcite . more recently, large language models (llms) have been shown to be strong translation systems @xcite . they provide a general way to involve various instructions, demonstrations, and constraints into the translation process @xcite ), enabling us to perform constrained translation using off-the-shelf, well-trained llms. while applying llms to constrained translation is straightforward, we observe empirically that even strong llms (i.e. gpt-3.5) do not always follow the instructions to obey constraints: llms' predictions often override the guide of constraints, which result in missing constraints during translation. see figure 1 for an example where we use an llm to translate an english sentence to a chinese sentence with a lexical constraint "covid-19→新型冠状病毒". we note that, despite significant effort in developing clear and instructive prompts, we were not able to improve the llm in a single run of the llm through the use of these constraints. for instance, we observed that when using open-source llm to translate covid-19, it tends to translate it as "新冠" more than 80% of the time, overlooking the constraint in the prompt to translate covid-19 as "新型冠状病毒". the problem consists of a real use case for what describes as 'memo trap' in the llm literature @xcite . to alleviate this problem and thus improve the accuracy to meet constraints, we propose to construct prompts iteratively that enable better focus on the unsatisfied constraints. the idea behind our approach is to leverage the auto correction skills of llms by explicitly prompting them with which constraints are not satisfied @xcite @xcite . to do this, we introduce a revision step after the initial run of llms where we provide the llms with both the already-generated translation and the constraints that have not been covered. then, we instruct the llm to revise its output by taking these constraints into account. we conduct experiments across four diverse constrained translation datasets, encompassing two distinct constraint types: lexical and structural. our proposed "translate-and-revise" (tar) approach consistently elevates the performance of llms in constrained translation, achieving state-of-the-art (sota) results on multiple datasets. the contributions of this work are as follows: ## experiments in this study, we evaluate the performance of the tar in constrained translation. while most previous research has typically focused on just one or two constrained translation tasks @xcite @xcite , our evaluation expands to two types of constraints: lexical constraints and structural constraints, covering four practical scenarios: general lexically constrained translation, translation with terminology constraints, translation with named entity constraints, and structured document translation. ## analysis specifically, we first the wmt21 champion model @xcite to obtain a preliminary translation result. since this model is not specifically trained for constraints, the initial translation often exhibits a suboptimal ccr. building on this, we apply tar to revise this outcome, iteratively optimizing to form the final translation result. and gpt-4. here "w/o tar" represents the use of the conventional translation prompt. "tar w/o revision" indicates the use of a prompt with constraints, but without reviser. meanwhile, "tar" denotes the full method that includes revisions. the experimental results on wmt21 tt and etc datasets are presented in ## conclusion in this work, we introduce the tar prompting method, adeptly leverages llms for constrained translation. our approach involves a two-step process: first using llms for constrained translation, and subsequently deploying them to revise translations with uncompleted constraints. our approach mainly improves the constraint accuracy while maintaining translation quality by overcoming the 'memo trap' from the llms during translation using dedicated revision prompts in an iterative manner. we further show that tar can be applied to llm based translation systems as well as traditional nmt systems, in both cases resulting in better constraint accuracy while maintaining translation quality and the technology is not limited to particular llms. more generally, our study sheds light on the importance of accurate feedback in general for llm revision to work effectively.