Terminology-Aware Translation with Constrained Decoding and Large Language Model Prompting (2023)

Abstract: Terminology correctness is important in the downstream application of machine translation, and a prevalent way to ensure this is to inject terminology constraints into a translation system. In our submission to the WMT 2023 terminology translation task, we adopt a translate-then-refine approach which can be domain-independent and requires minimal manual efforts. We annotate random source words with pseudo-terminology translations obtained from word alignment to first train a terminology-aware model. Further, we explore two post-processing methods. First, we use an alignment process to discover whether a terminology constraint has been violated, and if so, we re-decode with the violating word negatively constrained. Alternatively, we leverage a large language model to refine a hypothesis by providing it with terminology constraints. Results show that our terminology-aware model learns to incorporate terminologies effectively, and the large language model refinement process can further improve terminology recall.

## introduction one of the major obstacles encountered by neural machine translation (nmt) systems pertains to the utilization of suitable domain-related words when translating specialized content not present in the training data. an illustrative instance of this challenge arises when translating "transformer" from english into another language, where the accurate translation depends on the context or the preference of the audience (figure 1 ). a straightforward literal translation approach often leads to suboptimal outcomes, prompting human translators unfamiliar with domain-specific knowledge to resort to reference materials for terminology precision. this issue is prevalent in the translation industry, with many commercial translation service providers offering paid solutions to address it. furthermore, it * equal contribution. translate "transformer" to chinese? 变压器 (electric transformer) 变形金刚 (the transformer character) 变换器 (something that changes) figure 1 : terminology hints can help disambiguate polysemantic words when translating with limited context. we interpret that the no-constraint setting allows us to measure the competing systems' quality and understand to what degree the systems effectively utilize the provided random and terminology dictionaries. our baseline approach is to train a terminology-aware translation (tat) system inspired by @xcite , where, in the training data, source words are tagged with desired translations inline on the source side. then we propose two separate refinement strategies on top of it to aggressively encourage the appearance of terminologies: our proposed techniques can incorporate target terminology words with around 80% recall, using automatic and soft constraints in a two-step refinement process. we observe that for german-english, our terminology-aware training and negatively constrained decoding perform better, whereas, for chinese-english and english-czech, llm-based refinement achieves higher scores. in terms of overall translation accuracy, we find that negatively constrained decoding could lead to a tiny drop and llms are able to maintain or improve quality according to a reference-free neural metric. ## related work previous research on terminology translation could be divided into two categories: soft constraint and hard constraint, depending on whether the resulting translation system will enforce the appearance of desired target translations. in the soft constraint setting, the convention is to train a model that is able to ingest the target terminology words inline, directly placing them after the corresponding source words in the source input @xcite . many later implementations stem from this to include new elements such as additional lemmatization @xcite or grammatical error correction @xcite as a post-processing step in order to achieve a more fluent output. instead of placing the target constraint words inline, some other works train a system that takes the terminology constraint as either a prefix or a suffix @xcite . most hard constraint work involves postprocessing a translation with desired terminologies. @xcite inserted untranslatable tokens (also known as placeholders) into the source, which will remain unchanged through the translation process. then the placeholders are replaced with terminology words in the target language. this is entirely performed as a post-processing step. such terminology replacement could also be done by keeping and replacing the source word at inference time, and it is also feasible to run target word replacement as post-processing @xcite . a hard constraint method guarantees that the chosen terminology token will appear, but often results in less fluent output, especially for morphologically rich languages because the context is not taken into consideration during replacement. it also mandates more complicated post-processing than the soft constraint approaches. our first post-processing proposal relies on constrained decoding, which refers to either allowing certain tokens or blocking specific tokens during inference time @xcite . it has been applied to terminology injection, paraphrasing, parallel sentence mining, etc @xcite @xcite . we opt for negatively constraining the tokens that violated the given terminology alignments by preventing them from entering the hypothesis beam in the refinement stage. these alignments are computed using word alignment tools @xcite . another post-processing method in our study prompts an llm to refine a translation and incorporate terminology terms simultaneously. whilst previous studies have explored the translation capability of llms @xcite , the works closely relevant to us are from @xcite and @xcite . we adopt the paradigm from the latter, which re-words a constraint dictionary as a natural text and affixes it into a translation prompt. while they focused on rare words without directly benchmarking on terminology translation, our post-processing step can be seen as an extension of word-level controlled prompting to terminology translation with large language models. both of our post-processing methods should be categorized as soft constraint approaches since there is no guarantee that negatively constrained decoding or an llm will necessarily incorporate the constraints in a re-generation. ## terminology-aware training the goal of our system implementation is to create a general-purpose terminology-aware translation system that is unsupervised and domain-agnostic, and requires the minimum effort of pre-and postprocessing. ## post-translation terminology injection despite training our model with terminology awareness, there is no mechanism to ensure that the desired terminology constraint will appear on the target side. the neural network decoding behaviour is not entirely predictable, especially given the assumption of no additional domain adaptation. below, we present two distinct strategies to try harder to promote the terminology constraints, via automatic post-editing through constrained beam search and large language models. ## results and discussions we present our blind test results in ## conclusion and future work we participated in all tracks of the wmt 2023 terminology shared task with a terminology-aware translation baseline, and two distinct refinement procedures using negatively constrained beam search and large language models separately. the results we produced gave us insights into the pros and cons of our systems. in future work, we could explicitly enforce the generation of the terminology token by identifying the appropriate time step and manipulating the probability distribution after softmax computation, even in an open-source large language model. this is not entirely trivial due to the presence of subwords but could be achievable.