Coming to Terms with Glossary Enforcement: A Study of Three Approaches to Enforcing Terminology in NMT (2023)

Abstract: Enforcing terminology constraints is less straight-forward in neural machine translation (NMT) than statistical machine translation. Current methods, such as alignment-based insertion or the use of factors or special tokens, each have their strengths and drawbacks. We describe the current state of research on terminology enforcement in transformer-based NMT models, and present the results of our investigation into the performance of three different approaches. In addition to reference based quality metrics, we also evaluate the linguistic quality of the translations thus produced. Our results show that each approach is effective, though a negative impact on translation fluency remains evident.

## introduction ensuring translations use the preferred term can be business-critical for commercial translation providers. while there are existing methods to ensure the correct translation of specified terms, the impact of these methods on translation quality merits closer inspection. typically, they have been evaluated in terms of general translation metrics such as bleu, in addition to the accuracy of the terminology translation. however, there is a dearth of more detailed linguistic analysis of the performance of different techniques; for example, how often do the terms agree morphologically with the rest of the sentence? what are the potential issues when unruly, real-world, client glossaries are applied to models trained in more controlled laboratory conditions, and what steps can be taken to mitigate these issues? in the present work we implement three approaches to glossary/terminology enforcement in two language pairs (english-russian and japanese-english) and compare their performance on the terminology enforcement task. in particular, we investigate two methods based on interventions in the training data and one post-processing method which uses the model's attention mechanism to identify the tokens representing the translation of the input term in the output and replaces these tokens with the translation from the glossary. in addition to automated evaluation (chrf, comet, and accuracy), we also engaged professional linguists to design a test set of edge cases from their particular language pairs, and evaluate the performance of each approach using this bespoke test set. the ultimate objective of this research is to inform the implementation of a glossary feature for use by machine translation project managers and end users, and thus we must anticipate that the feature will be applied in a multitude of unexpected ways. for a guide to what our feature may be subjected to, we turned to a database of historical glossary enforcement requests kept by our company. these requests were created by a mixture of linguists, clients, and project managers in translation projects. the contents of these glossaries are very noisy and diverse, including nouns, adjectives, verbs, prepositions, numbers, and acronyms, and ranging in length from single characters to entire sentences. this resource served both as the source material to annotate our training data for the methods using data intervention, and the inspiration for our test cases. in addition to the practical motivation of our research, we hope to provide the mt community with an insight on the linguistic effects that each of these methods have on the translation output. below we share our methodology and the results of our experiments. ## related work the first approaches to introducing terminology enforcement in nmt were quite limited in terms of handling languages with inflections. for example, in one approach, a special placeholder token was used to mask the term in the source sentence, and then replaced with the correct term after the translation @xcite . in the more sophisticated alignment method, one of the attention heads of the transformer is trained with statistical word alignments, and the output of this attention head at translation time is used to identify the tokens in the translation that correspond to the source term, and replace this token by the translation from the glossary. while this method provides an improvement, it still poses a problem for languages with inflections, since the target term is inserted in its glossary form, and dependencies may be produced in the wrong form. in the constrained decoding method, the nmt decoder is guided to produce translation candidates that include the specified translation of a given source term that is present in the input sentence @xcite @xcite . this method, while certainly producing more fluent translations, adds a significant computational overhead @xcite . since our applications of mt include several time-sensitive use cases, such as chat and instant website translation, we did not consider the constrained decoding method for our experiments. later, @xcite proposed a method where intervention was made in the training data: they insert the target term directly in the source sentence and use factors to signal which tokens are actual source text and which are target translations. factor embeddings are concatinated to the token embeddings and the two are learned in parallel. through training, the model learns to essentially copy the input tokens marked as translations. more information on the practical implications of implementing this approach in a real-life production setting can be found in @xcite and @xcite address the application of this method to morphologically-rich languages. @xcite propose another approach to manipulate the training data: instead of using the source factors, they use special tokens to mark the source and target terms inserted in the source sentence. in addition, the authors apply token masking, which helps the model generalize better on unseen terms, and adapt the weighted cross-entropy loss to bias the model towards generating constraint tokens, resulting in improved translation quality and correctly generated constraint terms. this approach also accounts for different morphological variations of terms both on the source and on the target side by applying string-based approximate matching. until recently, most works only evaluated their results in terms of bleu scores and accuracy of the terminology enforcement. however, they did not provide any insight into how well the term fits in the sentence, if the surrounding translations are correct, etc. for this reason, @xcite proposed new metrics that can reflect correctness of terminology. in particular, they suggest to look at the tokens surrounding the term and compare them to the reference translation (window overlap) and to compute terminology-focused ter @xcite . these metrics are designed to complement the exact-match accuracy and the holistic mt quality metrics and were subsequently used in the first shared task dedicated to terminology in nmt @xcite . since the experiments described above demonstrate that terminology constraints can be successfully applied in nmt without a significant overall performance loss and computational overhead, we choose two methods that are most suitable for our production settings, as well as a baseline method (replacing target tokens by the correct term translation based on the word alignments) to analyse each method's advantages. our goal is not only to measure terminology accuracy and overall model performance, but also to get insight on how naturally the terms are incorporated into the target sentence. ## materials and methods we implemented three approaches to glossary enforcement: alignment-based replacement, annotation with special tokens as per @xcite , and factorization as per @xcite . as a control, we also obtain translation from a model trained with the same data without any terminology intervention. ## evaluation human and automated evaluation methods were used to judge the performance of each approach. for the human evaluation, we worked with linguists to design test sets covering different morphological forms and specific edge cases identified for their languages. the morphological forms covered included adjectives, verbs, simple nouns in nominative, plural, and genitive forms, phrasal nouns and verbs, and entire clauses. for example, the enru test set contained, among regular nouns and noun phrases, terms like men's, go back, turned off. these terms are usually not recommended to be applied in the mt context, but they are often found in client glossaries, so we wanted to understand the behavior of different terminology enforcement methods in these scenarios. among the edge cases tested were the japanese elision of the subject and other cases where grammatical differences between the languages create ambiguity. in total, there were jaen test set. once we had the test sets created, we requested native linguists in the target language to provide two different translations for each selected term. then, we found sentences that contained the source terms amoung our internal datasets or asked the linguists to artificially create them. these sentences were used for the human evaluation. during the human evaluation stage, evaluators were presented with translations of these sentences from the four different systems: the control system with no glossary enforcement, the system trained with the annotation approach, the system trained with the factors approach, and the system where the target term is inserted based on the alignments. for each source sentence, we first enforced the first translation of the term and then the second one. the linguists were asked the following questions about each of the translations: (a) is the term present in the translation? (b) is the term in the correct grammatical form? (c) are the grammatical dependencies on the term in the correct form? (d) does the term assume a non-existent form? (e) are there any duplicated words? (f) rate the overall accuracy of the translation from 1 to 10. (g) rate the overall fluency of the translation from 1 to 10. as the size of these bespoke test sets was necessarily quite small, the statistical significance of the results was not calculated and only the raw results are presented. for the automated evaluation, we used publicly available corpora for comparability. for the english-russian language pair, data from the wmt shared task on terminology enforcement were used. due to the lack of a public corpus designed for terminology enforcement in the japanese-english language pair, the bilingual corpus of wikipedia's kyoto articles1 and its accompanying lexicon were adapted. we selected terms without non-letter characters that were identified as organizations, proper names, or works of art using spacy's ner function. finally, we filtered both corpora to remove any sentences that did not contain terms to be enforced. for terms with multiple glossary translations, the form used in the reference translation was enforced. translations were scored with comet and chrf, and the number of exact and fuzzy matches were counted. exact match was defined as a 100% substring match with word boundaries on either side, 1https ://github.com/venali/bilingualcorpus and a fuzzy match was defined as at least 80% sub-string match. the threshold for statistical significance was established as @xmath0.01. ## discussion our results show that each method of enforcing terminology tested, which we have referred to in this paper as alignment, annotation, and factors, is effective in promoting the use of the requested translation. in both languages the approaches outperformed the baseline in this regard. the approaches did well in a wide variety of test cases, even test cases that may strain credulity. the benefit of giving this sort of guidance to the model seems to be more significant for input content that is out-ofdomain for the training data, but this improvement in terminology use does little to mitigate the quality drop observed in such translation scenarios. the alignment method seemed to have a larger negative impact on translation quality, as measured by accuracy, fluency, and morphological agreement, but was also the most likely to have the correct term present in the sentence. additionally, our results show that the use of noisy source material for glossary creation is viable. some intervention may still be required to retain only good quality term pairs. it remains to be seen how well this glossary actually approximates the distribution of input terms in production. contrary to the fears of @xcite , using only exact matches in data preparation does not limit the model to simple copying behavior. however, a tendency to restructure the output sentence so as to properly use the exact term provided is noticeable. users of glossary features should be guided on how best to work with polysemous terms in nmt. none of the methods emerged as clearly superior, with different models performing better in different tasks and different language pairs. we believe that this suggests that each approach can be viable, but must be carefully adapted for the specific language pair and usage scenario. a solution combining the annotation or factors method with the alignment method may present a good option. in such a solution, input data would be prepared according to the requirements for the former method, and alignment-based insertion can be used as a fallback, when the model does not produce the expected term. the use of lemmatization in this fallback method may help reduce the incidence of false positives for cases where the model has used the term correctly but in a morphological form different to that of the glossary term. ## future work this research suggests multiple potential paths for future research. firstly, our assumption that historical terminology enforcement requests approximate the distribution at inference time calls for proper scrutiny. research comparing the effects of using different glossaries to prepare training data under controlled conditions can show if there is any significant downstream effect in the translation task. furthermore, there are many avenues of investigation stemming from the data preparation procedure. what is the appropriate ratio of samples with and without glossary enforcement signals in the dataset? what are the effects of lemmatization or fuzzy matching of glossary pairs in the dataset? what would be the effect of adding the glossary signal at the start or end of the sequence instead of at the location where the source term occurs? should there be a limit to how many times a particular term appears? the frequency distribution this is because the surviving family has always kept saying, starting from the day the death was declared. 戻って来た they have returned city officials, residents, and tourists dragged some of them, and they somehow returned to the offshore, but many of them said they had returned.