Lingua Custodia’s Participation at the WMT 2021 Machine Translation Using Terminologies Shared Task (2021)

Abstract: This paper describes Lingua Custodia’s submission to the WMT21 shared task on machine translation using terminologies. We consider three directions, namely English to French, Russian, and Chinese. We rely on a Transformer-based architecture as a building block, and we explore a method which introduces two main changes to the standard procedure to handle terminologies. The first one consists in augmenting the training data in such a way as to encourage the model to learn a copy behavior when it encounters terminology constraint terms. The second change is constraint token masking, whose purpose is to ease copy behavior learning and to improve model generalization. Empirical results show that our method satisfies most terminology constraints while maintaining high translation quality.

## introduction neural-based architectures have become standard for machine translation (mt), they are efficient and offer state-of-the-art performance in many scenarios @xcite . however, these models often trained on very large corpora turn out to be less adequate in domains that require very careful use of terminology. for instance, consider the following english sentence from a biomedical corpus "now for the fever you can take a tachipirina sweet" . the term "tachipirina sweet" refers to "paracétamol" in french. unfortunately, a generic english-french neural mt (nmt) model would translate the above sentence as: "maintenant pour la fièvre tu peux prendre un tachipirina bonbon", where the term "tachipirina sweet" is translated "tachipirina bonbon". the goal of the wmt21 shared task on machine translation using terminology constraints is to explore methods that can take into account terminology constraints, in order to improve mt models' accuracy and consistency on specific domains. in the literature there are two main families of methods to take into account specific terminologies. one family incorporates terminology constraints at inference @xcite . members of this category can guarantee strict enforcement of constraints, however this often comes at the cost of higher decoding time and decreased accuracy @xcite . the other family of method integrates terminologies at training time @xcite , and they have the benefit of not changing the nmt model as well as of not incurring additional computational overheads at inference time @xcite @xcite . we participate in the following three directions: english to french, russian, and chinese, and the system we submit falls into the second family of method incorporating terminologies at training time. more precisely, we explore a variant of the models proposed in @xcite , which we train for each language pair. following this work, we first annotate our training data with the constraints using tags to distinguish constraints terms from other tokens in the sentences. second, we further perform constraint-token masking, which improves model robustness/generalization as supported by our experiments. the rest of the paper is organized as follows: section 2 reviews the details of our system, section 3 describes the training data selection, the development and test sets, as well as the terminologies used for each language pair, and section 4 presents the different experimental settings and results. ## method our objective is to encourage neural machine translation to satisfy lexical constraints. to this end, we rely on the approch proposed in @xcite , which introduces two changes to the standard procedure, namely training data augmentation and token masking. in the following we describe these two operations, which are also depicted in figures 1 and 2 . training data augmentation (tada). the purpose of this step is to encourage the nmt model to exhibit a copy behavior when it encounters constraint terms whose translation should be consistent with some terminology. this step, illustrated in figures 1 and 2, consists in using tags to annotate our training data with the terminology constraints, i.e., indicate the constraints (if any) in a given source sentence. note that in the literature, there are other variants that use additional information such as source factors @xcite . we do not use such information, and we specify terminologies using tags only. ## data this section provides information and some statistics regarding the datasets for the three language pairs we consider. training data selection. we consider three language pairs, namely english to french, russian, and chinese. since our method acts at training time, we first perform a training data selection in order to obtain a reasonable number of sentences containing at least one term from the provided terminologies. to do so, we consider both bilingual and monolingual data, provided as part of the shared task. in fact, we observe that bilingual data do not contain many sentences with terminology terms. thus, we rely on back-translation of monolingual data, which contains more recent news on covid-19, to obtain more sentence pairs with terminologies. we rely on opusmt foot_0 to back translate the russian monolingual to english. for chinese and french we use in-house translation engines. note that we further convert the chinese data into simplified chinese using opencc. following previous work on terminology control @xcite , only 10% of the training sentences are annotated in order to maintain the model's performance in terminology free cases. the details about training data selection for the different language pairs are summarized in tables 1, 2 and 3. development and test sets. for all language pairs, a development and test sets are provided. note that for the test sets we have access to the source part only. for the dev sets, the terminology constraints associated with each sentence are available, for the test sets this information is not available, and we leverage the terminology files to find constraint terms in these sets. just like the training data, both test and dev sets are augmented with the terminology constraints as presented in figures 1 and 2. the dev/test sets of the different language pairs share the same english source file containing 971/2100 sentences respectively. terminologies. for each language pair, we use the provided terminologies to annotate our train, dev and test sets. the terminologies consist of respectively 670, 925 and 710 unique source-target terms for @xmath0, russian and chinese. we also observe that one source term might be associated with one or more target terms. in that ## conclusion in this paper, we describe our submission to the wmt21 shared task on machine translation using terminologies. we participate in three language pairs, namely @xmath2, russian and chinese. our system integrates terminology constraints during training by augmenting the data with terminological terms. due to the lack of parallel training data containing the terminology terms, we rely on monolingual data for all language pairs to augment the number of sentences containing terminology terms. empirical results comparing our approach with terminology grounded as well as terminology free baselines show the effectiveness