Morphology Aware Source Term Masking for Terminology-Constrained NMT (2024)

Abstract: Terminology-constrained NMT systems facilitate the forced translation of domain-specific vocabulary. A notable method in this context is the “copy-and-inflect” approach, which appends the target term lemmas of constraints to their corresponding source terms in the input sentence. In this work, we propose a novel adaptation of the “copy-and-inflect” method, referred to as “morph-masking”. Our method involves masking the source terms of the constraints from the input sentence while retaining essential grammatical information. Our approach is based on the hypothesis that “copy-and-inflect” systems have access to both source and target terms, allowing them to generate the correct surface form of the constraint by either translating the source term itself or properly inflecting the target term lemma. Through extensive validation of our method in two translation directions with different levels of source morphological complexity, Basque to Spanish and English to German, we have demonstrated that “morph-masking” is capable of providing a harder constraint signal, resulting in a notable improvement over the “copy-and-inflect” method (up to 38% in term accuracy), especially in challenging constraint scenarios.

## introduction while neural machine translation (nmt) achieves high quality results in general-purpose translation scenarios, it frequently encounters challenges with precise technical terminology in specialized domains, as noted by @xcite . to address this limitation, terminology-constrained nmt facilitates the forced translation of specific terminology, ensuring consistent and reliable translation of domain-specific vocabulary, thus considerably reducing post-editing efforts. recent research in terminology-constrained nmt predominantly adopts a data-driven approach. this method involves teaching systems to apply terminology constraints through training with synthetic, task-specific data @xcite @xcite . specifically, @xcite introduced a copy-and-inflect method. this method appends the lemmas of constraints' target terms to their corresponding source terms within the input sentence. the system is then trained to produce translations by appropriately copying and inflecting these target terms based on the context (see annotation example in however, available evidence suggests that copyand-inflect methods do not consistently enforce terminology constraints @xcite . our hypothesis is that these methods, having access to both the source and target terms of the constraints, only provide a soft constraint. in other words, they might generate the correct surface form of the constraint either by translating the source term directly or by properly inflecting the lemma of the target term. given this hypothesis, we introduce a novel variation of the copy-and-inflect method designed to provide a stronger constraint signal to the system. specifically, we propose to mask the source terms of constraints in the input sentence while retaining the crucial grammatical information, such as as gender, number, grammatical cases, etc.. we contend that maintaining this information is vital, especially for morphologically rich languages like basque, to prevent any degradation in translation quality due to a loss of grammatical context after masking. while much of the previous research examining the effects of masking source terms has focused on english as the source language @xcite @xcite , we evaluate our approach on two translation directions, each with varying degrees of source morphological complexity: english to german and basque to spanish. these language pairs were selected to encompass a wide variety of linguistic features and complexities. spanish and basque, belonging to different language families, display significant differences in morphology and syntax. although english and german are both germanic languages and share some similarities, german has a much more complex morphology. consistent with previous research by @xcite , we translate to morphologically rich languages to assess the inflection capabilities of the systems. to the best of our knowledge, the basque to spanish translation direction has not been previously explored. consequently, we have manually created a challenging test set foot_0 for this translation direction, which we anticipate will be a valuable resource for subsequent research. ## related work works addressing terminology-constrained nmt mainly fall into two different categories: a) constrained decoding-based approaches and b) datadriven approaches. constrained decoding approaches modify the decoding algorithm to force the model to apply terminology constraints when predicting the next token @xcite @xcite . while constrained decoding ensures the presence of the required terminology, it can significantly slow down the decoding process @xcite and strict enforcement of the constraints can result in lower quality translations @xcite . data-driven approaches train systems with synthetic task-specific data to learn how to apply terminology constraints @xcite @xcite . the main advantage of this approach is that it does not require any changes in the model architecture nor in the decoding algorithm. there is no inference time overhead either. as a result, recent efforts have concentrated on methodologies employing various data generation strategies for this task. for instance, @xcite proposed a copy-and-inflect method which appends constraint's target terms lemmas to their corresponding source terms in the input sentence. with additional source factors @xcite they indicate whether the words in the input sequence belong to the source term of the con-straint, to the target term or the word is not part of the constraint. then, the system is trained to generate translations by properly copying and inflecting those target terms depending on the context. the method is based in the original copy method proposed by @xcite but they use lemmas instead of the final form of the terms. this is specially important when translating to morphologically rich languages where each word has several surface forms depending on the context. related to our masking approach, both @xcite and @xcite ) explore what they refer to it as the replace setting, in which the source term is entirely masked. while @xcite report findings similar to the append setting, @xcite find that the replace method underperforms. notably, both studies evaluate the replace setting using english as the source language, a language that has fewer surface forms per word compared to morphologically rich languages, such as basque. we introduce a novel adaptation of the copy-andinflect method (bergmanis and pinnis, 2021) which we call 'morphology aware source term masking', hereinafter referred to as, morph-masking. this approach involves masking the source term of the constraints within the input sentence, aiming to deliver a more robust constraint signal to the system. before masking, grammatical information -such as gender, number, and grammatical casesis extracted from the masked term. we argue that this information is crucial, especially for languages with complex morphology like basque, to prevent losing grammatical details after masking that could adversely impact the overall translation quality. the target term lemma and the tokens representing the extracted grammatical information are then inserted in place of the masked source term. as in bergmanis and pinnis (2021), we distinguish constraints from the original source sentence words using additional source factors @xcite . we employ bio tags -abbreviations for "beginning, inside, and outside" tags-which are frequently utilized in named entity recognition (ner) tasks, to annotate target terms. these tags are instrumental in structuring and labeling constraints, especially for multi-word terms. additionally, we use an extra information figure 1 : illustration of the proposed annotation method morph-masking. constraints' source terms in the input sentence are masked and replaced with the target terms while preserving the necessary grammatical information in the source terms such as as the gender, the number, grammatical cases, etc. we differentiate constraints from the original source sentence words using additional source factors @xcite . english translation: the sternoclavicular joint connects the sternum and the clavicle. we annotate a constraint only when the source term appears in the source sentence and the target term is present in the reference sentence. to identify annotation candidates, both source and reference sentences, as well as dictionary entries, are first lemmatized. this lemmatization step is essential to find words in morphologically complex languages such as basque. target terms are annotated in their dictionary form, that is, lemmatized. our annotations are limited to common nouns, proper nouns, and adjectives. to compare our morph-masking method to the copy-and-inflect method proposed by @xcite , we follow their annotation guidelines to generate the training data. that is, constraints' target terms lemmas are appended to the source terms in the input sentence. in this case, additional source factors are used to differentiate between source terms (1), target terms (2) and other words (0). ## experimentation all the systems were trained using the default configuration for the transformer architecture @xcite as implemented in the py-torch version of the opennmt toolkit @xcite . we apply bpe tokenization @xcite learned on 32,000 merge operations on the joint training parallel data. sentences larger than 100 subwords after tokenization are discarded from the training set. first, general purpose nmt systems were trained to be used as the baselines. the basque-spanish baseline was trained on the basque-spanish portion of the paracrawl corpus (v9) @xcite . data was splitted into train, validation and test sets with 3.3m/5k/5k parallel sentences respectively. the total vocabulary size after applying bpe tokenization was 42k for basque and 36k for spanish. similarly, the english-german baseline was trained on the english-german portion of the paracrawl corpus (v9). in this case, training, validation and test sets consist of 278m/5k/5k parallel sentences respectively. a vocabulary size of 58k tokens is used for both english and german. we followed an annotation method designed for easy extension across a broad spectrum of language pairs. to achieve this, we decided to leverage the apertium toolkit @xcite , an open-source rule-based machine translation toolkit that already covers many language pairs. this toolkit provides essential tools for lemmatization and morphological analysis, both crucial for our annotation process. additionally, apertium offers bilingual dictionaries, which we employ as constraints. although many of the dictionary entries can potentially be commonly used words, we argue that the system must learn how to apply terminology constraints rather than learning the annotated words themselves. apertium's basque-spanish and english-german bilingual dictionaries were used for the annotation step. these dictionaries were divided into train and test set, with the test set comprising 10% of the entries. for the basque-spanish language pair we annotate the entire training parallel data following the annotation procedure described in section 3. segment pairs lacking annotations -samples for which no constraint was found-were discarded. for the english-german translation direction, we limited our annotation to 10m sentences from the training data, also skipping samples without annotations. annotating the full training parallel data, 278m segments, in this case is an expensive task and there should be enough annotated training samples to learn the task. we generate samples with different number of constraints. specially, 50% of the samples have a single constraint while the remaining samples are annotated with 2 to 5 constraints randomly sampled. source factors are appropriately transposed from word-level to bpe token level. unlike prior works @xcite , the terminology-constrained systems were trained by fine-tuning the baseline system on the annotated data sets. this avoids training the system from scratch which means already existing strong baselines can be adapted to handle terminology constraints. to avoid catastrophic forgetting, as systems must perform equally well on terminology constrained and unconstrained data, we follow a mixed fine-tuning strategy @xcite . a weighted combination (2:1 ratio foot_1 ) of unconstrained and constrained data is used during training and validation steps. for validation purposes, we concatenate with a 1:1 ratio. the baseline and the fine-tuned systems were trained until they converged based on perplexity results from the validation set, using an early stopping criterion of 5 consecutive checkpoints. validation is performed every 10,000 steps in the case of the baseline system whereas fine-tuning validation is performed every 1,000 steps. all the systems were trained on a single rtx 2080-ti gpu device. we evaluate our systems using bleu and chrf++ scores provided by the sacrebleu tool @xcite . additionally, we report comet @xcite ) scores foot_2 , a metric which focuses on the semantic similarity by leveraging the recent breakthroughs in neural language modeling. while bleu, chrf++ and comet metrics measure the overall translation quality of the systems, task specific metrics are required. to address this, we determine the accuracy of the correctly translated constraints in terms of term-level constraint accuracy (tca) as in @xcite . we posit that terminology-constrained nmt becomes useful in cases where the baseline model fails to produce the correct target term of the constraints. the ideal test set should contain more complex and specialized terminology constraints that align with real-life requirements. basque-spanish test sets. to the best of our knowledge, the basque to spanish translation direction has not been previously addressed. as a result, we curated two high-quality and challenging test sets for this translation direction. these sets were meticulously crafted to emulate real-life applications of terminology-constrained nmt. in the following lines we describe the handcrafted test sets and euskalterm. the aim of this test set is to prioritize the incorporation of specific terminology constraints, focusing on the terms rather than on the parallel sentences. initially, a collection of 300 terms was curated from the publicly accessible euskalterm term database foot_3 . the euskalterm database contains specialized terminology for a diverse set of domains. terms with a varying number of words were chosen, ranging form one to five words. instead of relying on parallel corpora to find these terms, we asked a native speaker to craft up to two spanish sentences for each term. this approach was taken to ensure that the corresponding basque translations of the terms include a wide variety of complex suffix patterns. subsequently, these sentences were meticulously translated into basque, ensuring the inclusion of the constraints. euskalterm multi. in a similar fashion to zhang et al. (2023) we designed a test to measure the influence of varying constraint counts within a sentence. for this purpose, we utilized the elhuyar parallel corpus publicly available at opus (tiedemann, 2012). we carefully removed samples already present in the training data and selected a set of 50 parallel sentences. then, a linguistic expert manually selected 4 terms from each of the extracted parallel sentences. these terms comprised noun phrases and proper names of varying word lengths. furthermore, as in @xcite , we automatically annotated the test portion of the paracrawl data set using the test subset of the bilingual dictionary extracted from apertium (referred to as paracrawl). as mentioned earlier, this test set does not mimic a challenging real-life scenario. instead it is used for comparison purposes against the more complex and challenging euskalterm test set. english-german test sets. for the english to german translation direction we utilized two publicly available test sets: automotive test suite test set introduced in bergmanis and pinnis (2021) and iate from @xcite . the automotive test suite test set consists of parallel sentences in english, estonian, german, latvian, and lithuanian, with terminology constraints derived from a glossary constructed by professional translators. the iate test set was created by automatically annotating iate terms in the out-domain wmt newstest 2017 test set. consequently, many common nouns, such as sport, bridge, trip, are annotated. we note that some of them appear multiple times. additionally, terms are annotated in their surface form which means their final form is known beforehand. therefore, this test set is only used for comparison purposes with prior work. ## results this section presents the results of our experimental work, emphasizing a comparative analysis between our proposed morph-masking method and the copy-and-inflect method @xcite . we evaluate the performance of the terminology-constrained fine-tuned systems for the basque to spanish and english to german translation directions, aiming for comprehensive insights and conclusions 5 . ## conclusions in this work, we tackle terminology-constrained nmt using a data-driven approach that does not require changes in the system architecture or decoding algorithm. in particular, we introduce a novel variation of the copy-and-inflect method introduced by @xcite . our proposed method aims to provide a stronger constraint signal by masking the source terms of the constraints in the input sentence, while retaining essential grammatical information from the source terms, such as gender, number, grammatical cases, and so forth. by evaluating our approach on two translation directions -basque to spanish and english to german, each having varying degrees of source morphological complexity-we demonstrate that our morph-masking method offers a harder constraint signal. this leads to performance improvements over the copy-and-inflect method in all scenarios. removing source terms not only maintains the performance but also compels the model to utilize the provided target term in the output translations. this confirms our hypothesis that the copy-and-inflect method can sometimes allow the system to disregard the given target term, instead defaulting to its standard translation for the source term. through an ablation study, we further highlight the importance of preserving essential grammatical information, especially for morphologically rich languages like basque, to achieve superior translation quality and term accuracy. additionally, we show that fine-tuning a general purpose nmt system with synthetically generated data for the terminology-constrained nmt task is sufficient for the system to learn how to apply terminological constraints.