Improving Lexically Constrained Neural Machine Translation with Source-Conditioned Masked Span Prediction (2021)

Abstract: Accurate terminology translation is crucial for ensuring the practicality and reliability of neural machine translation (NMT) systems. To address this, lexically constrained NMT explores various methods to ensure pre-specified words and phrases appear in the translation output. However, in many cases, those methods are studied on general domain corpora, where the terms are mostly uni- and bi-grams (>98%). In this paper, we instead tackle a more challenging setup consisting of domain-specific corpora with much longer n-gram and highly specialized terms. Inspired by the recent success of masked span prediction models, we propose a simple and effective training strategy that achieves consistent improvements on both terminology and sentence-level translation for three domain-specific corpora in two language pairs.

## introduction despite its recent success in neural machine translation (nmt) @xcite @xcite , delivering correct terms in the translation output is still a vital component for high-quality translation. this concern becomes more salient in domain-specific scenarios, such as in legal documents, where generating correct and consistent terminology is key to ensuring the practicality and reliability of machine translation (mt) systems @xcite . to address this, lexically constrained nmt works have proposed various methods to preserve terminology in translations as lexical constraints with or without the help of a term dictionary at test time. in most lexically constrained nmt setups, datasets and terms used for training and evaluating the methods are extracted from wmt news corpora @xcite et al., 2020). since the terms, regardless of their source, can only be utilized as long as they exist in the corpus, the term coverage solely depends on the choice of the corpus. by analyzing the previous setups carefully, we discover that the terms found in wmt are mostly uni-or bi-grams (see figure 1 ) and highly colloquial (see in this paper, inspired by the recent masked span prediction models, which have demonstrated improved representation learning capability of contiguous words @xcite @xcite , we propose a simple yet effective training scheme to improve terminology translation in highly specialized domains. we specifically select two highly specialized domains (i.e., law and medicine) which contain domain-specific terminologies to address more challenging and realistic setups, in addition to applying it to both typologically similar and dissimilar pairs of languages (german-english (@xmath0. thanks to its sim-plicity, the proposed method is compatible with any autoregressive transformer-based model, including ones capable of utilizing term dictionaries at training or test time. in domain-specific setups where longer n-gram terms are pervasive, our method demonstrates improved performance over the standard maximum likelihood estimation (mle) approach in terms of terminology and sentence-level translation quality. our code and datasets are available at https://github.com/wns823/nmt_ssp . ## background lexically constrained nmt we could group lexically constrained nmt methods into two streams: hard and soft. the hard approaches aim to force all terminology constraints to appear in the generated output. the methods include replacing constraints @xcite , constrained decoding @xcite @xcite , and additional attention heads for external supervision @xcite . although those approaches are reliable and widely used in practice, they typically require a pre-specified term dictionary and an extra candidate selection module if there are multiple matching candidates for a single term (see caption in several soft methods address this problem without the help of a term dictionary, one of which is training on both constraint pseudo-labeled (with statistical mt) and unlabeled data @xcite . more recently, @xcite and @xcite proposed methods that do not assume any word alignment or dictionary supervision at training time to handle unseen terms at test time. for their flexibility, we choose them as our baselines. as discussed in section 1, most previous methods are trained and evaluated on general domain corpora. in this work, we instead tackle highly specialized domain-specific corpora such as law and medicine, where the terms are much longer and often rare. ## conclusion we propose a simple and effective training scheme for improving lexically constrained nmt by introducing the masked span prediction task on the decoder side. our method shows its effectiveness in terms of terminology and sentence-level translation over the standard mle training in highly specialized domains in two language pairs. as we publicly release our code and datasets, we hope that more people can join this area of research without much burden. in the future, we plan to further investigate applying our method to non-autoregressive methods. line 29 : distributor dup() first calculates the number of sentences and phrases to be distributed across train, valid, and test sets following the ratio in s, and then distributes sentences accordingly. line 30 : distributor uni() distributes unique sentences and phrases alternatively between train, valid, and test sets.