Understanding and Improving the Robustness of Terminology Constraints in Neural Machine Translation (2023)

Abstract: In this work, we study the robustness of two typical terminology translation methods: Placeholder (PH) and Code-Switch (CS), concerning (1) the number of constraints and (2) the target constraint length. We identify that existing terminology constraint test sets, such as IATE, Wiktionary, and TICO, are blind to this issue due to oversimplified constraint settings. To solve it, we create a new challenging test set of English-German, increasing the average constraint count per sentence from 1.1~1.7 to 6.1 and the length per target constraint from 1.1~1.2 words to 3.4 words. Then we find that PH and CS methods degrade as the number of constraints increases, but they have complementary strengths. Specifically, PH is better at retaining high constraint accuracy but lower translation quality as measured by BLEU and COMET scores. In contrast, CS has the opposite results. Based on these observations, we propose a simple but effective method combining the advantages of PH and CS. This approach involves training a model like PH to predict the term labels, and then during inference replacing those labels with target terminology text like CS, so that the subsequent generation is aware of the target term content. Extensive experimental results show that this approach can achieve high constraint accuracy and translation quality simultaneously, regardless of the number or length of constraints.

## introduction although neural machine translation (nmt) has achieved expressive performance improvement with the increase of model and data scale, it still struggles when involved in mismatched domains and rare entities @xcite . terminology constraints (tc) is a popular solution that requires the model to generate the translation following the pre-provided terminology pairs and has been widely applied in commercial translation systems, such as google, deepl, etc. perhaps the most popular approach for tc is learning the constraint-aware model through data augmentation @xcite @xcite . 2 early data augmentation is based on placeholder (ph). during training, ph methods replace the terminology terms in both source and target sentences with ordered labels (e.g., "t 1 ", "t 2 "), while the model predicts labels rather than the concrete terms at inference @xcite . the main drawback of ph methods is that the term labels lose the original semantic information, resulting in incoherent translation. unlike ph methods, code-switch (cs) methods follow the standard model and generate term translations word by word by injecting target constraints in the source sequence @xcite @xcite . in this work, we focus on understanding the robustness of existing terminology constraint methods in challenging constraint settings in practice. our contributions are four-fold: ## background let @xmath0, . . . , x m } be the source sentence, @xmath1, . . . , y n } be the target sentence, and @xmath2, t 1 ⟩, . . . , ⟨s k , t k ⟩} be the constraint set about x and y, where s i and t i are the i-th source and target constraint respectively. each constraint could be multi-word, e.g., then tc asks the system must translate s i into t i . in this section, we briefly introduce two typical tc methods based on data augmentation: placeholder (ph) @xcite and code-switch (cs) @xcite . we also describe some variants of them. figure 1 explains the differences between these methods. ## on the robustness of terminology constraint in this section, we explore the robustness of existing tc solutions from two aspects: (1) number of constraints and (2) target constraint length. we first point out the oversimplified problem in existing tc test sets in section 3.1. we describe our proposed test set #sent #term #avg term #avg word iate 414 452 1.1 1.2 wiktionary 727 884 1.2 1.1 tico 15676 26492 1.69 1.23 efa --<3 ∼ u(1, 3) † ours 500 3052 6.1 3.4 ## experimental results we first validate the effectiveness of proposed approach on the same setup as section 3.3. then, for fair comparisons to existing work, we also conduct experiments with wmt18 en-de training data (europarl, news commentary) and common test sets (iate, wiktionary). figure 3 : illustration of naive padding (a) and proposed dynamic padding (b) with a beam size of 2. red blocks represent term labels, respectively. the red, gray, and blue blocks represent the term label, pad, and normal token, respectively. for naive padding, the sequence length is 6 with two redundant pad symbols. in contrast, dynamic padding reduces the sequence length to 4 by moving pads at the beginning and then truncating. ## analysis 6.1 inference speed as illustrated in figure 5 , we compared the decoding step size and inference speed between our model and the vanilla transformer. we also study the effect when our model decodes with naive padding (np) and dynamic padding (dp). it is clear ## related work there have been several approaches to addressing the issue of translating specialized terminology in the field of machine translation. one branch of approaches focuses on the decoding process, such as extending the search space @xcite @xcite or using a finite-state acceptor @xcite , to enforce terminology translation strictly. however, data augmentation perhaps be the most widely used approach for terminology translation in machine translation. the placeholder method is an early solution for terminology translation by introducing special term labels @xcite . @xcite add linguistic information in the label to compensate for the semantic loss. although effective, placeholder techniques have difficulties producing smooth translations. recently, code-switch methods have become popular as it overcomes this problem by allowing the model to generate word-by-word constraint translation, like standard neural machine translation. @xcite directly replaces the source constraint with its translation in the input sequence; @xcite uses some tags to distinguish be-tween source constraints and target constraints; @xcite further improves performance by masking the source constraints; @xcite uses target lemma to make the model learn morphology knowledge. as observed in our experiments, code-switch methods are fluent in translation but degrade in constraint accuracy. in contrast, our approach attempts to combine the strengths of placeholder and code-switch, achieving high translation quality and constraint accuracy simultaneously. ## conclusion our study has highlighted the importance of taking robustness into account when comparing different methods of terminology constraint translation. we have found that the placeholder and code-switch families are superior in different metrics, and the gap between them increases when dealing with more and longer terms. additionally, we have observed that current tc test sets are inadequate for testing the robustness of different methods. to address this problem, we have created a new, more difficult terminology constraint test set. moreover, we have proposed the rtt model, which merges the best features of the placeholder and code-switch approaches and is capable of delivering both high translation quality and constraint accuracy regardless of the number of constraints and their length.