Fast and Accurate Neural Machine Translation with Translation Memory (2021)

Abstract: It is generally believed that a translation memory (TM) should be beneficial for machine translation tasks. Unfortunately, existing wisdom demonstrates the superiority of TM-based neural machine translation (NMT) only on the TM-specialized translation tasks rather than general tasks, with a non-negligible computational overhead. In this paper, we propose a fast and accurate approach to TM-based NMT within the Transformer framework: the model architecture is simple and employs a single bilingual sentence as its TM, leading to efficient training and inference; and its parameters are effectively optimized through a novel training criterion. Extensive experiments on six TM-specialized tasks show that the proposed approach substantially surpasses several strong baselines that use multiple TMs, in terms of BLEU and running time. In particular, the proposed approach also advances the strong baselines on two general tasks (WMT news Zh->En and En->De).

## introduction a translation memory (tm) is originally collected from the translation history of professional translators, and provides the most similar source-target sentence pairs for the source sentence to be translated @xcite @xcite @xcite . a tm generally provides valuable translation information particularly for those input sentences preferably matching the source sentences in the tm, and many efforts have been devoted to integrating a tm into statistical machine translation @xcite @xcite @xcite . recently there are increasing interests in improving neural machine translation (nmt) with a * corresponding author. tm @xcite @xcite @xcite . many notable approaches have been proposed to augment an nmt model by using a tm. for example, @xcite and @xcite extract scored n-grams from a tm and then reward each partial translation once it matches an extracted n-gram during beam search. @xcite and @xcite use an auxiliary network to encode a tm and then integrate it into the nmt architecture. @xcite and @xcite employ data augmentation to train an nmt model whose training instances are bilingual sentences augmented by their tms. despite their improvements on the tm-specialized translation tasks (aka jrc-acquis corpora) where a tm is very similar to test sentences, they consume considerable computational overheads in either training or testing, and particularly it is unclear whether they can deliver gains over standard nmt on general tasks where a tm is not very similar to test sentences. indeed, both @xcite and @xcite reported their failures on wmt news translation tasks. in this paper, we present a fast and accurate approach for tm-based nmt which can be applied to general translation tasks besides tm-specialized tasks. we first design a light-weight tm-based nmt model for efficiency: its tm includes a single bilingual sentence and we explore variant ways to encode the tm. also, the designed model outperforms strong tm-based baselines. second, we deeply analyze its translation performance and observe an issue of robustness: it decreases significantly for those input sentences which are not very similar to their tms, although it obtains substantial improvements for other inputs. to address this issue, we propose a novel training criterion for optimizing the parameters of our model inspired by multiple-task learning @xcite @xcite . the loss function includes two terms: the first term is induced by the bilingual corpus with a tm whereas the second term is induced by the bilingual corpus without any tm. in this way, the tm-based nmt model gains better performance and is robust to translate any input sentences no matter they are similar to their tm or not. additionally, this makes it possible that a single unified model can handle both translation situations (with or without a tm), which is practical for online services. to validate the effectiveness of the proposed approach, we conduct extensive experiments on eight translation tasks including both tm-specialized tasks and general tasks (wmt). our experiments justify that the proposed approach is better than several strong tm-based baselines in speed, and it further delivers substantial gains (up to 4.7 blue points) over those baselines on tm-specialized tasks, leading to up to 8.5 bleu points over standard transformer-based nmt. in particular, it also outperforms strong baselines on two general translation tasks, i.e., with a gain of 0.7 bleu points on wmt14 @xmath0.0 bleu point on wmt17 @xmath1. this paper makes the following contributions: recently, self-attention networks have attracted many interests due to their flexibility in parallel computation and modeling h d,l i . the state-of-theart nmt model is transformer @xcite , which uses stacked self-attention and fully connected layers for its encoder and decoder. selfattention relies on an attention mechanism to compute a representation of a sequence. in transformer, there are three kinds of attention mechanisms, including encoder multi-head attention, decoder masked multi-head attention and encoderdecoder multi-head attention. attention with h heads can be calculated by the equations: ## model architecture in this section, in order to preferably bridge tm and nmt, we propose the architecture of tm-based nmt within the transformer. to make our proposed model fast in running time and powerful in quality, at first, we present a configuration of tm to make the proposed model efficient. then we explore three different methods to encode the tm into a sequence of vectors in a coarse-to-fine manner. finally, we propose the architecture that decodes a target word given an input source sentence and its tm representation. ## training suppose the training corpus is @xmath2, y i , x i tm , y i tm | @xmath3, n ]}, where x i , y i is a bilingual sentence, and x i tm , y i tm is the related tm which consists of a single bilingual sentence. our goal is to learn the parameter θ of the tm-based nmt model p (y | x, x tm , y tm ; θ) defined in eq.( 9 ) using d. the common wisdom is to optimize the parameter under the maximum likelihood estimation (mle), i.e. standard training. formally, it minimizes the following criterion: robustness issue unfortunately, the model trained with mle suffers from an issue about robustness even if its overall performance is much better than standard transformer and outperforms tm-based baselines on the @xmath4. according to our experiments (see as a result, it would be dangerous to use the model for online services because users may provide an input sentence whose tm is not similar to itself. the possible reason for the above issue is explained as follows. on the average case, the reference y is strongly correlated to its tm target y tm in the training corpus d. for example, the average similarity score is about 0.58 for @xmath5, according to our statistics. because of the powerful fitting ability of neural networks, the model parameters will be guided to heavily depend on the given tm target y tm during training. in this way, if an input source sentence x has a high similarity with its given tm, the model will output high-quality results, as we also observed in training criterion in order to avoid the tm over-fitting, we propose a simple yet elegant method, inspired by data augmentation @xcite @xcite and multiple-task learning @xcite @xcite . specifically, we first construct another corpus in the constructed corpus, null, null plays a role of a tm, but both source and target sides of the tm are empty sentences. 4 then we train the model p (y | x, x tm , y tm ; θ) using both d and d 0 , i.e. joint training, which is similar to multiple-task learning. formally, we minimize the following joint loss function: note that the proposed method is slightly different from standard data augmentation @xcite @xcite and multiple-task learning @xcite @xcite in nmt research. these data augmentation techniques automatically generate pseudo data based on the original training data and then train a model using both original and generated data. however, the dataset d 0 is ## experiments in this section, we validate the effectiveness of the proposed approach: robustness for handling both translation situations (with or without a tm), running efficiency compared with the previous tmbased nmt models, translation quality on both tm-specialized tasks and general mt tasks. we use the case-insensitive bleu score as the automatic metric @xcite for the translation quality evaluation. ## related work in the statistical machine translation (smt) diagram, @xcite extract bilingual segments from a tm which matches the source sentence to be translated, and employ a heuristic score to decide whether the extracted segments should be used as decoding constraints or not, then hardly constrain smt to decode for those unmatched parts of the source sentence. @xcite design a fine-grained classifier, rather than the heuristic score, to predict the score for making more reliable decisions. @xcite , @xcite and @xcite add the extracted bilingual segments to the translation table of smt, and then bias the decoder in a soft constraint manner when decoding the source sentence with the augmented translation table. @xcite use the retrieved bilingual sentences to update the parameters for the log-linear model based smt. in recent years, many efforts are made on neural machine translation (nmt) associated with a tm. @xcite and @xcite make full use of the retrieved tm sentence pairs to fine-tune the pre-trained nmt model on-the-fly. the most obvious drawback of fine-tuning is that the delay is too long for testing sentences. to avoid the online tuning process, @xcite and @xcite dynamically integrate translation pieces, based on n-grams extracted from the matched segments in the tm target, into the beam search stage. the second type of approach is efficient but heavily depends on the global hyper-parameter λ, which is sensitive to the development set, leading to inferior performance. recently, there are notable approaches for the sake of further excavation on tm-based nmt. @xcite and @xcite propose data augmentation approaches by augmenting input sentences with a tm which do not modify the nmt model architecture. @xcite and @xcite employ an auxiliary network to encode tms and integrate it into the nmt architecture. our model architecture is simpler than @xcite and @xcite and we encode a single tm target sentence and utilize simple attention mechanisms on the tm. and the architecture is more efficient and leads to a faster translation speed compared with @xcite and @xcite . in particular, we propose a novel training criterion to make the tm-based nmt model more robust in different translation situations (with or without a tm). in parallel with our work, @xcite extend the translation memory from the bilingual setting to the monolingual setting through a cross-lingual retrieval technique, and @xcite report significant improvements in quality on general translation tasks as ours, but their inference speed is two orders of magnitude slower than transformer because they perform contextual word retrieval whose search space is much larger than that of sentence retrieval. ## conclusion this paper presents a simple tm-based nmt model that employs a single bilingual sentence as its tm and thus is fast in training and inference. although the presented model with the standard training outperforms strong tm-based baselines, it suffers from a robustness issue: its performance highly depends on the similarity of a tm. to address this issue, we propose a novel training criterion inspired by multiple-task learning and data augmentation. experiments on tm-specialized tasks demonstrate its superiority over strong baselines in terms of running time and bleu. also, it is shown that a tm-based nmt model can advance the strong transformer on general translation tasks like wmt.