Neural Machine Translation with Monolingual Translation Memory (2021)

Abstract: Prior work has proved that Translation Memory (TM) can boost the performance of Neural Machine Translation (NMT). In contrast to existing work that uses bilingual corpus as TM and employs source-side similarity search for memory retrieval, we propose a new framework that uses monolingual memory and performs learnable memory retrieval in a cross-lingual manner. Our framework has unique advantages. First, the cross-lingual memory retriever allows abundant monolingual data to be TM. Second, the memory retriever and NMT model can be jointly optimized for the ultimate translation goal. Experiments show that the proposed method obtains substantial improvements. Remarkably, it even outperforms strong TM-augmented NMT baselines using bilingual TM. Owning to the ability to leverage monolingual data, our model also demonstrates effectiveness in low-resource and domain adaptation scenarios.

## introduction augmenting parametric neural network models with non-parametric memory @xcite @xcite has recently emerged as a promising direction to relieve the demand for ever-larger model size @xcite @xcite . for the task of machine translation (mt), inspired by the computer-aided translation (cat) tools by professional human translators for increasing productivity for decades @xcite , the usefulness of translation memory (tm) has long been recognized @xcite . in general, tm is a database that stores pairs of source text and its corresponding translations. like for human translation, early work @xcite @xcite presents translations for similar source input to statistical translation models as additional cues. recent work has confirmed that tm can help neural machine translation (nmt) models as well. in a similar spirit to prior work, tm-augmented nmt models do not discard the training corpus after training but keep exploiting it in the test time. these models perform translation in two stages: in the retrieval stage, a retriever searches for nearest neighbors (i.e., source-target pairs) from the training corpus based on source-side similarity such as lexical overlaps @xcite @xcite , embedding-based matches @xcite , or a hybrid @xcite ; in the generation stage, the retrieved translations are injected into a standard nmt model by attending over them with sophisticated memory networks @xcite @xcite or directly concatenating them to the source input @xcite , or biasing the word distribution during decoding @xcite . most recently, @xcite propose a token-level nearest neighbor search using complete translation context, i.e., both the source-side input and target-side prefix. despite their differences, we identify two major limitations in previous research. first, the translation memory has to be a bilingual corpus consisting of aligned source-target pairs. this requirement limits the memory bank to bilingual pairs and precludes the use of abundant monolingual data, which can be especially helpful for low-resource scenarios. second, the memory retriever is non-learnable, not end-to-end optimized, and lacks for the ability to adapt to specific downstream nmt models. concretely, current retrieval mechanisms (e.g., bm25) are generic similarity search, adopting a simple heuristic. that is, the more a source sentence overlaps with the input sentence, the more likely its target-side translation pieces will appear in the correct translation. although this observation is true, the most similar one does not necessarily serve the best for nmt models. ideally, the retrieval metric would be learned from the data in a task-dependent way: we wish to consider a memory only if it can indeed boost the quality of final translation. in this work, we propose to augment nmt models with monolingual tm and a learnable crosslingual memory retriever. specifically, we align source-side sentences and the corresponding targetside translations in a latent vector space using a simple dual-encoder framework @xcite , such that the distance in the latent space yields a score function for retrieval. as a result, our memory retriever directly connects the dots between the source-side input and target-side translations, enabling monolingual data in the target language to be used alone as tm. before running each translation, the memory retriever selects the highest-scored memories from a large collection of monolingual sentences (tm), which may include but are not limited to the target side of training corpus, and then the downstream nmt model attends over those memories to help inform its translation. we design the memory retriever with differentiable neural networks. to unify the memory retriever and its downstream nmt model into a learnable whole, the retrieval scores are used to bias the attention scores to the most useful retrieved memories. in this way, our memory retrieval can be end-to-end optimized for the translation objective: a retrieval that improves the golden translation's likelihood is helpful and should be rewarded, while an uninformative retrieval should be penalized. one challenge for training our proposed framework is that, when starting from random initialization, the retrieved memories will likely be totally unrelated to the input. since the memory retriever does not exert positive influence on nmt model's performance, it cannot receive a meaningful gradient and improve. this causes the nmt model to learn to ignore all retrieved memories. to avoid this cold-start problem, we propose to warm-start the retrieval model using two cross-alignment tasks. experiments show that (1) our model leads to significant improvements over non-tm baseline nmt model, even outperforming strong tm-augmented baselines. this is remarkable given that previous tm-augmented models rely on bilingual tm while our model only exploits the target side. tm-augmented nmt this work contributes primarily to the research line of translation memory (tm) augmented neural machine translation (nmt). feng et al. (2017) augmented nmt with a bilingual dictionary to tackle infrequent word translation. gu et al. (2018) proposed a model that retrieves examples similar to the test source sentence and encodes retrieved source-target pairs with keyvalue memory networks. cao and xiong (2018); cao et al. (2019) used a gating mechanism to balance the impact of the translation memory. zhang et al. (2018) proposed guiding models by retrieving n-grams and up-weighting the probabilities of retrieved n-grams. bulte and tezcan (2019) and xu et al. (2020) used fuzzy-matching with translation memories and augment source sequences with retrieved source-target pairs. xia et al. (2019) directly ignored the source side of a tm and packed the target side into a compact graph. khandelwal et al. (2020) ran existing translation model on large bi-text corpora and recorded all hidden states for later nearest neighbor search at each decoding step, which is very compute-intensive. the distinctions between our work and prior work are obvious: (1) the tm in our framework is a collection of monolingual sentences rather than bilingual sentence pairs; retrieval for text generation discrete retrieval as an intermediate step has been shown beneficial to a variety of natural language processing tasks. one typical use is to retrieve supporting evidence for open-domain question answering (e.g., @xcite @xcite . recently, retrieval-guided generation has gained increasing interest in a wide range of text generation tasks such as language modeling @xcite @xcite , dialogue response generation @xcite @xcite ., source sentence encoder 𝐸 "#$ (𝑥) target sentence encoder 𝐸 ()( (𝑧) translation memory 𝒵 … dense index mips source encoder memory encoder 𝓏 -,/ /01 2 3 decoder retrieval model output y translation model bias attention input x 𝓏 1 𝓏 5 … 𝑓(𝑥, 𝓏 5 ) 𝑓(𝑥, 𝓏 1 ) … relevant tm relevance scores figure 1: overall framework. for an input sentence x in the source language, the retrieval model uses maximum inner product search (mips) to find the top-m tm sentences {z i } m @xmath0. the translation model takes {z i } m @xmath1, z i )} m @xmath2. nmt using monolingual data to our knowledge, the integration of monolingual data for nmt was first investigated by @xcite , who separately trained target-side language models using monolingual data, and then integrated them during decoding either through re-scoring the beam, or by feeding the hidden state of the language model to the nmt model. @xcite also explored re-ranking the nmt output with a n-gram language model. another successful method for leveraging monolingual data in nmt is back-translation @xcite @xcite , where a reverse translation model is used to translate monolingual sentences from the target language to the source language to generate synthetic parallel sentences. recent studies @xcite showed that self-training, where the synthetic parallel sentences are created by translating monolingual sentences in the source language, is also helpful. our method is orthogonal to previous work and bears a unique feature: it can use more monolingual data without re-training (see §4.3). ## proposed approach we start by formalizing the translation task as a retrieve-then-generate process in §3.1. then in §3.2, we describe the model design for the cross-lingual memory retrieval model. in §3.3, we describe the model design for the memory-augmented translation model. lastly, we show how to optimize the two components jointly using standard maximum likelihood training in §3.4 and therein we address the cold-start problem via cross-alignment pre-training. ## experiments we experiment with the proposed approach in three settings: (1) the conventional setting where the available tm is limited to the bilingual training corpus, (2) the low-resource setting where bilingual training pairs are scarce but extra monolingual data is exploited as additional tm, and (3) nonparametric domain adaptation using monolingual tm. note that existing tm-augmented nmt models are only applicable to the first setting, the last two settings only become possible with our proposed model. we use bleu score @xcite as the evaluation metric. ## conclusion we introduced an effective approach that augments nmt models with monolingual tm. we show that a task-specific cross-lingual memory retriever can be learned by end-to-end mt training. our approach achieves new state-of-the-art results on sev-