Neural Machine Translation with Contrastive Translation Memories (2022)

Abstract: Retrieval-augmented Neural Machine Translation models have been successful in many translation scenarios. Different from previous works that make use of mutually similar but redundant translation memories (TMs), we propose a new retrieval-augmented NMT to model contrastively retrieved translation memories that are holistically similar to the source sentence while individually contrastive to each other providing maximal information gain in three phases. First, in TM retrieval phase, we adopt contrastive retrieval algorithm to avoid redundancy and uninformativeness of similar translation pieces. Second, in memory encoding stage, given a set of TMs we propose a novel Hierarchical Group Attention module to gather both local context of each TM and global context of the whole TM set. Finally, in training phase, a Multi-TM contrastive learning objective is introduced to learn salient feature of each TM with respect to target sentence. Experimental results show that our framework obtains substantial improvements over strong baselines in the benchmark dataset.

## introduction translation memory (tm) is basically a database of segmented and paired source and target texts that translators can access in order to re-use previous translations while translating new texts @xcite . for human translators, such similar translation pieces can lead to higher productivity and consistency @xcite . for machine translation, early works mainly contributes to employ tm for statistical machine translation (smt) systems @xcite @xcite . recently, as neural machine translation (nmt) model @xcite has achieved impressive performance in many translation tasks, there is also an emerging interest @xcite in retrieval-augmented nmt model. the key idea of retrieval-augmented nmt mainly includes two steps: a retrieval metric is used to retrieve similar translation pairs (i.e., tm), and the tm is then integrated into an nmt model. in the first step, a standard retrieval method greedily chooses the most similar translation memory one by one solely based on similarity with the source sentence (namely greedy retrieval). this method would inevitably retrieve translation memories that are mutually similar but redundant and uninformative as shown in figure 1 . intuitively, it is promising to retrieve a diverse translation memory which would offer maximal coverage of the source sentence and provide useful cues from different aspects. unfortunately, empirical experiments in @xcite show that a diverse translation memory only leads to negligible improvements. as a result, greedy retrieval is adopted in almost all later studies @xcite @xcite @xcite . this paper aims to ask an important question whether diverse translation memories are beneficial for retrieval-augmented nmt. to this end, we propose a powerful retrieval-augmented nmt model called contrastive memory model which takes into account diversity in translation memory from three ways. specifically, (1) during tm retrieval, inspired by maximal marginal relevance (mmr) @xcite , we introduce a conceptually simple while empirically useful retrieval method called contrastive retrieval to find informative translation memories. the core is to retrieve a cluster of translation memories that are similar to the source sentence while contrastive to each other keeping inner-cluster uniformity in the latent semantic space, as shown in figure 1 . (2) in tm encoding, given multiple translation memories, the local and global information should both be captured by the translation model. separately encoding @xcite @xcite or treating them as a long sequence @xcite would inevitably lose such hierarchical structure information. thus, to facilitate the direct communication between different translation memories for local information and gather the global context via message passing mechanism, we propose a hierarchical group attention (hga) module to encode the diverse memories. (3) in the model training phase, to learn salient and distinct features of each tm with respect to target sentence, we devise a novel multi-tm contrastive learning objective (mtcl), which further contributes to a uniformly distributed translation memory cluster by forcing representation of every translation memory to approach the sentence to be translated while keep away from each other. to verify the effectiveness of our framework, we conduct extensive experiments on four benchmark datasets, and observe substantial improvement over strong baselines, proving that diverse translation memories is indeed useful to nmt. our main contributions are: tm-augmented nmt augmenting neural machine translation model with translation memories is an important line of work to boost the performance of the nmt model with non-parametric method. feng et al. (2017) stores memories of infrequently encountered words and utilizes them to assist the neural model. gu et al. (2018) uses an external memory network and a gating mechanism to incorporate tm. cao and xiong (2018) uses an extra gru-based memory encoder to provide additional information to the decoder. xia et al. (2019) adopts a compact graph representation of tm and perform additional attention mechanisms over the graph when decoding. bulté and tezcan (2019) and xu et al. (2020) directly concatenate tm with source sentence using cross-lingual vocabulary. zhang et al. ( 2018 ) augments the model with an additional bonus given to outputs that contain the collected translation pieces. there is also a line of work that trains a parametric retrieval model and a translation model jointly @xcite and achieves impressive results. recently, with rapid growth of computational power, a more fine grained token level translation memories are use in @xcite . this approach gives the decoder direct access to billions of examples at test time, achieving state-of-the-art result even without further training. ## proposed framework preliminary assuming we are given a source sentence @xmath0, ..., x s } and its corresponding target sentence @xmath1, ..., y t } where s, t are their respective length. for a tm-augmented nmt model, a set of similar translation pairs overview given a source sentence x and informative translation memories m , the translation model defines the conditional probability similar to the equation 1. at the high level, our framework, as shown in figure 2 , consists of contrastive retrieval, which searches a diverse translation memory, source encoder which transforms source sentence x into dense vector representations z x , memory encoder with hierarchical group attention module to jointly encode |m | translation memories into a series dense representation z m and decoder which attends to both z x and z m and generates target sentence y in an auto-regressive fashion, and contrastive learning which effectively trains the memory encoder as well as source encoder and decoder. among all these five modules, contrastive memory ( §3.1), memory encoder ( §3.3) and contrastive learing ( §3.5) are key in our framework compared with existing work of tm-augmented nmt. ## conclusion in this work, we introduce an approach to incorporate contrastive translation memories into a nmt system. our system demonstrates its superiority in retrieval, memory encoding and training phases. experimental results on four translation datasets verify the effectiveness of our framework. in the future, we plan to exploit the potential of this general idea in different retrieval-generation tasks.