Improving Retrieval Augmented Neural Machine Translation by Controlling Source and Fuzzy-Match Interactions (2023)

Abstract: We explore zero-shot adaptation, where a general-domain model has access to customer or domain specific parallel data at inference time, but not during training. We build on the idea of Retrieval Augmented Translation (RAT) where top-k in-domain fuzzy matches are found for the source sentence, and target-language translations of those fuzzy-matched sentences are provided to the translation model at inference time. We propose a novel architecture to control interactions between a source sentence and the top-k fuzzy target-language matches, and compare it to architectures from prior work. We conduct experiments in two language pairs (En-De and En-Fr) by training models on WMT data and testing them with five and seven multi-domain datasets, respectively. Our approach consistently outperforms the alternative architectures, improving BLEU across language pair, domain, and number k of fuzzy matches.

## introduction domain adaptation techniques such as fine-tuning @xcite are highly effective at increasing indomain performance of neural machine translation (nmt) systems, but are impractical in many realistic settings. for example, consider a single machine serving translations to thousands of customers, each with a private translation memory (tm). in this case, adapting, storing and loading large adapted models for each customer is computationally infeasible. in this paper we thus consider zero-shot adaptation instead, with a single generaldomain model trained from heterogeneous sources that has access to the customer or domain specific tm only at inference time. our work builds on retrieval augmented translation (rat) @xcite @xcite @xcite , a paradigm which combines a translation model @xcite with an external retriever module that retrieves the top-k most similar source sentences from a tm (i.e. "fuzzy matches") @xcite @xcite . the encoder encodes the input sentence along with the translations of the top-k fuzzymatches and passes the resulting encodings to the decoder. prior rat methods for nmt have fallen into two camps: early work @xcite concatenated the source sentence and the top-k fuzzy matches before encoding, relying on the encoder's self-attention to compare the source sentence to each target sentences and determine which target phrases are relevant for the translation. more recent work @xcite has opted to encode the source sentences and the top-k fuzzy matches independently, effectively shifting the entire burden of determining which target phrases are relevant to the decoder. we hypothesize that neither approach is ideal: in the first, the encoder has access to the information that we expect to be important (namely, the source and the fuzzy matches), but the self-attention also has potentially confusing/spurious connections. in the second, the encoder lacks the self-attention connections between the source and the fuzzy matches. to address these issues, we propose a novel architecture which has self-attention connections between the source sentence and each fuzzy-match, but not between fuzzy-matches. we denote this method rat with selective interactions (rat-si). our method is illustrated in figure 1 , along with two previously discussed approaches. experiments in five english-german (en-de) domain-specific test sets @xcite and seven english-french (en-fr) domain specific test sets @xcite , for @xmath0, 4, 5}, demonstrate that our proposed method outperforms both prior approaches in 32 out of 36 cases considered. the proposed method outperforms the closest competitor by +0.82 to +1.75 bleu for en-de and +1.57 to +1.93 for en-fr. ## method to isolate the effects of the underlying modeling strategy from the various tricks and implementation details employed in prior papers, we build baseline models which distill the two primary modeling strategies used in prior works: the first concatenates a source sentence with target-language fuzzy matches and then encodes the entire sequence, as in @xcite and @xcite . in this approach, the crossattention of the encoder must learn to find the relevant parts of target-language fuzzy-matches by comparing each fuzzy-match to the source sentence, while ignoring potential spurious fuzzymatch to fuzzy-match interactions (see the left diagram in figure 1 ). we denote this method rat-cat. the second encodes the source and each targetlanguage fuzzy-match separately (with two distinct encoders), and instead concatenates the encoded representations, inspired by @xcite and @xcite . in this approach, the spurious connections between the target-language fuzzymatches are eliminated, but the connections between the source and each fuzzy-match are also eliminated, forcing the attention in the decoder to find the relevant portions in the fuzzy-match that are relevant to the source (see the center diagram in figure 1 ). we denote this method rat-sep. finally, we propose a third method which attempts to build on the strengths of each of the prior methods. as in rat-sep, our method separately encodes (with the same encoder) the source and each target-language fuzzy-match; however, each fuzzy-match is jointly encoded with a copy of the source, as in rat-cat, allowing the encoder to find portions of the fuzzy-match which are relevant to the input. finally, all the encoded inputs are concatenated and exposed to the decoder; however, the encoding of the source is only provided to the encoder once, to avoid potentially spurious interactions between copies of the input (see the right diagram in figure 1 ). we denote our proposed method rat-si. ## experimental setup our experiments are in two language directions: english-german (en-de) and english-french (en-fr). we train models using the public wmt 2014 @xcite data set, with 4.5m en-de sentences and 36m en-fr sentences. during training, the model sees target-language fuzzy-match sentences from the same dataset it is being trained on (i.e. wmt14), but at inference, models must perform zero-shot adaptation to five en-de domain-specialized tms 1 and seven en-fr domain-specialized tms. 2 en-de data is taken from @xcite , which is a re-split version of the multi-domain data set from @xcite while en-fr data set is taken from the multi-domain data set of @xcite . to find target-language fuzzy matches for our model from domain specific tms, we use okapi bm25 @xcite , a classical retrieval algorithm that performs search by computing lexical matches of the query with all sentences in the evidence, to obtain top-ranked sentences for each input. to enable fast retrieval, we leverage the implementation provided by the elasticsearch library. 3 specifically, we built an index using source sentences of each tm, and for every input source sentence, we collect top-k similar source side sentences and then use their corresponding target side sentences as inputs to the model. to explore how each method performs (and how robust they are) under different conditions, we run a full set of experiments for @xmath1, 4, 5}. we train separate models for each language pair and k value, and then apply that model to each of the 5 (en-de) or 7 (en-fr) domains. we report translation quality with bleu scores computed via sacrebleu @xcite . 4 we use compare-mt @xcite to perform pairwise significance testing with @xmath2.05 for all pairs. all models employed transformers @xcite with 6 encoder and 6 decoder layers. hidden size was set to 1024 and maximum input length truncated to 1024 tokens. all models employed a joint source-target language subword vocabulary of size 32k using sentencepiece algorithm @xcite . we use the adam optimizer (kingma and ba, 2015) with β 1 = 0.9, β 2 = 0.98 and ϵ = 10 -9 ; and (ii) increase the learning rate linearly for the first 4k training steps and decrease it thereafter; (iii) use batch size of 32k source tokens and 32k target tokens. checkpoints are saved after every 10k iterations during training. we train models with maximum of 300k iterations. we use dropout of 0.1 and label-smoothing of 0.1. ## results results for en-de are shown in we observe several trends in the results. first, our proposed rat-si method outperforms both the rat-cat and rat-sep methods across both language pairs, having the best performance in 32/36 cases considered. in en-de, the proposed rat-si method has an average improvement of 1.43 bleu over rat-cat and 2.35 bleu over rat-sep, while in en-fr we observe an average improvement of 1.73 bleu over rat-cat and 2.98 over rat-sep. these results support our hypothesis that attention connections between the source sentences and each fuzzy match are critical to translation quality and the connections between the fuzzy matches are actually harmful. second, on average, @xmath3, but only by a small amount. however, considering individual language pair / domain combinations, there are many cases where @xmath4, sometimes by several bleu points. we hypothesize that this is due to the different domains containing, on average, different amounts of relevant data. this observation underscores the importance of tuning k, as well as testing new rat methods under a variety of conditions, including different k values. finally, consistent with prior work, we see large improvements for all online domain-adapted methods (rat-cat, rat-sep, and rat-si) over the non-domain-adapted baseline, with improvements of up to +13.85 bleu. this is not surprising, since the baseline model does not take advantage of any domain-specific data. ## conclusion previous work in retrieval augmented translation has used architectures which either have full connections between source and all fuzzy matches, or independently encode the source and each fuzzy match. based on our hypothesize that the attention connections between source and each fuzzy match are helpful, but that the the connections between different fuzzy matches are harmful, we propose a new architecture (rat-si) with the former connections but not the latter. experiments on several language pairs, domains, and different numbers of fuzzy matches (k) demonstrate that rat-si substantially outperforms the prior architectures.