Retrieving Examples from Memory for Retrieval Augmented Neural Machine Translation: A Systematic Comparison (2024)

Abstract: Retrieval-Augmented Neural Machine Translation (RAMT) architectures retrieve examples from memory to guide the generation process. While most works in this trend explore new ways to exploit the retrieved examples, the upstream retrieval step is mostly unexplored. In this paper, we study the effect of varying retrieval methods for several translation architectures to better understand the interplay between these two processes.We conduct experiments in two language pairs in a multi-domain setting and consider several downstream architectures based on a standard autoregressive model, an edit-based model, and a large language model with in-context learning. Our experiments show that the choice of the retrieval technique impacts the translation scores, with variance across architectures. We also discuss the effects of increasing the number and diversity of examples, which are mostly positive across the board.

## introduction retrieval-augmented language models and translation models are getting a lot of traction (see @xcite for a recent review). for translation tasks, the use of retrieval-based techniques that identify the most relevant segment(s) in a translation memory (tm) has long been used in professional computer aided translation environments @xcite , where the retrieved segments provide translators with valuable suggestions. segments closely resembling the source sentence to be translated can also be directly edited to speed up translation. such ideas have also been used in machine translation (mt), first in the examplebased tradition @xcite @xcite , then in the statistical-based paradigm @xcite , more recently for neural machine translation (nmt). there are several ways to take advantage of translation examples in nmt architectures: @xcite use a small set of examples to perform on-the-fly, lightweight, fine-tuning (using both source and target sides); bulte and tezcan (2019) simply concatenate the (target side of) a handful of examples on the source side of the encoder, leaving the rest of their autoregressive decoder unchanged; @xcite repurpose the edit-based architecture of @xcite to compute new translations from existing ones with a nonauto-regressive (nat) decoder; finally, in-context learning (icl) in large language models (llms) provides yet another way to seamlessly combine tms with text generation (see @xcite , inter alia). these studies (and several others, fully discussed in §5) not only differ in the way they use examples but also in the way tms are searched, retrieved, and filtered. this makes the direct comparison between these proposals sometimes difficult to reproduce and analyze. furthermore, it also prevents precisely assessing the computational complexity of the complete translation pipeline. in this paper, we perform experiments with several representative retrieval methods that we systematically combine with multiple ramt architectures. in doing so, our main goal is not to compare these downstream architectures, but rather to better understand the interplay between the retrieval and generation tasks, to make the trade-offs between these steps explicit, and to formulate recommendations regarding future uses of tms in nmt. specifically, we address the following questions: we notably find that (a) retrieval actually matters for edit-based and in-context learning; (b) existing retrieval pipelines can be simplified at inference; (c) optimizing source coverage and/or instance diversity is helping, especially when the closest match is poor. ## related work as for other text generation applications @xcite , efforts to integrate a retrieval component in nmt have intensified in recent years. one motivation is to increase the transparency of ml models by providing users with the retrieved examples that were used to compute the actual output @xcite . for mt, this is achieved by integrating fuzzy matches retrieved from memory as an additional context. this can be performed by concatenating the retrieved target instance to the source text, an approach that also accommodates several tm matches (bulte and tezcan, 2019), or by the simultaneous use of their source and target sides @xcite . more complex schemes to combine retrieved examples with the source sentence are in @xcite @xcite . the recent studies of @xcite ; @xcite and @xcite handle several complementary tm examples retrieved in a contrastive manner that aims to enhance source coverage. @xcite propose a general formulation in the application of icl in various tasks. @xcite also handle multiple matches and introduce two novelties: (a) retrieval is performed directly in the target language and (b) similarity scores are trainable, which allows to evaluate retrieved instances based on their usefulness in translation. most of these attempts rely on auto-regressive (ar) decoding, meaning that the impact of tm match(es) on the output is only indirect. the use of tm memory match with a nat decoder is studied by @xcite ; @xcite , who adapt levt for this specific setting, using one single retrieved instance to initialize the edit-based decoder; @xcite extends this technique to process multiple retrieved examples. @xcite explore a different set of techniques to improve translation using retrieved segments instead of full sentences. generalizing nearest neighbor language models (nnlms) @xcite to conditional lms, @xcite perform k-nnmt as follows: at each decoding step, the k target contexts that closest to the current contextualized representations are retrieved and used to select the next token. this approach is further elaborated in @xcite and extended to chunks by @xcite . a final thread of relevant papers concerns the use of large language models, which, provided with suitable prompts and in-context examples, can be turned into effective translation systems. such approaches have been tested with most llms, with the goal to illustrate the multi-tasking abilities of such models. closer to our work, a series of work have tried to optimize llms performance for the mt task, systematically studying the effect of the prompt change, of the number of shots, and of the in-context examples selection procedures @xcite @xcite . @xcite were the first to combine llms with tms, using an embedding-based retrieval system and combining (via concatenation) up to 5 tm-matches in the mt prompt; @xcite followed suit, with a different llm and a two-stage retrieval strategy (first 500 closest matches for a lucene-based engine; then using up to 9 closest matches for the edit-distance). @xcite studies a way to optimally select k examples so as to maximize coverage, an approach akin to our "contrastive" scenario -using a bm25 retriever in a first stage, and a greedy heuristic selection in a second stage. @xcite explores another benefit of selecting good in-context examples, that of maintaining consistency in the generated text -for this, they retrieve examples from a moving context window of past translations. finally, m et al. (2023) go one step further by training a linear regression model predicting the goodness of tm instances based on a small set of features. ## conclusion this paper has investigated the effect of varying the retrieval strategy for three commonly used retrievalaugmented machine translation architectures, trying to get a better understanding of the interplay of these two components. while auto-regressive encoder-decoder architecture seems quite robust w.r.t. changes in the retrieval strategy, this is less so for the two other architectures, for which optimizing the retrieval policy can yield significant returns. our experiments have also highlighted the benefits of coverage-oriented retrieval policies, based on lcs, especially for the non-autoregressive model. finally, we have validated the use of the "indomain" selection policy and proposed to simplify the inference step by eliminating the filtering process, yielding better performance at the expense of an increased latency. in our future work, we would like to continue the exploration of the interplay between retrieval and translation, with the aim to jointly optimize these two processes rather than have them designed independently.