Towards Example-Based NMT with Multi- L evenshtein Transformers (2023)

Abstract: Retrieval-Augmented Machine Translation (RAMT) is attracting growing attention. This is because RAMT not only improves translation metrics, but is also assumed to implement some form of domain adaptation. In this contribution, we study another salient trait of RAMT, its ability to make translation decisions more transparent by allowing users to go back to examples that contributed to these decisions. For this, we propose a novel architecture aiming to increase this transparency. This model adapts a retrieval-augmented version of the Levenshtein Transformer and makes it amenable to simultaneously edit multiple fuzzy matches found in memory. We discuss how to perform training and inference in this model, based on multi-way alignment algorithms and imitation learning. Our experiments show that editing several examples positively impacts translation scores, notably increasing the number of target spans that are copied from existing instances.

## introduction neural machine translation (nmt) has become increasingly efficient and effective thanks to the development of ever larger encoder-decoder architectures relying on transformer models @xcite . furthermore, these architectures can readily integrate instances retrieved from a translation memory (tm) @xcite @xcite , thereby improving the overall consistency of new translations compared to past ones. in such context, the autoregressive and generative nature of the decoder can make the process (a) computationally inefficient when the new translation has very close matches in the tm; (b) practically ineffective, as there is no guarantee that the output translation, regenerated from scratch, will resemble that of similar texts. based on this observation, our primary goal is to optimize further the number of tokens copied from the tm. to do so, we propose simultaneously editing multiple fuzzy matches retrieved from memory, using a computational architecture -multi-levt, or tm n -levt for short -which extends tm-levt to handle several initial translations. the benefit is twofold: (a) an increase in translation accuracy; (b) more transparency in the translation process. extending tm-levt to tm n -levt however requires solving multiple algorithmic and computational challenges related to the need to compute multiple string alignments (msas) between the matches and the reference translation, which is a notoriously difficult problem; and designing appropriate training procedures for this alignment module. our main contributions are the following: our code and experimental configurations are available on github. foot_1 2 preliminaries / background ## optimal alignment training the combination operation introduced above requires specifying the expert decision for each state. while levt derives its expert policy π * from the computation of edit distances, we introduce another formulation based on the computation of maximal covers. for @xmath0, these formulations can be made equivalent 5 @xcite . ## related work as for other machine learning applications, such as text generation @xcite , efforts to integrate a retrieval component in neural-based mt have intensified in recent years. one motivation is to increase the transparency of ml models by providing users with tangible traces of their internal computations in the form of retrieved examples @xcite . for mt, this is achieved by integrating fuzzy matches retrieved from memory as an additional conditioning context. this can be performed simply by concatenating the retrieved target instance to the source text (bulte and tezcan, 2019), an approach that straightforwardly accommodates several tm matches @xcite , or the simultaneous exploitation of their source and target sides @xcite . more complex schemes to combine retrieved examples with the source sentence are in @xcite @xcite . the recent work of @xcite handles multiple complementary tm examples retrieved in a contrastive manner that aims to enhance source coverage. @xcite also handle multiple matches and introduce two novelties: (a) retrieval is performed in the target language and (b) similarity scores are trainable, which allows to evaluate retrieved instances based on their usefulness in translation. most of these attempts rely on an auto-regressive (ar) decoder, meaning that the impact of tm match(es) on the final output is only indirect. the use of tm memory match with a nat decoder is studied in @xcite @xcite , which adapt levt for this specific setting, using one single retrieved instance to initialize the edit-based decoder. other evolutions of levt, notably in the context of constraint decoding, are in @xcite , while a more general account of nat systems is in @xcite . @xcite explore a different set of techniques to improve translation using retrieved segments instead of full sentences. extending knnbased language models @xcite to the conditional case, @xcite proposes knearest neighbor mt by searching for target tokens that have similar contextualized representations at each decoding step, an approach further elaborated by @xcite ; @xcite and extended to chunks by @xcite . ## conclusion and outlook in this work, we have extended the levenshtein transformer with a new combination operation, making it able to simultaneously edit multiple fuzzy matches and merge them into an initial translation that is then refined. owing to multiple algorithmic contributions and improved training schemes, we have been able to (a) increase the number of output tokens that are copied from retrieved examples; (b) obtain performance improvements compared to using one single match. we have also argued that retrieval-based nmt was a simple way to make the process more transparent for end users. next, we would like to work on the retrieval side of the model: first, to increase the diversity of fuzzy matches e.g. thanks to contrastive retrieval, but also to study ways to train the retrieval mechanism and extend this approach to search monolingual (target side) corpora. another line of work will combine our techniques with other approaches to tm-based nmt, such as keeping track of the initial translation(s) on the encoder side.