Empirical Assessment of k NN - MT for Real-World Translation Scenarios (2023)

Abstract: This paper aims to investigate the effectiveness of the k-Nearest Neighbor Machine Translation model (kNN-MT) in real-world scenarios. kNN-MT is a retrieval-augmented framework that combines the advantages of parametric models with non-parametric datastores built using a set of parallel sentences. Previous studies have primarily focused on evaluating the model using only the BLEU metric and have not tested kNN-MT in real world scenarios. Our study aims to fill this gap by conducting a comprehensive analysis on various datasets comprising different language pairs and different domains, using multiple automatic metrics and expert evaluated Multidimensional Quality Metrics (MQM). We compare kNN-MT with two alternate strategies: fine-tuning all the model parameters and adapter-based finetuning. Finally, we analyze the effect of the datastore size on translation quality, and we examine the number of entries necessary to bootstrap and configure the index.

## introduction the remarkable advances in neural models have brought significant progress in the field of machine translation @xcite @xcite . however, current systems rely heavily on a fully-parametric approach, where the entire training data is compressed into the model parameters. this can lead to inadequate translations when encountering rare words or sentences outside of the initial training domain @xcite , requiring several stages of fine-tuning to adapt to data drift or to new domains. by combining the advantages of parametric models with non-parametric databases built from parallel sentences, retrieval-augmented models showed to be a promising solution, particularly in domain adaptation scenarios @xcite @xcite @xcite @xcite . one notable example is the k-nearest neighbor machine translation model (knn-mt) @xcite , known for its simplicity and very promising results. the model first creates a tokenlevel datastore using parallel sentences, and then it retrieves similar examples from the database during inference, enhancing the generation process via interpolation of probability distributions. however, despite its potential, the knn-mt model has yet to be tested in real-world scenarios. previous studies have primarily focused on evaluating it using only the bleu metric, which correlates poorly with human judgments. in order to gain a deeper understanding of when and how knn-mt can be effective, we conduct a thorough analysis on various datasets which comprise 4 different language pairs and 3 different domains, using bleu @xcite , @xcite , and multidimensional quality metrics (mqm) -quality assessments obtained from the identification of error spans in translation outputs by experts @xcite . to sum up, our main contributions are: parametric component neighbors softmax interpolation nn datastore 0 1 2 k -1 ... • we compare using knn-mt with directly using a pre-trained multilingual model, finetuning all the model parameters, and with adapter-based fine-tuning, reporting results in several automatic metrics. ## k-nearest neighbor machine translation in machine translation, the goal is to take a sentence or document in a source language, represented as @xmath0, . . . , x l ], and generate a corresponding translation in a target language, represented as @xmath1, . . . , y n ]. this is typically achieved using a fully-parametric sequenceto-sequence model @xcite @xcite . in these models, the encoder takes in the source sentence and outputs a set of hidden states. the decoder then generates the target translation one token at a time by attending to these hidden states and outputting a probability distribution over the vocabulary for each step, p nmt (y t |@xmath2, x). finally, a search procedure, such as beam search @xcite , is applied using these probability distributions to generate the final translation. the k-nearest neighbor machine translation model (knn-mt) @xcite , il- translation contexts targets j'ai été à paris. i have j'avais été à la maison. i j'apprécie l'été. i enjoy ... j'ai ma propre chambre. been had summer ... have ... been had summer ... have datastore keys values lustrated in figure 1 , is a retrieval-augmented model. it combines a standard sequence-tosequence model as the one described above, with an approximate nearest neighbor retrieval mechanism, that allows the model to access a datastore of examples at inference time. ## experimental settings in order to analyze how knn-mt performs in realworld scenarios, we performed experiments using datasets from several domains and different language pairs (as described in §3.1). we compared the results with that of a pre-trained multilingual model (referred to as the base model; see §3.2), fine-tuning all the parameters of the base model (as discussed in §3.3), and using adapter-based finetuning (as described in §3.4). the specific settings of knn-mt are detailed in §3.5 and the automatic metrics employed are described in §3.6. ## results with automatic metrics we report the results of our experiments using automatic metrics in tables 4 and 5 , which we discuss in the following sections. ## results with mqm assessments to complement this analysis, we evaluated the performance of the pre-trained model with and without retrieval, as well as the fully fine-tuned model with and without retrieval using multidimensional quality metrics (mqm) -quality assessments obtained from the identification of error spans in translation outputs @xcite . to conduct this assessment, we had professional linguists assessing the models' translations for the en-ko, en-de (2), and en-fr test sets. we asked the annotators to identify all errors and independently label them with an error category (accuracy, fluency, and style, each with a specific set of subcategories) and a severity level (neutral, minor, major, and critical). ures 5, 6, and 7 provide a breakdown of the error typology distribution. the mqm assessment indicates that both fine-tuning and knn-mt significantly improve translation performance when compared to the base model, resulting in a substantial increase in mqm score and a notable reduction in critical, major, and minor errors. interestingly, according to the mqm scores and in contrast to the automatic metric scores, knn-mt slightly outperforms fine-tuning in two out of the three datasets. moreover, in the customer service datasets (en-fr and en-de (2)), knn-mt proved to be useful in mitigating source sentence errors, which are prevalent in this domain and can adversely impact the translation quality (gonc ¸alves et al., 2022). additionally, combining knn-mt with fine-tuning results in marginal improvements for two datasets. retrieval-augmented models have also been explored in the field of machine translation. earlier works have proposed using a search engine to retrieve similar sentence pairs and incorporating them through shallow and deep fusion @xcite or attention mechanisms @xcite , or retrieving n-grams to up-weight token probabilities @xcite . more recently, the knn-mt model has been proposed as an adaptation of the knn-lm for machine translation @xcite , and was then extended with a network that determines the number of retrieved tokens to consider @xcite . as knn-mt can be up to two orders of magnitude slower than a fully-parametric model, @xcite and ( wang et al., 2021) proposed the fast and faster knn-mt, in which the model has a higher decoding speed by creating a different datastore based on the source sentence for each example. (martins et al., 2022a) proposed efficient knn-mt by adapting the methods introduced by (he et al., 2021) to machine translation and introducing a retrieval distributions cache to speed-up decoding. (martins et al., 2022b) proposed retrieving chunks of tokens instead of single tokens. however, most of these methods have been evaluated on a limited number of datasets and language pairs, and using only the bleu metric. our paper addresses this gap by evaluating knn-mt across five "real-world" datasets and four language pairs using comet and mqm evaluation. ## conclusions in this paper, we conducted a study to assess the performance k-nearest neighbor machine translation (knn-mt) in real-world scenarios. to do so, we augmented a pre-trained multilingual model with knn-mt's retrieval component and compared it against using the pre-trained model, performing fine-tuning, and doing adapter-based finetuning on five datasets comprising four language pairs and three different domains. the results on automatic metrics, comet and bleu, revealed that while knn-mt significantly improves the translation quality over the pre-trained language model, it falls short when compared to fine-tuning and adapter-based fine-tuning. furthermore, we observed that incorporating knn-mt's retrieval component into a fine-tuned model resulted in small improvements. we also assessed the knn-mt model using multidimensional quality metrics (mqm) by having professional linguists evaluate the translations for the en-ko, en-de (2), and en-fr test sets. the mqm scores revealed a significant improvement in the knn-mt model over the base model, with knn-mt slightly outperforming fine-tuning in two out of three language pairs. combining knn-mt with a fine-tuned model resulted in minor improvements. additionally, we analyzed the effect of the number of entries in the datastore on translation quality and the number of entries required to train the faiss index. our findings suggest that having larger datastores improves translation quality, with the improvement steepness being higher when increasing the size of a small datastore. the number of entries used to train the faiss index has a small impact on the final translation quality, which is relevant when creating a dynamic datastore that can be updated when more data becomes available.