Revisiting Source Context in Nearest Neighbor Machine Translation (2023)

Abstract: Nearest neighbor machine translation ( k NN-MT), which interpolates target token probabilities with estimates derived from additional examples, has achieved significant improvements and attracted extensive interest in recent years. However, existing research does not explicitly consider the source context when retrieving similar examples, potentially leading to suboptimal performance. To address this, we comprehensively revisit the role of source context and propose a simple and effective method for improving neural machine translation via source context enhancement, demonstrating its crucial role in both retrieving superior examples and determining more suitable interpolation coefficients. Furthermore, we reveal that the probability estimation can be further optimized by incorporating a source-aware distance calibration module. Comprehensive experiments show that our proposed approach can be seamlessly integrated with representative k NN-MT baselines, resulting in substantial improvements over these strong baselines across a number of settings and domains. Remarkably, these improvements can reach up to 1.6 BLEU points.

## introduction nearest neighbor machine translation @xcite enhances conventional neural machine translation (nmt) by adopting a retrieval-based strategy to interpolate the target token probability distribution produced by the nmt model with the one estimated from similar examples of an auxiliary datastore. since demonstrating promising results in a wide range of machine translation scenarios such as multilingual translation @xcite , do- the ratio of the top-k retrieved examples whose value is equal to the ground-truth token, which is roughly inversely correlated with the source context distance, indicating the potential benifit of source context for knn-mt. the development set of it domain from @xcite is used and @xmath0. please refer to introduction for more detials. preliminary study shows that source context may have significant potiential in knn-mt. typically, an example in knn-mt is denoted as a key-value pair @xcite , where key is the decoder intermediate representation given the source sentence and the target prefix, and value is the next target token observed in training data. the topk examples ranked descendingly according to the distance between the keys and the current decoder hidden state are used to estimate the probability distribution to be interpolated. figure 1 shows the ratio of the top-k examples whose value is equal to the ground-truth target token v.s. the source context distance duing inference, where source context distance is defined as the l2 distance between the representations of the input sentence and the corresponding source sentence of an example (see sec. 3.1 for more details). it is evident that the ratio roughly inversely correlates to the source context distance, indicating potential usefulness of source context in knn-mt. moreover, @xcite show that source context is effective in selecting subset of parallel sentence pairs for improving the efficiency of knn-mt. therefore, source context deserves further attention in knn-mt. in this work, we revisit the source context in knn-mt and reveal that it is effective in enhancing the three fundamental components of knn-mt, namely, example retrieval, probability distribution estimation and interpolation. by leveraging a dynamic datastore, built with data selected based on source context similarities, we manage to recall additional candidate examples that hold potential utility, yet might be overlooked by the existing knn-mt methods. we then perform a distance calibration of all candidates based on both the source and target contexts, thereby achieving a more precise probability distribution estimation. additionally, we propose to determine the interpolation coefficient based on the similarity of the source context. since an advantage of our proposed method is its seamless integration capability with existing knn-mt methods, we integrate our method with representative knn-mt baselines and conduct experiments on five benchmark datasets from different domains. experimental results demonstrate that our method consistently outperforms all the baselines across all these datasets, with improvements of up to 1.6 bleu points. further ablation studies justify that source context indeed plays a crucial role in all the three fundamental components. furthermore, our source context-enhanced knn-mt method achieves significant improvements in other sequence-to-sequence tasks that are challenging for existing knn-mt methods, indicating that source context merits further consideration within the knn-mt framework. ## background this section briefly introduces knn-mt @xcite , formulated as the following two main steps: datastore construction and prediction with datastore. datastore construction a datastore for knn-mt usually contains decoder intermediate representations and corresponding target tokens as keyvalue pairs. specially, given a sentence pair (x, y) from the training dataset @xmath1, y (i) )}, where @xmath2, x 2 , • • • , x m and @xmath3, y 2 , • • • , y n , it is fed into a pre-trained nmt model m in a teacher forcing way. the decoder representation at position t is denoted as f (x, @xmath4, and the corresponding target token is y t , then the datastore d is as follows: prediction with datastore given a source sentence x, the nmt model m generates the prediction ŷ token by token. considering the t-th time step, the decoder representation is denoted as ĥt = f (x, ŷ<t ). a knn-mt model uses ĥt as a query to retrieve the most similar k entries @xmath5, v (i) )} k @xmath6. then the knn distribution p knn (y t |x, ŷ<t ) is calculated based on the l2 distances d( ĥt , h (i) ) between query ĥt and the key h i : ## knn-mt via enhanced source context in this section, we will detail the proposed method, which improves three crucial processes of knn-mt, including retrieval, calibration, and interpolation with source context enhancement. the complete process is illustrated in figure 2 . ## analysis in this section, we attempt to explore several important and meaningful questions through experimental analysis. ## conclusion in this paper, we revisit the role of source context and propose a novel method for improving neural machine translation via source context enhancement, which augments the key processes of knn-mt (i.e., retrieval, calibration, and interpolation). experimental results demonstrate that it can simply and effectively enhance most mainstream knn-mt models, achieving consistent improvements. moreover, our method can significantly improve the robustness of the knn-mt model and help improve its effectiveness in dialogue generation and question generation. we look forward to further exploring the practical utility and efficiency of this method in more natural language generation tasks in the future.