Rethinking Translation Memory Augmented Neural Machine Translation (2023)

Abstract: This paper rethinks translation memory augmented neural machine translation (TM-augmented NMT) from two perspectives, i.e., a probabilistic view of retrieval and the variance-bias decomposition principle. The finding demonstrates that TM-augmented NMT is good at the ability of fitting data (i.e., lower bias) but is more sensitive to the fluctuations in the training data (i.e., higher variance), which provides an explanation to a recently reported contradictory phenomenon on the same translation task: TM-augmented NMT substantially advances NMT without TM under the high resource scenario whereas it fails under the low resource scenario. Then this paper proposes a simple yet effective TM-augmented NMT model to promote the variance and address the contradictory phenomenon. Extensive experiments show that the proposed TM-augmented NMT achieves consistent gains over both conventional NMT and existing TM-augmented NMT under two variance-preferable (low resource and plug-and-play) scenarios as well as the high resource scenario.

## introduction the effectiveness of translation memory (tm) in machine translation has long been recognized @xcite @xcite @xcite , because a tm retrieved from a bilingual dataset (i.e., training data or an external dataset) may provide valuable knowledge for the source sentence to be translated. many notable approaches recently have been proposed to enhance neural machine translation (nmt) by using a tm @xcite @xcite @xcite . for example, on the standard jrc-acquis task, tm-augmented nmt achieves substantial gains over the vanilla nmt (without tm) under the conventional high-resource training scenario. unfortunately, @xcite surprisingly find that tm-augmented nmt fails to advance the vanilla nmt model on the same task under a lowresource scenario, if a quarter of the full data is used for training and tm retrieval, as reproduced in in this paper, we first cast tm-augmented nmt as an approximation of a latent variable model where the retrieved tm is the latent variable through a probabilistic view of retrieval. from this statistical viewpoint, we identify that the success of such an approximation depends on the variance of tm-augmented nmt with respect to the latent variable. then, we empirically estimate the variance of tm-augmented nmt from the principle of variance-bias decomposition in the learning theory. our findings demonstrate that tmaugmented nmt is worse than the vanilla nmt in terms of variance which indicates the sensitivity to fluctuations in the training set, although tmaugmented nmt is better in terms of the bias which indicates the ability of fitting data. the finding about the variance takes the responsibility of the contradictory phenomenon in to better trade off the variance and the bias, we further propose a simple yet effective method for tm-augmented nmt. the proposed method is general to be applied on top of any tm-augmented nmt models. to validate the effectiveness of the proposed approach, we conduct extensive experiments on several translation tasks under different scenarios including lowresource scenario, plug-and-play scenario, and high-resource scenario. contributions of this paper are three-fold: • it rethinks and analyzes the variance of tmaugmented nmt models from the probabilistic view of retrieval and the bias-variance decomposition perspective. ## proposed approach for lower variance in this section, we first propose two techniques to reduce the variance of the tm-augmented nmt model. then we propose a new tm-augmented nmt on the basis of the two techniques to address the contradictory phenomenon as presented in §1. finally we empirically quantify the variance and bias for the proposed tm-augmented nmt. note that the proposed tm-augmented nmt is general enough to be applied on top of any specific tmaugmented nmt models. ## experiments in this section, we validate the effectiveness of the proposed methods in three scenarios: (1) the lowresource scenario where training pairs are scarce, (2) the plug-and-play scenario where additional bilingual pairs are added as the data store and the model is not re-trained any more as the data store is enlarged, and (3) the conventional high-resource scenario where the entire training data are used for training and retrieval. we use bleu score @xcite as the automatic metric for the translation quality evaluation. ## related work this work mainly contributes to the research line of translation memory (tm) augmented neural machine translation (nmt). early works mainly concentrated on model architecture designing @xcite @xcite @xcite @xcite ; xu et al., 2020 another research line highly related to our work is knn-mt @xcite @xcite @xcite @xcite @xcite . however, knn-mt retrieves similar key-value pairs on the token level, resulting in slow retrieval speed @xcite and large storage space @xcite , whereas tm-augmented nmt retrieves similar pairs on the sentence level, which has more efficient retrieval speed and takes up less storage space. ## conclusion existing work surprisingly finds that tmaugmented nmt fails to advance the nmt model under low-resource scenario, but it is unclear the reason why such a contradictory phenomenon happens. this paper rethinks tm-augmented nmt from latent variable probabilistic model view and variance-bias decomposition view respectively, giving the reason for the failure under low-resource scenario. estimation for variance and bias indicates that tm-augmented nmt is better at fitting data (bias) yet worse at sensitivity to fluctuations in the training data (variance). to better trade off the bias and variance, this paper proposes a simple yet effective weighted ensemble method for tm-augmented nmt. experiments under three scenarios demonstrate that the proposed method outperforms both vanilla transformer and baseline tm-augmented models consistently. future work could aim to find factors that influence the bias of tm-augmented nmt models, such as the quality of retrieved tms.