Multilingual k -Nearest-Neighbor Machine Translation (2023)

Abstract: k -nearest-neighbor machine translation has demonstrated remarkable improvements in machine translation quality by creating a datastore of cached examples. However, these improvements have been limited to high-resource language pairs, with large datastores, and remain a challenge for low-resource languages. In this paper, we address this issue by combining representations from multiple languages into a single datastore. Our results consistently demonstrate substantial improvements not only in low-resource translation quality (up to +3.6 BLEU), but also for high-resource translation quality (up to +0.5 BLEU). Our experiments show that it is possible to create multilingual datastores that are a quarter of the size, achieving a 5.3x speed improvement, by using linguistic similarities for datastore creation.

## introduction recently, semi-parametric approaches such as knearest-neighbor machine translation (knn-mt) @xcite have attracted interest due to a series of impressive results in language modeling and machine translation @xcite @xcite . these techniques capitalize on information retrieved from an extensive repository of translation examples cached in a datastore. one of the most important limitations of knn-mt is that the extent of quality improvements strongly depends on the size of the datastore @xcite @xcite . this dependence on datastore size is problematic for low-resource languages and the improvements that knn-mt can offer in lowresource settings are modest at best @xcite . on the other hand, there are general methods that can improve low-resource performance, such as transfer learning @xcite and multilingual nmt (mnmt) @xcite @xcite . preliminary findings on combining knn-mt with mnmt suggest that mnmt representations generalize sufficiently well across languages to make cross-lingual retrieval effective @xcite . however, its effectiveness for lowresource languages remains an open question. in this paper, we investigate to what extent mnmt can be useful for improving low-resource knn-mt translation quality. first, we experiment with cross-lingual datastores from related languages and find that low-resource languages generally benefit from larger cross-lingual datastores. we then propose a simple yet effective approach, multilingual knn-mt, which uses multilingual datastores that are constructed by merging bilingual datastores. our results show substantial improvements for low-resource languages, and also noticeable improvements for high-resource languages. finally, we show that it is possible to create multilingual datastores that are significantly smaller-thereby resulting in substantially faster decoding times-by relying on linguistic similarities when creating multilingual datastores. ## multilingual k-nearest-neighbor mt despite the potential benefits, the integration of mnmt with knn-mt has only been rarely explored. using a datastore with english on the source side improves performance for other source languages @xcite , but it is not known to what extent this holds for low-resource languages. pre-trained multilingual language models can be used to build monolingual datastores of a target language @xcite , but the required alignment training does not work for low-resource languages due to data scarcity. our goal is to improve performance for low-resource languages by constructing cross-lingual and multilingual datastores. these datastores consist of keys generated from mnmt representations, allowing semantically related sentences from different languages to cluster together @xcite . ## conclusion we have proposed a simple and effective approach to enhance quality for low-resource languages in knn-mt. we augmented an mnmt model with cross-lingual and multilingual datastores of related and unrelated languages. we show that using multilingual datastores substantially improves translation quality for low-resource languages, while high-resource languages also improve. we find that by harnessing linguistic similarity, we can limit multilingual datastore size while preserving quality and significantly increasing inference speed. finally, we show that by further aligning multilingual representations, we can more effectively use crosslingual and multilingual datastores.