Low-resource Neural Machine Translation with Cross-modal Alignment (2022)

Abstract: How to achieve neural machine translation with limited parallel data? Existing techniques often rely on large-scale monolingual corpus, which is impractical for some low-resource languages. In this paper, we turn to connect several low-resource languages to a particular high-resource one by additional visual modality. Specifically, we propose a cross-modal contrastive learning method to learn a shared space for all languages, where both a coarse-grained sentence-level objective and a fine-grained token-level one are introduced. Experimental results and further analysis show that our method can effectively learn the cross-modal and cross-lingual alignment with a small amount of image-text pairs, and achieves significant improvements over the text-only baseline under both zero-shot and few-shot scenarios.

## introduction neural machine translation (nmt) has shown excellent performance and becomes the dominant paradigm of machine translation. however, nmt is a data-driven approach, which requires a large amount of parallel data. when the data is insufficient, it is impractical to train a reasonable nmt model. unfortunately, there are many languages in the world for which sufficient training data is not available, and sometimes there is no parallel data at all. therefore, the translation of low-resource languages is a vital challenge for nmt. in recent years, researchers have attempted to improve the performance of nmt for low-resource languages. @xcite proposed an unsupervised approach to learn weak mappings between languages with large amount of monolingual data (>1m), which is also costly for low-resource languages. @xcite ; @xcite ; pan et al. (2021); @xcite proposed multilingual nmt models, which learn a shared space of multiple languages to achieve translations between languages that appear in the training set but do not have the corresponding parallel data. however, they still require auxiliary parallel data of source and target languages along with many other languages, which is still infeasible for low-resource languages. in recent years, with increasing attention of multi-modal tasks, resource of image-text pairs have become more abundant. inspired by recent efforts on cross-modal alignment @xcite @xcite , in this paper, we propose a cross-modal contrastive learning method, which align different languages with images as the pivot to enable zero-shot and few-shot translations for low-resource languages. with parallel sentence pairs between one high-resource auxiliary language and the target language, we can achieve the translation from lowresource languages to the target language only by obtaining small amounts of image-text pairs (<0.1m) for those languages. the parallel sentence pairs are used to learn the mapping from the high-resource language to the target language, and the image-text pairs are used to learn a shared space for all languages through cross-modal alignment. with images as the pivot, the mapping from the low-resource languages to the target language are learned, thus achieving zero-shot translation without any parallel sentence pairs between them. as shown in figure 1 , the high-resource language german and the low-resource language french are brought together by cross-modal alignment, which transfers the translation ability from @xmath0. experiments and analysis show that our method consistently outperforms the baseline under both zero-shot and few-shot scenarios. furthermore, our method can effectively realize crossmodal and cross-lingual alignment. ## method in this section, we present our proposed crossmodal contrastive learning method, which includes both sentence-level and token-level objectives. ## conclusion in this paper, we propose a cross-modal contrastive learning method including sentence-level and token-level objectives, which realizes zeroshot and few-shot translation. experimental results show that our method gains significant improvements over baseline under both scenarios. further analysis demonstrate that our method learns good cross-modal and cross-lingual alignment. in the future, we will explore how our method enables cross-lingual transfer on more tasks.