Visual Pivoting Unsupervised Multimodal Machine Translation in Low-Resource Distant Language Pairs (2024)

Abstract: Unsupervised multimodal machine translation (UMMT) aims to leverage vision information as a pivot between two languages to achieve better performance on low-resource language pairs. However, there is presently a challenge: how to handle alignment between distant language pairs (DLPs) in UMMT. To this end, this paper proposes a visual pivoting UMMT method for DLPs. Specifically, we first construct a dataset containing two DLPs, including English-Uyghur and Chinese-Uyghur. We then apply the visual pivoting method for both to pre-training and fine-tuning, and we observe that the images on the encoder and decoder of UMMT have noticeable effects on DLPs. Finally, we introduce informative multi-granularity image features to facilitate further alignment of the latent space between the two languages. Experimental results show that the proposed method significantly outperforms several baselines on DLPs and close language pairs (CLPs).

## introduction neural machine translation (mt) @xcite @xcite has become a promising method for mt, which depends on the availability of large-scale parallel corpora. however, the preparation of such corpora in the low-resource language is extremely challenging, and existing studies @xcite have shown that neural mt achieves much worse translation quality than statistical mt with a small number of corpora. therefore, developing methods to alleviate the need for annotation of large parallel corpora has attracted increasing attention from researchers. to alleviate this problem, unsupervised mt @xcite has been proposed, which relies on monolingual corpora and trains mt model in an unsupervised manner. since the alignment of the source-target sentence of the unsupervised mt is uncertain, it is highly subject to initialization. therefore, researches @xcite @xcite have found that exploiting visual content for unsupervised mt while leveraging a language model pre-trained on large-scale monolingual data is a feasible way to improve translation quality. visual content is qualified to improve alignment in the latent space of language because the physical visual perception of people who speak different languages is similar. however, previous works @xcite @xcite mainly consider high-resource clps, such as english-german and english-france. because unsupervised mt aims to achieve high-quality translation results with low-resource language. the study of unsupervised mt solely on high-resource clps makes it challenging to assess its effectiveness in low-resource languages, diminishing its applicability and hindering the advancement of ummt efficiency. in dlps, even initialization with a monolingual pre-trained model does not yield significant improvements, as unsupervised mt performs well when the monolingual data in both languages belong to the same language family @xcite . therefore, the ummt task needs to be extended to translation between low-resource dlps, which is beneficial for a more comprehensive exploration of the influence of linguistic distance and the contribution of visual content. to address these challenges, we propose a visual pivoting ummt method for dlps. specifically, we first manually translate the mainstream multimodal mt dataset multi30k @xcite , which primarily contains high-resource clps, into chinese and uyghur. both chinese and uyghur belong to different language families from the languages of the multi30k. even their scripts and grammar structures are different, as shown in figure 1 ; for example, uyghur is a subject-objectverb language, while english and chinese are subject-verb-object languages. moreover, uyghur is a low-resource language, hence the main data studied in this paper are the low-resource dlps composed of english-uyghur and chinese-uyghur sentences. we then extend mlm @xcite by leveraging visual information to generate a visual pre-training language modeling (vplm) model, which is subsequently applied to initialize a ummt model. finally, we use images as a pivot to semantically align source-target languages into a shared latent space. specifically, the image is introduced into the encoder to correct the pseudo-sentence, while the input image in the decoder is treated as a pivot between the source and target languages. we conducted experiments on dlps and clps and the results show tha proposed method consistently outperforms several baselines. overall, we make the following contributions: (i) we construct a dataset with dlps and the ummt is implemented on this dataset. it provides a benchmark for further research on this challenging task. (ii) we find that visual content is more qualified to improve the alignment of dlps latent space. (iii) the experimental results show that in unsupervised mt between gender and gender-neutral language, images contribute to improving gender accuracy. ## method in this section, we first detail the visual pivoting for ummt and multimodal alignment, and then introduce the ummt model and the training strategy. ## limitations although our method achieves good results, it also has some limitations in dealing with dlps. as can be seen from figure 4 , incorporating more image features may hurt the accuracy of a highscore translated sentence. more persons are needed to join our human evaluations since translation is subjective to some degree. the differences among different persons could be analyzed in detail.