Enabling Unsupervised Neural Machine Translation with Word-level Visual Representations (2023)

Abstract: Unsupervised neural machine translation has recently made remarkable strides, achieving impressive results with the exclusive use of monolingual corpora. Nonetheless, these methods still exhibit fundamental flaws, such as confusing similar words. A straightforward remedy to rectify this drawback is to employ bilingual dictionaries, however, high-quality bilingual dictionaries can be costly to obtain. To overcome this limitation, we propose a method that incorporates images at the word level to augment the lexical mappings. Specifically, our method inserts visual representations into the model, modifying the corresponding embedding layer information. Besides, a visible matrix is adopted to isolate the impact of images on other unrelated words. Experiments on the Multi30k dataset with over 300,000 self-collected images validate the effectiveness in generating more accurate word translation, achieving an improvement of up to + 2.81 BLEU score, which is comparable or even superior to using bilingual dictionaries.

## introduction unsupervised neural machine translation (unmt) @xcite @xcite seeks to avoid the necessity for high-quality bilingual data in conventional machine translation by exploring the possibility of completing translations relying solely on the monolingual corpus. consisting of three main components, cross-lingual language model initialization, denoising auto-encoder, and iterative back translation, recent methods @xcite have achieved promising results. nonetheless, recent unmt approaches have been observed to make particular errors in comparison to supervised machine translation, such as confusing nouns that pertain to the same semantic category. for instance, @xcite demonstrates that their models translate "lion" to "rabbit" or "snake" despite getting good chrf scores. in the context of supervised machine translation, pinpointing accurate translations of words is straightforward with aligned bilingual data. however, with only monolingual data, the unsupervised model must rely on large amounts of context for word mapping, which can lead to confusion between words with similar distributions, particularly when there is insufficient data. to address this issue, a direct solution is to integrate high-quality bilingual dictionaries into the unmt process using codeswitching or spliced methods @xcite . high-quality bilingual dictionaries can be obtained through manual annotation and bilingual data extraction, yet these methods either involve high costs or deviate from the original objective of the unmt task. in addition to utilizing explicit bilingual dictionaries for word mapping, implicit methods can also be employed. as the saying goes, "a picture is worth a thousand words". we can leverage the "thousand words" represented by corresponding images to learn the mapping between words in different languages. visual content has the potential to enhance word mapping abilities in latent spaces, as the physical visual perception is congruent among speakers of diverse languages @xcite . furthermore, the cost of annotating a bilingual dictionary is relatively high, whereas the cost of searching images with monolingual keywords is relatively low in the era of sophisticated search engines. describing an image is also less demanding for humans compared to translating a word, as the former only requires mastery of one language while the latter requires at least two. drawing on these insights, we propose to integrate word-level image information into the unmt process to mitigate the issue of lexical confusion. it's worth noting that we don't use sentence-level images like some multi-modal unmt approaches @xcite al. (2019); huang et al. (2020) ). instead, we focused on word-level images since they are more effective and flexible in addressing the challenge of lexical confusion and are easier to collect. our approach involves augmenting the encoded images to their corresponding words, modifying the corresponding position and language encoding information, and utilizing a visible matrix to isolate the impact of images on other words. we conduct experiments utilizing over 300,000 self-collected images, and further analysis demonstrates that we have effectively mitigated lexical confusion to a certain extent. furthermore, we achieve new stateof-the-art results for the unmt task in certain language directions on the multi30k dataset. the main contributions of this work can be summarized as follows: ## methods in this section, we first review the text-only unmt process. then, we present our proposed method for incorporating images, which involves the modified embedding layer and mask transformer. finally, we introduce some training strategies. ## experiments in this section, we provide a detailed description of the datasets and experimental setup employed in our study. subsequently, we perform an in-depth analysis of the results obtained. ## conclusion and future work in this paper, we propose a method to alleviate the problem of lexical confusion in unsupervised machine translation by utilizing word-level images and provide a concise yet potent framework for integrating them into an unsupervised neural machine translation model. through extensive experiments, we demonstrate that our method can effectively mitigate the issue of word confusion and even outperform the use of costly bilingual dictionaries in some directions. furthermore, we release an open-source word-image dataset that covers 5 languages and involves over 300,000 images in total. in the future, we will endeavor to explore the potential of utilizing images for augmenting the translation quality of low-resource languages in a large-language-model-based translation system.