Multimodal Neural Machine Translation Using Synthetic Images Transformed by Latent Diffusion Model (2023)

Abstract: This study proposes a new multimodal neural machine translation (MNMT) model using synthetic images transformed by a latent diffusion model. MNMT translates a source language sentence based on its related image, but the image usually contains noisy information that are not relevant to the source language sentence. Our proposed method first generates a synthetic image corresponding to the content of the source language sentence by using a latent diffusion model and then performs translation based on the synthetic image. The experiments on the English-German translation tasks using the Multi30k dataset demonstrate the effectiveness of the proposed method.

## introduction recently, multimodal neural machine translation (mnmt) @xcite , which uses images in addition to source language sentences for translation, has attracted attention in the field of machine translation @xcite . images related to source language sentences are considered to improve translation performance by resolving ambiguity during translation and complementing information that is difficult to capture with source language sentences. however, a source language sentence often only describes one aspect of the contents included in its related image. figure 1 shows an example from a standard dataset in mnmt, the multi30k dataset @xcite . as shown in figure 1 , multiple source language sentences with differing content are associated with a single image in the multi30k. for example, source language sentence 2 does not mention the house in the related image. therefore, related images are not necessarily optimal as auxiliary information for mt. therefore, in this study, we propose a new mnmt model using a synthetic image generated by image conversion with a latent diffusion model. specifically, an original related image is converted with a latent diffusion model based on its source language sentence; content unrelated to the source language sentence is eliminated from the original image, and an image conforming with the source language sentence is generated. subsequently, translation is performed by using the converted synthetic image instead of the original related image. our aim is to improve translation performance by using related images that better reflect the content of source language sentences as auxiliary information for translation. we verified the effectiveness of our proposed method on the english-german translation tasks using the multi30k dataset @xcite and the ambiguous coco dataset @xcite . the results confirmed that, compared with a conventional mnmt using the original related images in the multi30k, our method improved the bleu score by 0.14 on both the multi30k test 2016 and test 2017, and by 0.39 on the ambiguous coco. additionally, clipscore @xcite , which was used to calculate the similarity between a source language sentence and an image, confirmed that the synthetic images used in our method more closely match the source language sentences than the original related images. ## conventional mnmt models mnmt models based on transformer @xcite have recently become mainstream in the field of mnmt. various attempts have been made to improve their translation performance, including the introduction of visual attention mechanisms @xcite , as well as the method of simultaneously learning feature representations of text and images using a shared encoder @xcite . li et al. (2022) have proposed a transformer mnmt model incorporating selective attention, an attention mecha-original related image transformed image 2 a girl going into a wooden building . a child in a pink dress is climbing up a set of stairs in an entry way . image transformation multimodal neural machine translation using transformed images with latent diffusion model mnmt model ein kleines mädchen klettert in ein holzhäuschen, das als stall dient. source language sentence 1 source language sentence 2 transformed image 1 target language sentece 1 image transformation figure 1: overview of the proposed method the selective attention mnmt model first encodes the source language sentence x text and the related image x img into feature expressions h text and h img by eqs. ( 1 ) and ( 2 ), respectively. finally, h out is input to the transformer decoder to generate a translated sentence. ## proposed method in this section, we propose an mnmt model that uses synthetic images transformed from related images based on source language sentences. figure 1 shows an overview of the proposed method. the mnmt dataset consists of the triplets of a source language sentence, a target language sentence, and a related image. in typical mnmt datasets, each source language sentence usually only represents one aspect of the content included in the related images; there are many cases where content unrelated to the source language sentence exists in the related image. for example, the image in figure 1 shows a scene where a girl in a pink dress climbs the stairs to enter a wooden house, but source language sentence 1 does not mention the climbing of stairs. further, source language sentence 2 does not refer to a house. therefore, related images are not necessarily the best aids to translation. accordingly, our proposed method first uses a latent diffusion model to eliminate content unrelated to the source language sentence from the related image and generate a synthetic image that corresponds to the source language sentence (see section 3.1). then, translation is performed with a conventional mnmt model (e.g., the selective attention mnmt model in our experiments) using the generated synthetic image and the source language sentence. because this makes it easier to capture the relationship between the input image and text during translation, we expect the improvement of translation performance. ## discussion this section analyzes the synthetic images used in the proposed method. examples of transformed images are shown in appendix a. in order to investigate how much of the image corresponds to the source language sentence, we computed clip-score @xcite , which measures the similarity between the image used and the source language sentence by using clipscore(c, v) = w • max(cos(c, v), 0), where c and v are the feature vectors from the text encoder and the image encoder of the clip @xcite , respectively. w is used to rescale the output, and following @xcite , we set it to 2.5. the evaluation results are shown in ## conclusion in this study, we proposed a new mnmt model that uses a latent diffusion model to transform related images into synthetic images that more closely conform to source language sentences and uses the transformed images as auxiliary information for mt. the experiments on the english-german translation tasks using the multi30k dataset showed that the proposed method can achieve higher translation performance than conventional methods, demonstrating the effectiveness of our proposed method. the evaluation using clipscore confirms that the images used in our method possess more similarities to the source language sentences than the original images.