Bridging the Gap between Synthetic and Authentic Images for Multimodal Machine Translation (2023)

Abstract: Multimodal machine translation (MMT) simultaneously takes the source sentence and a relevant image as input for translation. Since there is no paired image available for the input sentence in most cases, recent studies suggest utilizing powerful text-to-image generation models to provide image inputs. Nevertheless, synthetic images generated by these models often follow different distributions compared to authentic images. Consequently, using authentic images for training and synthetic images for inference can introduce a distribution shift, resulting in performance degradation during inference. To tackle this challenge, in this paper, we feed synthetic and authentic images to the MMT model, respectively. Then we minimize the gap between the synthetic and authentic images by drawing close the input image representations of the Transformer Encoder and the output distributions of the Transformer Decoder. Therefore, we mitigate the distribution disparity introduced by the synthetic images during inference, thereby freeing the authentic images from the inference process. Experimental results show that our approach achieves state-of-the-art performance on the Multi30K En-De and En-Fr datasets, while remaining independent of authentic images during inference.

## introduction multimodal machine translation (mmt) integrates visual information into neural machine translation (nmt) to enhance language understanding with visual context, leading to improvement in translation quality @xcite @xcite . however, most existing mmt models require an associated image with the input sentence, which is difficult to satisfy at inference and limits the usage scenarios. hence, overcoming src: on horseback a man attempts to rope a young bull synthetic authentic the dependence on images at inference stands as a pivotal challenge in mmt. to tackle this challenge, some researchers have proposed to employ text-to-image generation models @xcite to generate synthetic images to use as the associated image for the source text at inference, while during training mmt models still use the available authentic images as the visual context to generate translation. despite the remarkable capabilities of textto-image generation models in generating highly realistic images from textual descriptions @xcite , the synthetic images may exhibit different distribution patterns compared to the authentic (ground truth) images. as shown in figure 1 , the synthetic images may depict counterfactual scenes, omit information in the text, or add information irrelevant from the text. therefore, training with authentic images but inferring with synthetic images causing the distribution shift in the images, leading to a decline in translation performance. in this paper, we embrace the utilization of textto-image generation models in mmt and propose a method to bridge the gap between synthetic and authentic images. in our method, we also introduce synthetic images during training and feed synthetic images and authentic images to the mmt model, respectively. then we minimize the gap between the two images by drawing close the following two terms of the mmt model when the two images are inputted respectively: the input image representations to the transformer encoder and the output distributions from the transformer decoder. regarding the input representations, we leverage the optimal transport (ot) theory to mitigate the disparity of the representations. kullback-leibler (kl) divergence is utilized to ensure the consistency of the output distributions. consequently, we effectively eliminate the disparities introduced by synthetic images used during the inference process, thereby freeing the authentic images from the inference process. experimental results show that our approach achieves state-of-the-art performance on the multi30k en-de and en-fr datasets, even in the absence of authentic images. further analysis demonstrates that our approach effectively enhances the representation and prediction consistency between synthetic and authentic images. ## method the original multimodal transformer typically requires a paired image of the input sentence to provide visual context, which is often impractical in many scenarios. to address this limitation, we leverage the stable diffusion model during inference to generate a synthetic image based on the source sentence. however, as synthetic images may follow different distributions compared to the authentic (ground truth) images, using authentic images for training and synthetic images for inference could result in a performance decline in translation quality due to the distribution shift in the images. next, we will introduce our proposed approach for bridging the gap between synthetic and authentic images. ## conclusion in this paper, we strive to address the disparity between synthetic and authentic images in multimodal machine translation. firstly, we feed synthetic and authentic images to the mmt model, respectively. secondly, we minimize the gap between the synthetic and authentic images by drawing close the input image representations of the transformer encoder and the output distributions of the transformer decoder through two loss functions. as a result, we mitigate the distribution disparity introduced by the synthetic images during inference, thereby freeing the authentic images from the inference process. through extensive experiments, we demonstrate the effectiveness of our proposed method in improving the quality and coherence of the translation results.