Vision Matters When It Should: Sanity Checking Multimodal Machine Translation Models (2021)

Abstract: Multimodal machine translation (MMT) systems have been shown to outperform their text-only neural machine translation (NMT) counterparts when visual context is available. However, recent studies have also shown that the performance of MMT models is only marginally impacted when the associated image is replaced with an unrelated image or noise, which suggests that the visual context might not be exploited by the model at all. We hypothesize that this might be caused by the nature of the commonly used evaluation benchmark, also known as Multi30K, where the translations of image captions were prepared without actually showing the images to human translators. In this paper, we present a qualitative study that examines the role of datasets in stimulating the leverage of visual modality and we propose methods to highlight the importance of visual signals in the datasets which demonstrate improvements in reliance of models on the source images. Our findings suggest the research on effective MMT architectures is currently impaired by the lack of suitable datasets and careful consideration must be taken in creation of future MMT datasets, for which we also provide useful insights.

## introduction multimodal machine translation (mmt) aims to improve machine translation by resolving certain contextual ambiguities with the aid of other modalities such as vision, and have shown promising integration in conventional neural machine translation (nmt) models @xcite . on the other hand, recent studies reported some conflicting results regarding how the additional visual information is exploited by the models for generating higher-quality translations. a number of mmt models @xcite @xcite @xcite have been proposed which showed improvements over text-only models, whereas @xcite ; @xcite ; @xcite observed that the multimodal integration did not make a big difference quantitatively or qualitatively. following experimental work showed that replacing the images in image-caption pairs with incongruent images @xcite or even random noise @xcite might still result in similar performance of multimodal models. in light of these results, @xcite suggested that gains in quality might merely be due to a regularization effect and the images may not actually be exploited by models during the translation task. in this paper, we investigate the role of the evaluation benchmark in model performance and whether its tendency to ignore visual information in the input could be a consequence of the nature of the dataset. the most widely-used dataset for mmt is multi30k @xcite @xcite , which extends the flickr30k dataset @xcite to german, french, and czech translations. captions were translated without access to images, and it is posited that this heavily biases mmt models towards only relying on textual input @xcite . mmt models may well be capable of using visual signals, but will only learn to do so if the visual context provides information beyond the text. for instance, the english word "wall" can be translated into german as either "wand" (wall inside of a building) or "mauer" (wall outside of a building), but we find that reference translations in multi30k are not always congruent with images. a number of efforts have been put into creating datasets where correct translations are only possible in the presence of images. @xcite degrade the multi30k dataset to hide away crucial information in the source sentence, includ-ing color, head nouns, and suffixes. similarly, @xcite mask high-frequency words in multi30k. multisense @xcite collects sentences whose verbs have cross-lingual sense ambiguities. however, due to the high cost of data collection, datasets of such kind are often limited in size. multisubs @xcite is another related dataset. which is primarily used for lexical translation because the images are retrieved to align with text fragments rather than whole sentences. in this work, we propose two methods to necessitate the visual context -back-translation from a gender-neutral language (e.g. turkish) and word dropout in the source sentence. they are simple and cheap to implement, allowing them to be applied on much larger datasets. we test the methods on two mmt architectures and find that they indeed make the model more reliant on the images. ## method in this section, we elaborate two methods to conceal important information in the source textual inputs that can be recovered with the aid of visual inputs. ## results the results of our experiments are shown in tab. 2. ## conclusion our experiments explain recent failures in mmt, and show that the models we examine successfully learn to rely more on images when textual ambiguity is high (as in our back-translated turkish-english dataset) or when textual information is dropped out. our results suggest that simple mmt models have some capacity to integrate visual and textual information, but their effectiveness is hidden when training on datasets where the visual signal provides little information. in the long term, we hope to identify real-world applications where multimodal context naturally provides a strong disambiguation signal. for the near future, we release our dataset and encourage researchers to utilize it to validate future research on multimodal translation models. for example, we are interested under which conditions multimodal models learn to exploit visual signal: does the absolute frequency of examples with textual ambiguity matter more, or their proportion?