Good for Misconceived Reasons: An Empirical Revisiting on the Need for Visual Context in Multimodal Machine Translation (2021)

Abstract: A neural multimodal machine translation (MMT) system is one that aims to perform better translation by extending conventional text-only translation models with multimodal information. Many recent studies report improvements when equipping their models with the multimodal module, despite the controversy of whether such improvements indeed come from the multimodal part. We revisit the contribution of multimodal information in MMT by devising two interpretable MMT models. To our surprise, although our models replicate similar gains as recently developed multimodal-integrated systems achieved, our models learn to ignore the multimodal information. Upon further investigation, we discover that the improvements achieved by the multimodal models over text-only counterparts are in fact results of the regularization effect. We report empirical findings that highlight the importance of MMT models’ interpretability, and discuss how our findings will benefit future research.

## introduction multimodal machine translation (mmt) aims at designing better translation systems by extending conventional text-only translation systems to take into account multimodal information, especially from visual modality @xcite . despite many previous success in mmt that report improvements when models are equipped with visual information @xcite @xcite @xcite , there have been continuing debates on the need for visual context in mmt. in particular, @xcite ; @xcite ; @xcite argue that visual context does not seem to help translation reliably, at least as measured by automatic metrics. @xcite ; @xcite provide further evidence by showing that mmt models are, in fact, insensitive to visual input and can translate without significant performance losses even in the presence of features derived from unrelated images. a more recent study @xcite , however, shows that under limited textual context (e.g., noun words are masked), models can leverage visual input to generate better translations. but it remains unclear where the gains of mmt methods come from, when the textual context is complete. the main tool utilized in prior discussion is adversarial model comparison -explaining the behavior of complex and black-box mmt models by comparing performance changes when given adversarial input (e.g., random images). although such an opaque tool is an acceptable beginning to investigate the need for visual context in mmt, they provide rather indirect evidence @xcite . this is because performance differences can often be attributed to factors unrelated to visual input, such as regularization @xcite , data bias @xcite , and some others @xcite . from these perspectives, we revisit the need for visual context in mmt by designing two interpretable models. instead of directly infusing visual features into the model, we design learnable components, which allow the model to voluntarily decide the usefulness of the visual features and reinforce their effects when they are helpful. to our surprise, while our models are shown to be effective on multi30k @xcite and vatex @xcite datasets, they learn to ignore the multimodal information. our further analysis suggests that under sufficient textual context, the improvements come from a regularization effect that is similar to random noise injection @xcite and weight decay @xcite . the addi-tional visual information is treated as noise signals that can be used to enhance model training and lead to a more robust network with lower generalization error @xcite . repeating the evaluation under limited textual context further substantiates our findings and complements previous analysis @xcite . our contributions are twofold. first, we revisit the need for visual context in the popular task of multimodal machine translation and find that: (1) under sufficient textual context, the mmt models' improvements over text-only counterparts result from the regularization effect (section 5.2). (2) under limited textual context, mmt models can leverage visual context to help translation (section 5.3). our findings highlight the importance of mmt models' interpretability and the need for a new benchmark to advance the community. second, for the mmt task, we provide a strong text-only baseline implementation and two models with interpretable components that replicate similar gains as reported in previous works. different from adversarial model comparison methods, our models are interpretable due to the specifically designed model structure and can serve as standard baselines for future interpretable mmt studies. our code is available at https://github. com/lividwo/revisit-mmt . ## background one can broadly categorize mmt systems into two types: (1) conventional mmt, where there is gold alignment between the source (target) sentence pair and a relevant image and (2) retrieval-based mmt, where systems retrieve relevant images from an image corpus as additional clues to assist translation. conventional mmt most mmt systems require datasets consist of images with bilingual annotations for both training and inference. many early attempts use a pre-trained model (e.g., resnet @xcite ) to encode images into feature vectors. this visual representation can be used to initialize the encoder/decoder's hidden vectors @xcite @xcite . it can also be appended/prepended to word embeddings as additional input tokens @xcite . recent works @xcite @xcite employ attention mechanism to generate a visual-aware representation for the decoder. for instance, doubly-att @xcite @xcite insert an extra visual attention sub-layer between the decoder's source-target attention sub-layer and feed-forward sub-layer. while there are more works on engineering decoders, encoder-based approaches are relatively less explored. to this end, @xcite and @xcite replace the vanilla transformer encoder with a multi-modal encoder. besides the exploration on network structure, researchers also propose to leverage the benefits of multi-tasking to improve mmt @xcite . the imagination architecture @xcite decomposes multimodal translation into two subtasks: translation task and an auxiliary visual reconstruction task, which encourages the model to learn a visually grounded source sentence representation. retrieval-based mmt the effectiveness of conventional mmt heavily relies on the availability of images with bilingual annotations. this could restrict its wide applicability. to address this issue, @xcite propose uvr-nmt that integrates a retrieval component into mmt. they use tf-idf to build a token-to-image lookup table, based on which images sharing similar topics with a source sentence are retrieved as relevant images. this creates image-bilingual-annotation instances for training. retrieval-based models have been shown to improve performance across a variety of nlp tasks besides mmt, such as question answering @xcite , dialogue @xcite , language modeling @xcite , question generation @xcite , and translation @xcite . ## method in this section we introduce two interpretable mmt models: (1) gated fusion for conventional mmt and (2) dense-retrieval-augmented mmt (rmmt) for retrieval-based mmt. our design philosophy is that models should learn, in an interpretable manner, to which degree multimodal information is used. following this principle, we focus on the component that integrates multimodal information. in particular, we use a gating matrix λ @xcite to control the amount of visual information to be blended into the textual representation. such a matrix facilitates interpreting the fusion process: a larger gating value λ @xmath0, 1] indicates that the model exploits more visual context in translation, and vice versa. ## experiment in this section, we evaluate our models on the multi30k and vatex benchmark. ## model analysis taking a closer look at the results given in the previous section, we are surprised by the observation that our models learn to ignore visual context when translating (sec 5.1). this motivates us to revisit the contribution of visual context in mmt systems (sec 5.2). our adversarial evaluation shows that adding model regularization achieves comparable results as incorporating visual context. finally, we discuss when visual context is needed (sec 5.3) and how these findings could benefit future research. ## conclusion in this paper we devise two interpretable models that exhibit state-of-the-art performance on the widely adopted mmt datasets -multi30k and the new video-based dataset -vatex. our analysis on the proposed models, as well as on other existing mmt systems, suggests that visual context helps mmt in the similar vein as regularization methods (e.g., weight decay), under sufficient textual context. those empirical findings, however, should not be understood as us downplaying the importance existing datasets and models; we believe that sophisticated mmt models are necessary for effective grounding of visual context into translation. our goal, rather, is to (1) provide additional clarity on the remaining shortcomings of current dataset and stress the need for new datasets to move the field forward; (2) emphasise the importance of interpretability in mmt research.