On Vision Features in Multimodal Machine Translation (2022)

Abstract: Previous work on multimodal machine translation (MMT) has focused on the way of incorporating vision features into translation but little attention is on the quality of vision models. In this work, we investigate the impact of vision models on MMT. Given the fact that Transformer is becoming popular in computer vision, we experiment with various strong models (such as Vision Transformer) and enhanced features (such as object-detection and image captioning). We develop a selective attention model to study the patch-level contribution of an image in MMT. On detailed probing tasks, we find that stronger vision models are helpful for learning translation from the visual modality. Our results also suggest the need of carefully examining MMT models, especially when current benchmarks are small-scale and biased.

## introduction multimodal machine translation (mmt) has emerged as an active field of research which marries the worlds of computer vision (cv) and natural language processing (nlp) @xcite . early models of this kind produce a translation given the fused representation of both the visual and textual inputs @xcite @xcite . as expected, such a paradigm achieves promising bleu improvements and inspires the community to follow up. but soon researchers found that mmt systems did not act as what they ordinarily designed: the visual modality contributes to translation little. for example, it is not harmful to mmt systems when the input image is irrelevant to the text @xcite , or even when the vision features are absent @xcite . more recently, @xcite have pointed out that the * corresponding author. note that previous work has for the most part focused on integrating off-the-shelf vision models (such as resnet-50) into mmt. the underlying assumption here is that the existing vision models are powerful enough to encode the image. this implicitly ignores the quality of vision models in representing images. but computer vision is facing a new trend by moving from cnns to transformer as the backbone model @xcite @xcite . a natural question that arises is: how will mmt systems behave if stronger vision models are adopted? in this work, we address this question by a systematic study of using various vision models in mmt, in particular using the most successful models in recent studies (such as vision transformer, or vit for short). we find that the patch method used in transformer-based vision models offers an opportunity to detail the patch-level contribution of the image. this leads us to develop a selective attention model to correlate words with image patches. beyond this, we introduce object-detection and image captioning features into mmt for further improvements of the vision models @xcite . following @xcite 's work, we design more detailed probing tasks to examine to what degree the visual modality contributes to mmt. we run an extensive set of experiments on en-de and en-fr mmt tasks. our findings are we hope that the results here can inspire more research on exploring better vision models and evaluation methods for multimodal nlp. ## preliminary we start with a description of the probing tasks. it is followed by a design of vision features and a selective attention mechanism for introducing vit-like representations into mmt. ## related work multimodal machine translation is a cross-domain task in the field of machine translation. early attempts mainly focused on enhancing the mmt model by better incorporation of the vision features @xcite @xcite . however, directly encoding the whole image feature brings additional noise to the text @xcite . to address the above issue, yao and wan (2020) proposed a multimodal self-attention to consider the relative difference of information between two modalities. similarly, @xcite used a gumbel softmax to achieve the same goal. researchers also realize that the visual modality may be redundant. irrelevant images have little impact on the translation quality, and no significant bleu drop is observed even the image is absent @xcite . encouraging results appeared in @xcite 's work. they pointed out that the visual modality is still useful when the linguistic context is scarce, but is less sensitive when exposed to complete sentences. more recently, @xcite attributed the bleu gain on mmt tasks to the regularization training, and they again emphasized the imperative of constructing proper insufficient textual input. it is worthy to note that the proposed probing task is an improved version based upon previous work @xcite . we also opensource the preprocessed data and the corresponding scripts for the subsequent researchers to experiment on. another line of research is to explore large-scale cross-modal pretraining models. in this way, the mmt task is regarded as a downstream task. for example, clip @xcite ) is a general cross-modal pretraining model, which learns to perform a wide variety of tasks via natural language prompting. @xcite presented a mmt-specific pretraining model which combines the translation language modeling with masked region classification objectives. in this work, we make a systematic study on whether stronger vision features are helpful. we also extend the research to enhanced features, such as object-detection and image captioning, which are complementary to previous work. ## conclusions in this work, we show that stronger vision features (e.g. vit-like models) strengthen mmt systems on three proposed probing tasks. we present a selective attention method for vit-based models to make better use of the patch-level representation. the result here shows a promising line of research on developing better vision models for multimodal tasks. as far as we know, this is the first attempt to build mmt systems with transformer only. in future work, we are willing to investigate whether it is possible to use a single set of parameters to encode the vision and text modalities.