Decoupling the Role of Data, Attention, and Losses in Multimodal Transformers (2021)

Abstract: Recently, multimodal transformer models have gained popularity because their performance on downstream tasks suggests they learn rich visual-linguistic representations. Focusing on zero-shot image retrieval tasks, we study three important factors that can impact the quality of learned representations: pretraining data, the attention mechanism, and loss functions. By pretraining models on six datasets, we observe that dataset noise and language similarity to our downstream task are important indicators of model performance. Through architectural analysis, we learn that models with a multimodal attention mechanism can outperform deeper models with modality-specific attention mechanisms. Finally, we show that successful contrastive losses used in the self-supervised learning literature do not yield similar performance gains when used in multimodal transformers.

## multimodal pretraining significant progress in pretraining of natural language processing (nlp) models has been made through both architectural innovations (e.g., transformers; @xcite as well as a huge increase in the size of pretraining data and the model (e.g., @xcite . this success in language pretraining has inspired parallel multimodal vision-language efforts; in particular, multimodal image-language transformers, pretrained on large noisy imagetext datasets, have achieved state-of-the-art results on a range of downstream tasks such as image retrieval, visual question answering, and visual reasoning (e.g., @xcite @xcite . however, even though many variants of multimodal image-language transformer models have been proposed recently, it is unclear how learned representations are impacted by the large amounts of pretraining data, the transformer architecture and self-attention, or their specific losses. we address this gap, by first establishing a baseline that is trained on the same pretraining data as multimodal transformers but with a different architecture. we then perform an investigative analysis to better understand the extent to which these aspects contribute to models' performance. our evaluation mainly focuses on zero-shot tasks where evaluation data is taken from a dataset unseen during pretraining. measuring zero-shot performance enables us to evaluate whether a pretrained model learns general representations. previous work in nlp has considered probing classifiers to evaluate representations; however, this approach can be misleading as the performance of probing classifiers does not solely depend on the quality of representations (e.g., @xcite . similarly, evaluation after fine-tuning is a less direct measure of strength of representations since performance on these tasks is highly dependent on the fine-tuning experimental set-up and the size of fine-tuning data @xcite . we first study the importance of different properties of multimodal datasets such as their size and their noise level (i.e., how closely the language describes a given image's content). recent work has introduced image-text datasets with different qualities-for example, noisy but very large ones @xcite as well as carefully annotated but smaller ones @xcite . better understanding of what aspect of a dataset is more important can result in better task performance and also guide us in future dataset curation efforts. we find that a dataset's size does not always predict multimodal transformers' performance; its noise level and language similarity to the evaluation task are both important contributing factors. we also show that multimodal transformers can achieve competitive results without relying on language-only or image-only pretraining for weight initialization or feature extraction. we also dissect multimodal transformers' architecture, analyzing the effectiveness of different attention mechanisms, depth, and number of parameters. we show that multimodal attention, where both language and image transformers attend to each other, are crucial for these models' success. multimodal attention achieves the best results when combined with multi-level (deep) interactions. moreover, models with other types of attention (even with more depth or parameters) fail to achieve comparable results to shallower and smaller models with multimodal attention. additionally, inspired by the success of selfsupervised representation learning (e.g., @xcite , we examine whether using a contrastive image-text matching loss instead of a classification one improves the quality of representations in our models. surprisingly, we find that the choice of image-text matching loss does not matter much in multimodal transformers. on the other hand, models without multimodal attention (a multi-level ''cross-talk'' between modalities) benefit significantly from a contrastive loss. finally, we believe that advances in multimodal pretraining can have significant impacts on a wide range of downstream applications; however, it is important to form a clear understanding of how and why multimodal transformer models perform well to avoid overfitting to a set of downstream evaluation tasks. our analysis of pretraining data, attention, and loss functions is an important step towards gaining a deeper understanding of these powerful models. ## multimodal transformers the success of transformer-based language models on a variety of language tasks (e.g., @xcite has inspired similar multimodal efforts (e.g., @xcite @xcite ). 1 the main distinction is that image-text multimodal transformers take image-text pairs as input, attend over both modalities, and are trained with additional losses. similar to the language models, multimodal transformers are often fine-tuned on down-stream tasks but multimodal ones; e.g., image retrieval @xcite or visual question answering @xcite . we give a brief overview of the bert model @xcite , which forms the backbone of multimodal transformers. the bert architecture consists of a stack of transformer blocks @xcite and has three main components. first, the input text is tokenized and three embedding functions are used to embed the token, its position in the sentence (i.e., positional encoding), and the sentence it belongs to. the final language embedding is a summation of these three vectors. the bert model also includes @xmath0, which can be thought of as an aggregate representation of the input text. second, the sequence of token embeddings are input into a series of transformer layers where tokens are combined through self-attention. third, two different losses are applied to the model output: a masked language modeling loss, in which the model predicts a masked word (denoted by @xmath1, and a next sentence prediction loss which, given two sentences, predicts if the second sentence follows the first. multimodal transformer models facilitate learning from multimodal data via three changes to the bert architecture: multimodal data preprocessing (more specifically images), adding multimodal attention by changing self-attention such that it combines image and text modalities, and introducing image and multimodal loss functions. ## experimental setup here we outline the details of our experimental setup: the base multimodal transformer model used in most of our experiments, our baseline model, and the pretraining datasets. ## experimental results we first compare mmt to a baseline and then investigate how pretraining data, attention, and loss functions impact model performance. dataset # images zs ft train test flickr30k 29k 1k mscoco n/a 5k vqa 440k 210k ## related work multimodal transformers are the first family of multimodal models to be pretrained on large data and applied to a range of different language and vision tasks @xcite @xcite . the recent image-text transformers share the same backbone but have slight differences in data preprocessing and other architectural choices. notably, the uniter model @xcite achieves stateof-the-art results on most existing image-language benchmarks by using a larger dataset and a number of different loss functions. @xcite removes the need for using image features (taken from a pretrained object detector) by training models on raw images (pixels). to combine image and text modalities, lxmert @xcite and vilbert @xcite propose coattention mechanisms, similar to the coattention originally proposed for vqa @xcite . in vilbert, feed-forward layers are applied after the coattention and self-attention layers, whereas in lxmert, a feed-forward layer is only applied after the self-attention layer. a few of our findings are similar to observations in prior work: (i) lxmert and vilbert show that more layers improve results, (ii) vil-bert and uniter show that more data boosts performance, and (iii) lxmert shows that transferring bert weights is not beneficial. in contrast to uniter, we show that with the right hyper-parameters, the mrm loss is not needed. finally, while joint-space approaches to multimodal training are applied to multilingual data @xcite , all existing multimodal transformers are applied to english; an interesting future direction is to extend these models to other languages. ## discussion we rigorously examined different aspects of training multimodal transformers (datasets, attention, and losses) that contribute to the quality of their learned representations. we focused on zero-shot image retrieval tasks to evaluate learned representations. zero-shot tasks are advantageous because they directly measure what a model has learned and do not introduce confounds such as the size of a fine-tuning dataset and its experimental setup. at the same time, datasets do not always capture what they are designed to measure; e.g., @xcite show that models can do well on a referring expression task while ignoring the linguistic structure.thus, we argue that designing and curating specialized zero-shot evaluation tasks and datasets is an important future direction that will allow us to better understand our models' limitations. we find the quality of language and the degree to which the language describes its corresponding image (noisiness) plays an important role in our results. moreover, language-only and imageonly pretraining do not notably contribute to the performance of multimodal transformers. these suggest curating less noisy image-text datasets to be more important than relying on singlemodality datasets. previous work has successfully removed some of the noise in automatically harvested datasets through preprocessing (e.g., @xcite but such approaches are still limited in their robustness to noise, and the far from negligible degree of noise in large-scale real-world datasets (e.g., @xcite still poses a challenge. an alternative approach is to aim to remove this noise by designing models that better tap into statistical regularities of image-text pairs (e.g., @xcite and thus are more robust to noise. we show that multimodal attention-where each modality is informed by both modalities-is crucial in these models' performance. smaller models with multimodal attention outperform deeper models with no or other multi-head attention mechanisms. this suggests that we can potentially train smaller models (than the existing multimodal transformers) for a given task, especially when the pretraining data is chosen carefully. moreover, with multimodal attention, we can achieve the best zero-shot retrieval results using a classification loss which uses only one negative example per image-text pair (compare to a contrastive loss with 16384 negatives used in @xcite and also removes the need for mining more hard negatives @xcite . additionally, we observe that comparable results can be achieved without the image (masked region modeling) loss in multimodal transformers. this suggests that our current models are not tapping into the useful signal in the image modality, presumably because of the image loss formulation. an interesting future direction is designing better generative pretraining losses for images; previous work shows that the choice of loss significantly impacts the quality of language representations @xcite . finally, we believe that examining why and how multimodal transformers perform so well can guide future work in more effectively measuring progress in learning rich visual-linguistic features.