One does not fit all! On the Complementarity of Vision Encoders for Vision and Language Tasks (2023)

Abstract: Current multimodal models, aimed at solving Vision and Language (V+L) tasks, predominantly repurpose Vision Encoders (VE) as feature extractors. While many VEs—of different architectures, trained on different data and objectives—are publicly available, they are not designed for the downstream V+L tasks. Nonetheless, most current work assumes that a single pre-trained VE can serve as a general-purpose encoder. In this work, we focus on analysis and aim to understand whether the information stored within different VEs is complementary, i.e. if providing the model with features from multiple VEs can improve the performance on a target task, and how they are combined. We exhaustively experiment with three popular VEs on six downstream V+L tasks and analyze the attention and VE-dropout patterns. Our analyses suggest that diverse VEs complement each other, resulting in improved downstream V+L task performance, where the improvements are not due to simple ensemble effects (i.e. the performance does not always improve when increasing the number of encoders). We demonstrate that future VEs, which are not repurposed , but explicitly designed for V+L tasks, have the potential of improving performance on the target V+L tasks.

## introduction the dominant strategy for solving vi-sion+language (v+l) tasks involves using transformer models @xcite that jointly attend over the representations of the respective modalities @xcite @xcite @xcite . while representationlearning of the text modality is comparatively straightforward using token embeddings, 1 image representations are more difficult to learn. given an image, a common approach is to use pre-trained vision encoders (ve), where the ve's output features are passed as inputs, together with the text embeddings, into a transformer model. the attention mechanism then learns a cross-modal representation space over the text and image features to solve the target v+l task. consequently, the success of a multimodal model builds heavily on the features extracted from a ve and is thus highly dependent on the ve's architecture, training objectives (e.g. image classification, image encoding, object detection, etc.), and pre-training data. this dependency is further exacerbated for multimodal models that utilize ves as static feature extractors (i.e. the weights of the ve are frozen), but also for models that are trained end-to-end, as the biases introduced by the architecture, objectives, and data of the ve remain. since many computer vision models can be repurposed as ves for v+l tasks, a few prior works have focused on identifying individual ves that perform the best on downstream tasks @xcite @xcite . a common assumption is that a single pre-trained ve can perform the best for a target task or even serve as a general-purpose encoder for a wide range of v+l tasks. however, a natural question arises: to what extent is this assumption correct? given that all ves differ in architecture, objectives, and pre-training data, we hypothesize that the extracted features of multiple different ves encode complementary information. in this work, we focus on answering: 1) do different ves encode complementary features? 2) how are features from different ves utilized by transformers? we provide comprehensive analyses for multi-ve models and test whether combining ves is beneficial over a single-ve setup under the viewpoint of feature complementarity. similar to prior work that analyzed other components of v+l transformers @xcite , we will not focus on improving the performance through ensembling like @xcite . rather, we utilize combinations of ves as the setting for answering our research questions. we cover three popular classes of ves in our experiments: 1) object detection models providing a feature representation of salient image parts containing objects (region) @xcite , 2) cnn models computing a feature map of the image for grid features (grid), and 3) vision transformers (vit) @xcite computing contextualized patch features of the image (patch). as the downstream domain and task type can be heavily impacted by the different ves, we probe all combinations of the three ves on six different v+l tasks, covering retrieval, q&a, and reasoning. to investigate the ve complementarity and feature utilization, we analyze 1) the attention patterns across modalities and ves, and 2) the dependency of specific ves when performing ve-dropout during training and inference. while multi-ve seems to perform better than single-ve (which could partially attribute to the increased parameter count), we consistently observe performance gaps between different multi-ve configurations (e.g. a gap as large as 8.9 points for the same task) and no single winning combination for all task types. our attention patterns analysis across the different ves reveals that the distinctive information encoded in the ves is important for different tasks, and the model composes the representations by enriching a dominant ve with complementary information of the other ves. to sum up, our results and analysis suggest that ves trained on different objectives, architectures, and data can have a high impact on the model's v+l task performance. we cannot rely on simple ensemble effects to improve performance; selecting and repurposing off-the-shelf ves is non-trivial, which emphasizes the necessity to design ves explicitly for v+l tasks in the future. ## related work multimodal transformer architectures. multimodal transformer architectures can be divided into single-stream and dual-stream models @xcite . the single-stream transformer takes the concatenated visual and text tokens as input and processes them modalityagnostic, i.e. the self-attention jointly attends over the tokens of both modalities. dual-stream models use separate transformers for each modality that are connected through a co-attention mechanism @xcite , concatenated in a single-stream model on top @xcite , or the image model output is used asymmetrically for cross-attention in the text model @xcite . the faster r-cnn (ren et al., 2015) object detector has been the dominant choice for multimodal models as a region ve, where most methods propose to use it as a static feature extractor @xcite @xcite @xcite @xcite , with the notable exception being @xcite who backpropagate through the faster r-cnn model. less popular ves are grid @xcite @xcite @xcite , and patch @xcite @xcite . in contrast to region ves, grid and patch ves are commonly fine-tuned on the target v+l task, with the notable exception being @xcite . following @xcite ; hendricks et al. @xcite we focus on single-stream models as they have been shown to perform on par with dual-stream models while being easier to extend to multi-ve setups. comparing and combining ves. recently, several works aim to compare different ves for v+l tasks. @xcite compare region and grid for visual qa tasks, showing that training data, objectives and other factors all affect the downstream task performance. @xcite ; @xcite compare different pre-trained grid and patch ves building on clip @xcite . @xcite compare different design choices for region ves with grid ves trained on the same data. @xcite compares different ves in influence object hallucinations in caption generation. closest to our work is the work by @xcite . while they also experiment with combining representations of grid-, patch-, and region ves, they only focus on the visual question answering (vqa; @xcite dataset and only use the combination of all three ves. our work provides a more in-depth evaluation of different multi-ve setups while experimenting with six diverse tasks, and shows that different combinations work best for each task. analysis of multimodal transformers. our anal-ysis methods draw inspiration from recent works that probe and analyze pre-trained multimodal transformers for a better understanding of their different components @xcite @xcite @xcite . cao et al. (2020) propose a range of different probing tasks to understand the inner workings of multimodal models. li et al. (2020a) analyze how accurate the attention heads of pre-trained models can perform visual grounding. frank et al. (2021) mask parts of the text and image input and measure how the prediction performance changes for the respective other modality to test how symmetric the learned cross-modal connection is. bugliarello et al. (2021); hendricks et al. (2021) evaluate and disentangle which components of multimodal pre-training proposed in different works are important for their success. while previous work has only focused on models with a region ve, we also experiment with grid and patch ves. in summary, our work is the first in-depth study of multimodal transformers that use multiple ves. ## multimodal multi-ve transformers recently, cross-modal attention is the dominant strategy to learn multimodal representations with v+l transformers. in this work, we follow @xcite and focus on the singlestream architecture, which shares the attention components across all modalities, i.e. the concatenated visual and text tokens are processed modalityagnostic. this architecture achieves state-of-the-art results @xcite while being easily extendable to multiple ves, by concatenating all vision tokens. 2 figure 1 illustrates our architecture. ## experiments we evaluate the impact of three different ves on six downstream v+l tasks to assess the complementarity of different image representations. here, we experiment with all possible combinations of the three ves (i.e. single ve, 2-ve, and 3-ve setups). 3 to fairly compare the information stored in the respective ves, we only fine-tune the multimodal models on the target v+l task in order to circumvent potentially beneficial information leaking into the multimodal model from auxiliary tasks. we therefore initialize all models with bert weights @xcite ). 2 : web-crawled & cleaned image-caption pairs (proprietary). @xcite (base-size). we note, however, that gains can be achieved when pre-training the multimodal model on auxiliary data prior to fine-tuning on the target v+l task @xcite @xcite , inter alia). ## analysis to better understand how the representations are combined in different multi-ve setups, we analyze the flow of attention, phrase-to-image grounding, and the robustness to dropping ves at test time. we overload 'cross-modality' to include both ve-text but also ve-ve interactions for simplicity. we present the analysis for the best performing model combinations in what follows but provide a full list of results in appendix b. ## conclusion in this work, we investigated whether different ves-based on repurposed pre-trained vision models-encode complementary information, which improves the performance on downstream v+l tasks. we experimented with three popular ve classes region, grid, and patch, and trained models with all possible combinations on six different v+l tasks. while combining ves improve over single-ve setups, our further analysis of attention patterns reveals that diverse ves encode complementary information, which motivates future work on designing ves explicitly for v+l tasks-e.g. by incorporating more diverse datasets, and training objectives.