Localization vs. Semantics: Visual Representations in Unimodal and Multimodal Models (2024)

Abstract: Despite the impressive advancements achieved through vision-and-language pretraining, it remains unclear whether multi-modal learning can help understand each individual modality. In this work, we conduct a comparative analysis of the visual representations in existing vision-and-language models and vision-only models by probing on a broad range of tasks. Five probing tasks are evaluated in order to assess the quality of the learned representations in a nuanced manner. Our results on five probing tasks suggest vision-and-language models are better at label prediction tasks like object and attribute prediction, while vision-only models are stronger at dense prediction tasks that require more localized information. We hope our study sheds light on the role of language in visual learning, and serves as an empirical guide for various pretrained models.

## introduction the joint learning of vision and language offers mutual benefits. as evident by the recent advancements in vision-and-language pretraining (vlp) models @xcite @xcite , they attain not only impressive performance on multi-modal tasks like visual question answering, but also on specialized uni-modal vision tasks like imagenet classification @xcite , or language tasks glue language understanding @xcite . despite the superior performance, there is little understanding of how multimodal learning can help visual representations. therefore, we hereby are motivated to compare the visual representations in existing vision-and-language (vl) models and vision-only (v) models from a probing perspective. specifically, we probe the visual representations through a range of probing tasks that evaluate different properties, including semantics knowledge and localized information, in order to gain a finegrained understanding of the visual representations. this is inspired by recent works on multimodal feature probing @xcite , which studies the opposite question to ours, i.e., the role of vision in language models. fig. 1 illustrates our probing pipeline. we first extract image features using different pretrained models, and then train a simple prediction head to align the model's representation space with the label space of interest. we make the head as simple as possible based on the intuition that less expressive heads can more selectively reflect the quality of the representations @xcite . the probing is done on various tasks and datasets: object name classification on the visual genome dataset @xcite , attribute prediction on the vaw dataset @xcite , object detection and instance segmentation on the mscoco dataset @xcite , and semantic object part segmentation on the partimagenet dataset @xcite . with these probing tasks, we compare vision-and-language pretrained models including ofa @xcite , flava @xcite and clip @xcite with advanced vision-only models including mae @xcite and mocov3 @xcite . interestingly, our experiments suggest that vl models are much better at the label prediction tasks (e.g., object class and attribute prediction), while vision-only models are stronger at dense prediction tasks like object detection and segmentation. in other words, multimodal models encode more semantic information in visual representations to better predict fine-grained labels, but fail to enrich the localization information that is required by spatial-aware tasks. this finding is further verified by a more detailed analysis of the segmentation and attribute prediction results, which reveals intriguing properties of the unimodal and multimodal representations. how good are the features for … in summary, we probe the visual representations in popular vl and vision-only pretrained models on a broad spectrum of tasks and suggest that multimodal representations encode better semantics. we hope our extensive probing results can serve as a fine-grained benchmark for the publicly released pretrained models, which provides an empirical guide to help researchers choose which model to use for different downstream tasks. moreover, by offering these insights into the role of language in multi-modal learning, we hope to catalyze future explorations in this direction. ## related work vision-and-language pretraining (vlp). vlp methods perform well on multi-modal downstream tasks like visual question answering @xcite and image captioning @xcite and show potential on single-modal tasks. for example, dual encoders trained with a contrastive loss like clip @xcite and align @xcite achieve superior visual learning performance. while earlier vlp methods (like lxmert @xcite , uniter @xcite , oscar @xcite , vinvl @xcite ) rely on image features extracted by separately trained vision models like faster-rcnn @xcite or resnet @xcite , more recent works learn the visual features jointly with language. representative works include ofa @xcite , florence @xcite , flava @xcite , unified-io @xcite , coca @xcite , and simvlm @xcite etc. we refer readers to @xcite for more details. vision and language benefit each other. several recent works in nlp suggest that multimodal learning can help language understanding. vokenization @xcite suggests vision improves the grounding ability of language models. @xcite shows reduced reporting bias in multimodal world. z-lavi @xcite and vidlankd @xcite show language understanding performance can be improved by better visual imagination or knowledge distillation from videos. recent work @xcite analyzes language and multi-modal models and shows that vision can help language models learn better commonsense knowledge and mitigate reporting bias. however, there is little understanding of the opposite question, i.e. how does the visual learning differ in multimodal and unimodal models. probing. probing is a widely used strategy in nlp for interpreting representations @xcite . various works use probing to show that language representations encode a broad range of properties like part-of-speech @xcite , syntax @xcite , semantics @xcite , sentence length @xcite , etc., and to compare different language models in those properties @xcite . probing has also been adopted to understand multimodal representations in terms of the capacity for instance retrieval @xcite , inter-modality knowledge @xcite , understanding of verbs @xcite , entity and syntactic grounding @xcite , and visual commonsense knowledge @xcite , etc. with probing, multi-modal vl models are compared with uni-modal language models to assess the advantage of multi-modal learning. however, probing has not been widely explored for visual representations, despite as a fast on-the-fly metric for model evaluation @xcite @xcite complementary to fine-tuning. to our knowledge, we are the first to compare vl models and vision-only models using probing. ## method to analyze the capacity of the learned representations of different models, we choose a set of tasks to probe the models. for each task, we first extract features using the pretrained models, then we train a simple standard head to predict the results. mathematically, for every image @xmath0, we extract its features @xmath1. here (w, h) is the size of the input image and (c, w, h) is the size of the feature. then a prediction head p is trained to predict the task-specific results based on feature f . in the whole process, only the head p is trained while the pretrained model (i.e., feature extractor) is frozen. in this section, we will first describe the probing tasks, datasets and the prediction head for each task (sec. 3.1), then we describe the evaluated models (sec. 3.2), and finally how to make the comparison settings fair for every model (sec. 3.3). ## conclusion this work compares the visual representations in multimodal and unimodal models by feature probing. by comparing three representative vl models and two v models on five probing tasks, we find that vl models are stronger in label prediction tasks, while vision-only models are better in dense prediction tasks. we hope our diagnostic findings serve as an empirical guidance for future works in choosing models for different downstream tasks, as well as exploring the role of language in visual representation learning.