When does CLIP generalize better than unimodal models? When judging human-centric concepts (2022)

Abstract: CLIP, a vision-language network trained with a multimodal contrastive learning objective on a large dataset of images and captions, has demonstrated impressive zero-shot ability in various tasks. However, recent work showed that in comparison to unimodal (visual) networks, CLIP’s multimodal training does not benefit generalization (e.g. few-shot or transfer learning) for standard visual classification tasks such as object, street numbers or animal recognition. Here, we hypothesize that CLIP’s improved unimodal generalization abilities may be most prominent in domains that involve human-centric concepts (cultural, social, aesthetic, affective...); this is because CLIP’s training dataset is mainly composed of image annotations made by humans for other humans. To evaluate this, we use 3 tasks that require judging human-centric concepts”:” sentiment analysis on tweets, genre classification on books or movies. We introduce and publicly release a new multimodal dataset for movie genre classification. We compare CLIP’s visual stream against two visually trained networks and CLIP’s textual stream against two linguistically trained networks, as well as multimodal combinations of these networks. We show that CLIP generally outperforms other networks, whether using one or two modalities. We conclude that CLIP’s multimodal training is beneficial for both unimodal and multimodal tasks that require classification of human-centric concepts.

## introduction vision-language pretraining in neural networks is gaining popularity due to the growing interest in multimodal tasks such a visual question answering or image captioning @xcite @xcite , but also to the availability of online resources that allow to build large-scale training datasets without manual annotations @xcite . in theory, training a model on multimodal data should help improve its representation of data from each of the modalities. for an image-text model, for instance, the image features could be enriched by the abstraction of the linguistic data -the semantic grounding property, and inversely, the linguistic features could gain informativeness through visual grounding @xcite . unfortunately, this does not always happen in practice. recently, @xcite evaluated the visual generalization abilities of clip @xcite , a popular network trained with a contrastive learning objective on more than 400m image-caption pairs scraped from the web, and other multimodal models @xcite . they showed that for standard object classification tasks (e.g. digit, fashion item or natural image classification), multimodal networks like clip underperformed compared to other unimodal (vision-only) models like bit-m @xcite in transfer learning, few-shot learning and unsupervised learning settings. here, we revisit this question using datasets focusing on more "human-centric" concepts. human learning generally involves interacting with multimodal data. thus, one could expect that clip's representations of images and text should be somewhat closer to human representations than those learned by unimodal models. moreover, given that clip was trained on image-caption pairs from a variety of sources from the internet (including social networks), we can assume that an important part of its training captions was written by humans for other humans. this is different from standard vision datasets, in which labels or annotations are sometimes human-generated (e.g. through amazon's mechanical turk), but always produced for machine-learning purposes. again, this difference should bring clip's representations closer to human ones when compared to unimodal models. thus, there should exist at least some specific tasks for which clip's multimodal training provides advantages over unimodal models. as an example, consider the task of assigning a genre to a movie based on its poster and title. this requires retrieving fine-grained information about, among other things, the artistic, emotional or stylistic aspect of an image or a piece of text (or both). this can only be properly achieved if the model's training offered appropriate exposure to such human-centric concepts. here, we use the term human-centric whenever a concept refers to cultural, social, aesthetic and/or affective components of the world. we thus make the hypothesis that clip should perform better than unimodal models in generalization tasks where human-centric concepts are involved. we evaluate this hypothesis on three tasks involving such human-centric concepts: sentiment analysis on tweets; genre classification of books; genre classification of movies. all tasks can be performed based on visual data (images), text data (tweet, book or movie title, movie plot summary), or both. for the movie genre classification, we introduce a new, large-scale multimodal dataset obtained by a crawling on the movie database (tmdb). as detailed below, we find that clip outperforms unimodal models in both vision and text-based classification, as well as pairwise combinations of these unimodal models in the case of multimodal (image+text) classification. consequently, clip establishes a new sota on these tasks. we provide our code for reproducibility 1 . ## models we compare clip (trained contrastively on both images and text) against several unimodal models. for fairer comparisons, all the vision models are resnet50 @xcite based architectures and all the text models are transformer encoders. clip was trained using a contrastive loss on a large (400m) set of image-text pairs. the training of clip consists in creating a joint (multimodal) embedding space. for one batch of image-text pairs, the objective of the network is that the embedding of an image (through a resnet50 backbone, here simply referred to as clip) and the embedding of its text description (through a transformer backbone, here referred to as clip-t) are as close as possible, while the embedding of an image and 1 link not displayed here to preserve anonymity the embeddings of text descriptions of other images in the batch are as far as possible. after training, the text encoder and the image encoder can be used as single-modality encoders. for unimodally trained vision networks, we use two pretrained resnet50-based models: the standard resnet50 that was trained for classification on imagenet-1k (here referred to as rn50), and bit-m that was trained on imagenet-22k @xcite . for unimodal text embeddings, we test two standard text encoders against clip's: bert-large and bert-base @xcite . we use the bert sentence transformer version @xcite , based on bert's @xcite token and fine-tuned on snli @xcite and multinli @xcite . among the transformer encoders provided in the huggingface (hug) repository at the time our experiments were conducted, these were the two best-performing across several text classification tasks, and are now still close to sota. these versions of bert-large and bert-base are fine-tuned on downstream text classification tasks, but we refer to them in this paper simply as bert-large and bert-base. although all 3 text encoders are transformer encoders @xcite , they do not have the same number of parameters. bert-large has 300m, bert-base has 110m, and clip-t has 80m parameters. this gives a structural disadvantage to clip-t, which only strengthens our conclusions, as we found clip-t to be the overall best-performing text model. we consider both unimodal tasks (classification of images or text), as well as multimodal tasks (classification of image-text pairs). when performing a unimodal task, the encoding of the image (resp. the text) is used directly by the corresponding classifier. when performing a multimodal task (image-text based classification), the encoding of an image by a visual model and the encoding of the corresponding text by a textual model are simply concatenated to create the multimodal vector that is used for the classification. for bit-m and rn50, we use the last layer output before the classification head used for their training, which counts 2048 dimensions. for clip, we use the latent vector in the multimodal space generated by the visual pipeline, counting 1024 dimensions; for clip-t, the one generated by the textual pipeline (1024 dimensions); and for the two bert models, we use the vectors directly provided by the sentence transformer pipelines (1024dimensional for bert-large and 768-dimensional for bert-small). figure 1 : an original cover from the book cover dataset (left) and the associated masked cover (right). the title, the name of the author and parts of the text have been blacked out by the east algorithm, while the white text was incompletely detected, but subsequently blurred by the second algorithm. this sample belongs to the "children's books" genre. its title is: "frances audio collection cd (i can read level 2)". this image is copyright from amazon.com, inc. and used here for academic purpose only. ## datasets we evaluate the models on three datasets composed of labelled image and text data, that can be inputted as pairs for multimodal classification tasks, or used as single inputs for unimodal classification tasks. the language part of all these datasets is in english. ## results to compare the generalization capabilities of our text, vision, and multimodal models, we focus on transfer learning and few-shot learning settings. ## discussion and conclusion clip's generalization abilities were originally described in the context of zero-shot learning (radford et al., 2021), but they may also extend to other settings, including transfer learning and few-shot learning. past work has revealed that this is not always the case @xcite . considering the latent representations learned by clip may help us better understand when multimodal training does or does not benefit generalization abilities, continuing the work of @xcite . in our case, it appears that one of the domains where the improvement is most significant is when humancentric concepts are being judged. during their joint contrastive training, clip and clip-t have learned to extract common information between image and text modalities, so that the two streams would result in similar embedding vectors. this means that the representation of text in clip-t has been enriched with visual data, and symmetrically, that the representation of images in clip has been improved by semantic or linguistic enrichment. this is what is collectively referred to as the "semantic grounding" property @xcite . however, another consequence of this multimodal contrastive training is that when learning a common ground between modalities, some relevant information could be lost. for text, what cannot be directly linked to images (including grammatical or syntactic properties); and for images, what is not directly relevant to the text description (including fine-grained visual details that are rarely mentioned in the corresponding caption). this information loss might be the reason why clip was found to perform worse than standard vision-only models in a unimodal setting with standard visual datasets @xcite . for the same reason, one could actually expect that in a multimodal setting, the combination of clip's vision and text streams (clip+clip-t) could lead to worse performance than other combinations (e.g. rn50+bert). the unimodal networks are trained to capture the relevant features of their modality, and when combined, could cover the multimodal feature space more fully than clip, a network trained to discard information that is not redundant across modalities. our results show that, at least in our human-centric classification tasks, this limitation was not consequential: clip, clip-t and their combination often performed optimally. this may be because human-centric information is particularly well captured by features expressed in both images and text, rather than in each modality independently. on the other hand, this same reasoning could explain why clip+bert combinations performed slightly better than clip+clip-t on mvsa: bert may have provided additional information not captured by clip, which was lacking in clip-t because of their redundant embeddings (or, this might simply be due to the fact that bert has many more parameters than clip-t). our suggestion that clip (and clip-t) perform particularly well when judging human-centric concepts resonates with recent findings relating clip's representations to human brain representations. @xcite reported that some artificial neurons in clip's visual stream (but not in standard visual models like inception or resnet) are systematically activated by specific "concepts" such as a particular person, emotion, country, religion, etc. furthermore, these neurons could be equally activated by visual features (e.g., a photograph or drawing of the person's face) or by written text (e.g., the person's name). the authors related this multimodal invariance to properties of specific biological neurons found in the human hippocampus and temporal medial lobe, called "concept cells": these cells would also systematically activate when presented with a picture, drawing or written word representing a specific concept, such as a photograph of the actress jennifer aniston or her written name @xcite . indeed, more recently @xcite compared brain fmri representations in the human hippocampus with the patterns of representations measured in various vision models. they found that clip and other networks trained with multimodal objectives were more similar to human hippocampus representations than standard vision models (including rn50 and bit-m). this could explain why a multimodal network like clip performs better when judging "human-centric concepts". to conclude, we think that it is crucial to investigate the specific domains in which a multimodal training such as clip's can (or cannot) improve generalization. our work indicates that multimodality will be key for developing algorithms designed for human-centric tasks (even for unimodal tasks) such as detecting emotions, analyzing personality, conducting a conversation or, more generally, when human-machine interactions are involved.