Does language help generalization in vision models? (2021)

Abstract: Vision models trained on multimodal datasets can benefit from the wide availability of large image-caption datasets. A recent model (CLIP) was found to generalize well in zero-shot and transfer learning settings. This could imply that linguistic or “semantic grounding” confers additional generalization abilities to the visual feature space. Here, we systematically evaluate various multimodal architectures and vision-only models in terms of unsupervised clustering, few-shot learning, transfer learning and adversarial robustness. In each setting, multimodal training produced no additional generalization capability compared to standard supervised visual training. We conclude that work is still required for semantic grounding to help improve vision models.

## introduction learning vision models using language supervision has gained popularity @xcite @xcite @xcite @xcite @xcite for two main reasons: firstly, vision-language training allows to build massive training datasets from readily available online data, without manual annotation; secondly, language provides additional semantic information that cannot be inferred from visiononly datasets, and this could help with semantic grounding of visual features. recently @xcite introduced clip, a language and vision model that shows outstanding zero-shot learning capabilities on many tasks, and compelling transfer-learning abilities. a recent report @xcite showed that clip produces neural selectivity patterns comparable to "multimodal" concept cells observed in the human brain @xcite . from these results, it is tempting to assume figure 1 : size of the training dataset used by the models (number of images, in log scale). icmlm and virtex are trained on coco, tsm on howto100m, clip on a (not publicly available) scrape of the internet, rn50 is trained on imagenet-1k, the ar models and sin models are trained on augmented versions of imagenet-1k and bit-m is trained on imagenet-21k. that clip's generalization properties stem from semantic grounding provided by the joint visionlanguage training. here, we show that clip and other visionlanguage models do not perform better than visiononly, fully supervised models on a number of generalization settings and datasets. representation similarity @xcite analysis reveals that the multimodal representations that emerge through vision-language training are different from both linguistic and visual representations-and thus possibly unsuitable for transfer-learning to new visual tasks. in conclusion, additional work on linguistic grounding is still needed, if it is to improve generalization capabilities of vision models. we provide our code for reproducibility foot_0 . ## models we use a number of publicly available vision, text or multimodal pretrained models, and compare their representations and generalization abilities. to facilitate interpretation and comparisons between the models, figure 1 reports the training dataset size for each of the visual models (including the vision-language models). they are all based on the same backbone (a resnet50 architecture). in clip, the authors train the joint embedding space of a visual network (hereafter called simply clip) and a language network (hereafter called clip-t) using contrastive learning on 400m imagecaption pairs. note that in the present paper, the visual backbone of clip is a resnet50, even though the visual-transformer-based clip model could reach higher performance; this choice allows for a fair comparison with the other visual models that are all based on the resnet50 architecture. in addition, we also consider tsm @xcite , another multimodal network trained with a contrastive loss on video, audio and text inputs from the howto100m dataset @xcite (containing more than 136m video clips with captions. for training, the authors effectively used 120m video clips of 3.2s sampled at 10 fps). the effects of clip's and tsm's contrastive training paradigm can be compared with virtex and icmlm-two other recent multimodal networks. in virtex, the visual feature representations are optimized for an image captioning task @xcite , and for a text-unmasking task in icmlm @xcite . such text-based objectives aim to provide a form of linguistic grounding using significantly fewer images than clip (virtex and icmlm models are trained on the coco dataset @xcite with approximately 120k captioned images). to understand the potential effects of linguistic training, we compare the multimodal networks to vision-only networks. we include a baseline architecture (resnet50) trained on imagenet-1k @xcite (1.3m labelled images). second, we consider a similar architecture (resnet50 backbone) called bit-m @xcite , trained on imagenet-21k, a much larger dataset (14m labelled images). while generalization and robustness properties can often be derived from access to large labelled image datasets (as in bit-m), obtaining such labels is costly. an alternative is to train models with additional datapoints based on assumptions about the real-life data distribution-as done, e.g., with adversarial training. in this study, we use the adversarially robust (ar) resnet50 models pro-vided by @xcite , trained on the 1.3m imagenet training set plus 110 adversarial attacks of each image (i.e. more than 140m images overall). the different model variants (ar-l2, ar-li4, ar-li8) correspond to distinct adversarial attacks (refer to @xcite for more details). this adversarial training was found to produce more perceptually aligned features and to improve generalization (e.g. transfer learning) in some settings @xcite . another such technique was used for stylizedimagenet (sin) models @xcite , where a variant of the imagenet dataset (1.3m images) was designed via style-transfer to specifically reduce the network's reliance on texture information. the authors provide weights for models that are (i) only pretrained for sin images (sin), (ii) trained on sin and ima-genet (sin+in) combined, or where (iii) a sin+in model is finetuned on imagenet (sin+in-fin). for the vanilla resnet50, sin, ar and bit-m models, we use activations after the final average pooling operation as feature representations. although all these models share a resnet50 backbone, there are minor differences in their implementations. we assume that such small architectural differences would not dramatically affect the feature spaces learned by these models. finally, we also use two text-only language models, gpt-2 @xcite and bert @xcite , in our feature-space comparisons. as these models are not designed to process visual inputs, they cannot be tested on visual generalization; but we can use their representations of class labels (or sentence captions) as a basis for comparison with visual or multimodal network representations. in a similar way, the language stream of the clip model (clip-t) can be treated as a third language model for our comparisons. ## generalization tasks in @xcite , clip was systematically tested in a zero-shot setting. however, this requires a language stream to describe the different possible targets, which is not available in standard vision models. to compare the generalization capabilities of multimodal and vision-only models, we thus focus on few-shot, transfer and unsupervised learning. in each case, we evaluate performance on @xcite , cifar10, cifar100 @xcite , fashion-mnist @xcite , @xcite , @xcite 2 . these datasets test generalization capabilities for natural images of various classes. ## model comparison to better understand the similarities and differences between the feature spaces learned by the various models, we now compare them using representational similarity analysis (rsa) @xcite . method rsa is a comparison method originally used to compare fmri data. it allows us to compare different models (with different latent space dimensions, norms, ...) which share the same structure. this works by comparing the models' representational dissimilarity matrix (rdm). rdms are obtained by computing the 2 by 2 distances for each class of the latent representations (see figure 7 ). more specifically, for each visual model, we define for each class the set f c containing the feature vectors of all the images of class c, its average fc and its standard deviation σ c . the rdm matrix is then defined as [rdm i,j ] where we use the norm of the unequal variance t-test @xcite as our distance metric between the latent representations, because it allows to normalize the distances between class centroids with respect to their variances. indeed, each class is represented by a cluster of latent vectors of different sizes. in the case of language models (all transformerbased), we use as latent representations, the encoding of the sentence "a photo of x." where we replace "x" by the corresponding label. we then use the contextualization of the label as the text feature vector. compared to the vision models, there is only one representation per class (only one sentence per class) hence a lack of variance associated with the feature vector of each class. as a result, the distance used in the rdm matrix becomes an l 2 norm. the rdm matrix obtained with this method contains the respective distances between pre-defined concepts (in our case the 1000 classes of imagenet). rdms can therefore be considered as a standardized representation of latent spaces. this means that we can compare our models' representations by computing the pearson correlation between their respective rdms. the corresponding comparison matrix, for all pairs of models, is illustrated in fig 8 . results figure 9 shows the results of a hierarchical clustering (a) or t-sne (van der @xcite embedding (b) of the rdms using pearson correlation as a distance. looking at the dendrogram, all the vision-only models are very close to one another with a maximum @xmath0.2. then, multimodal models stand a bit further (clip, tsm, virtex, icmlm); and finally, clip-t and the language models (bert, gpt2) are the furthest away. this indicates that the language supervision (contrastive embedding, text-generation or text-unmasking objectives) has changed the structure of the resnet latent space for clip, tsm, virtex and icmlm models (respectively). yet these multimodal models are not truly linguistic either, as they are very distant also from the standard language models. this conclusion is also supported by the t-sne plot, showing a cluster of bit-m, rn50 and sin vision models, a second cluster with the ar models, and further along the same direction, the multimodal networks (clip, virtex, icmlm, tsm). note that, although this arrangement might suggest that multimodal networks possess adversarial robustness properties in common with ar models, this suggestion was not supported by our tests using actual adversarial attacks (fig 6 ). finally, the language models (bert, gpt2 and clip-t) are separated from the rest, along a distinct direction. overall, the analysis suggests that multimodal representations are neither visual nor linguistic, but surprisingly, not really in-between either foot_1 . this is surprising as we should expect that representations trained with access to both vision and language would derive information from both modalities, and consequently end up somewhere in-between purely visual and purely textual representations. ## performance on linguistic tasks this suggestion might be further supported by evaluating the usefulness of the learned visual representations on linguistic tasks. according to the above findings, visual representations obtained via multimodal training may fare no better than visiononly representations. to test this, for each vision model, we collect the imagenet features for each image class, and train a standard word embedding (skip-gram method) while constraining the class label words to these visual feature vectors. the resulting linguistic space will thus capture some of the structure of the vision model's latent space. ## discussion and conclusion it is a highly appealing notion that semantic grounding could improve vision models, by introducing meaningful linguistic structure into their latent space, and thereby increasing their robustness and generalization properties. unfortunately, our experiments reveal that current vision-language training methods do not achieve this objective: the resulting multimodal networks are not better than vision-only models, neither for few-shot learning, transfer learning or unsupervised clustering, nor for adversarial robustness. in addition, compared to vision-only models, the multimodal networks' visual representations do not appear to provide additional semantic information that could serve as a useful constraint for a word-embedding linguistic space. the present inability of linguistic grounding methods to deliver their full promise does not imply that this cannot happen in the future. in fact, we believe that exploring the current models' performance and representations, as we do here, can help us understand their limitations and adjust our methods accordingly. specifically, we found that multimodal representations are neither visual nor linguistic, but are not really in-between either (fig 9 ). in clip and tsm, for instance, the contrastive learning objective encourages the visual and language streams to agree on a joint embedding of images and corresponding captions. however, such agreement, by itself, does not constrain either latent space to remain faithful to its initial domain. as a result, clip's (and tsm's) visual representations may discard information that could prove critical for transfer-learning to other visual tasks. if this is true, we predict that adding domain-specific terms to the multimodal loss function (e.g. selfsupervision) could be a way to improve visual generalization, while retaining the advantages of multimodal training-possibly including semantic grounding.