Visual Commonsense in Pretrained Unimodal and Multimodal Models (2022)

Abstract: Our commonsense knowledge about objects includes their typical visual attributes; we know that bananas are typically yellow or green, and not purple. Text and image corpora, being subject to reporting bias, represent this world-knowledge to varying degrees of faithfulness. In this paper, we investigate to what degree unimodal (language-only) and multimodal (image and language) models capture a broad range of visually salient attributes. To that end, we create the Visual Commonsense Tests (ViComTe) dataset covering 5 property types (color, shape, material, size, and visual co-occurrence) for over 5000 subjects. We validate this dataset by showing that our grounded color data correlates much better than ungrounded text-only data with crowdsourced color judgments provided by Paik et al. (2021). We then use our dataset to evaluate pretrained unimodal models and multimodal models. Our results indicate that multimodal models better reconstruct attribute distributions, but are still subject to reporting bias. Moreover, increasing model size does not enhance performance, suggesting that the key to visual commonsense lies in the data.

## introduction the observation that human language understanding happens in a rich multimodal environment has led to an increased focus on visual grounding in natural language processing (nlp) @xcite , driving comparisons between traditional unimodal text-only models and multimodal models which take both text and image inputs. in this work, we explore to what extent unimodal and multimodal models are able to capture commonsense visual concepts across five types of relations: color, shape, material, size, and visual cooccurrence (cf. fig. 1 ). we further explore how this ability is influenced by reporting bias @xcite , the tendency of large corpora to over-or under-report events. we define visual commonsense as knowledge about generic visual concepts, e.g. "knobs are usually round", and we measure this knowledge via frequency distributions over potential properties (e.g. round, square, etc). a visually-informed language model should be able to capture such properties. our color, shape, material, and co-occurrence data are mined from visual genome @xcite , and our size data are created from object lists. they contain a large number of examples of per-object attribute distributions and "object-attribute" pairs. @xcite evaluate language models' color perception using a human-annotated color dataset (coda), finding that reporting bias negatively influences model performance and that multimodal training can mitigate those effects. in this work, we confirm those findings while extending the evaluation to a broader range of visually salient properties, resulting in a more comprehensive metric for visual commonsense. in order to elicit visual commonsense from language models, we utilize soft prompt tuning @xcite , which trains optimal templates by gradient descent for each model and relation type that we explore. we also utilize knowledge distillation to enhance a textonly model's visual commonsense ability, where the vision-language model serves as the teacher. the major contributions of this work are: (1) we design a comprehensive analytic dataset, vi-comte, for probing english visual commonsense, that is applicable to any language model; (2) we use vicomte to study models' ability to capture empirical distributions of visually salient properties. we examine unimodal language models, multimodal vision-language (vl) models, and a knowledgedistilled version of a vl model; and (3) we analyze the effects of reporting bias on the visuallygrounded vs. ungrounded datasets and models. ## results the experimental results show that multimodal models outperform text-only models, suggesting their advantage in capturing visual commonsense. however, all models are subject to the influence of reporting bias, as they correlate better with the distributions from wikipedia than those from coda ## conclusion in this paper, we probe knowledge about visually salient properties from pretrained neural networks. we automatically extract dataset of five visual relations: color, shape, material, size, and cooccurrence, and show that our vicomte dataset has a much higher correlation with human perception data for color than data mined from wikipedia. we then apply several probing techniques and discover that visually-supervised models perform better than pure language models, which indicates that they can better capture such visual properties. distilling the knowledge from a visually-supervised model into a pure language model results in comparable performance with the teacher model. we also observe less reporting bias in both visually-grounded text (vg-mined datasets) than wikipedia text and visually-grounded models (oscar, distilledoscar, visualbert, and clip) than pure language models. however, visuallygrounded models are still subject to the influence of reporting bias, as seen in the per-group analysis, where both types of models perform better for the multi group than the single group. per-object analysis fig. 4 illustrates the finegrained spearman @xmath0. size per-object fig. 5 shows how the per-object projection scores on the size spectrum from bert and oscar are correlated. per-subject comparison fig. 6 and fig. 7 show how the spearman correlations of 10 individual subjects improve after soft prompt tuning and after multimodal pretraining. consistent improvement can be seen in color, material, and cooccurrence. although we report average spearman correlations in with shape, the improvement is less obvious (45.9 to 50.4 for prompt tuning and 49.2 to 50.4 for multimodal pretraining).