Visual Commonsense in Pretrained Unimodal and Multimodal Models
Abstract
Our commonsense knowledge about objects includes their typical visual attributes; we know that bananas are typically yellow or green, and not purple.
Text and image corpora, being subject to reporting bias, represent this worldknowledge to varying degrees of faithfulness.
In this paper, we investigate to what degree unimodal (language-only) and multimodal (image and language) models capture a broad range of visually salient attributes.
To that end, we create the Visual Commonsense Tests (ViComTe) dataset covering 5 property types (color, shape, material, size, and visual co-occurrence) for over 5000 subjects.
We validate this dataset by showing that our grounded color data correlates much better than ungrounded text-only data with crowdsourced color judgments provided by Paik et al.
We then use our dataset to evaluate pretrained unimodal models and multimodal models.
Our results indicate that multimodal models better reconstruct attribute distributions, but are still subject to reporting bias.
Moreover, increasing model size does not enhance performance, suggesting that the key to visual commonsense lies in the data.
Introduction
The observation that human language understanding happens in a rich multimodal environment has led to an increased focus on visual grounding in natural language processing (NLP) (Baltrusaitis et al., 2019; Bisk et al., 2020), driving comparisons between traditional unimodal text-only models and multimodal models which take both text and image inputs.
In this work, we explore to what extent unimodal and multimodal models are able to capture commonsense visual concepts across five types of relations: color, shape, material, size, and visual cooccurrence (cf. Fig. 1).
We further explore how this ability is influenced by reporting bias (Gordon and Van Durme, 2013), the tendency of large corpora to over- or under-report events.
We define visual commonsense as knowledge about generic visual concepts, e.g. “knobs are usually round”, and we measure this knowledge via frequency distributions over potential properties (e.g. round, square, etc).
A visually-informed language model should be able to capture such properties.
Our color, shape, material, and co-occurrence data are mined from Visual Genome (Krishna et al., 2016), and our size data are created from object lists.
They contain a large number of examples of per-object attribute distributions and “object-attribute” pairs.
Paik et al.
(2021) evaluate language models’ color perception using a human-annotated color dataset (CoDa), finding that reporting bias negatively influences model performance and that multimodal training can mitigate those effects.
In this work, we confirm those findings while extending the evaluation to a broader range of visually salient properties, resulting in a more comprehensive metric for visual commonsense.
In order to elicit visual commonsense from language models, we utilize soft prompt tuning (Qin and Eisner, 2021), which trains optimal templates by gradient descent for each model and relation type that we explore.
We also utilize knowledge distillation to enhance a textonly model’s visual commonsense ability, where the vision-language model serves as the teacher.
The major contributions of this work are: (1) we design a comprehensive analytic dataset, ViComTe, for probing English visual commonsense, that is applicable to any language model; (2) we use ViComTe to study models’ ability to capture empirical distributions of visually salient properties.
We examine unimodal language models, multimodal vision-language (VL) models, and a knowledgedistilled version of a VL model; and (3) we analyze the effects of reporting bias on the visuallygrounded vs. ungrounded datasets and models.
Does the model know …
It is larger than:
It is smaller than:
Unimodal
Multimodal
BERT, …
Oscar, …
A girl is looking at the penguin.
Penguins are a group of aquatic flightless birds.
The word penguin first appears in the 16th century as a name for the great auk.
A plastic penguin is sitting on a chair.
VS.
what is the shape of a penguin?
what are the co-occurring objects of a penguin?
what is the size of a penguin?
ice beak grass table water stone what is the material of a penguin?
what is the color of a penguin?
Related Work 2. Vision-Language Modeling
Recent advances in vision-language (VL) modeling have led to increased success on benchmark tasks.
Most VL models learn joint image and text representations from cross-modal training of transformers with self-attention, including LXMERT (Tan and Bansal, 2019), ViLBERT (Lu et al., 2019),
VisualBERT (Li et al., 2019), UNITER (Chen et al., 2020), etc.
Oscar (Li et al., 2020) additionally uses object tags in images as anchor points to facilitate the learning of image-text alignments and VinVL (Zhang et al., 2021) presents an improved object detection model.
CLIP (Radford et al., 2021) learns by predicting caption-image alignment from a large internet corpus of (image, text) pairs.
While our work uses textual prompt tuning techniques, there have also been work on visual prompt engineering to enhance the performance of pretrained vision-language models.
Zhou et al.
(2021) model context in prompts as continuous representations and learn to optimize that context.
Yao et al.
(2021) develop a cross-modal prompt tuning framework that reformulates visual grounding as a fill-in-the-blank problem for both image and text.
2. Visual Commonsense
In one of the early attempts at learning visual commonsense, Vedantam et al.
(2015) measure the plausibility of a commonsense assertion in the form of (obj1, relation, obj2) based on its similarity to known plausible assertions, using both visual scenes and accompanying text.
Zellers et al.
(2021) learn physical commonsense via interaction, and use this knowledge to ground language.
Frank et al.
(2021) probe whether VL models have learned to construct cross-modal representations from both modalities via cross-modal input ablation.
Note that our definition of visual commonsense differs from that of Zellers et al.
(2019), where the model is required to perform commonsense reasoning based on an image.
Our definition of visual commonsense is more similar to the idea of stereotypic tacit assumptions (Prince, 1978) – the propositional beliefs that humans hold about generic concepts, such as “dogs have to be walked”.
Weir et al.
(2020) probe neural language models for such human tacit assumptions and demonstrate the models’ success.
We extend this intuition to visual concepts and explore how visual information may help language models to capture such assumptions.
There has also been earlier work on the McRae feature norms (McRae et al., 2005), in which human annotators wrote down attributes that describe the meaning of words.
For instance, “car” can be labeled as “has four wheels” and “apple” can be labeled as “is green”.
Silberer et al.
(2013) expand the McRae dataset into a set of images and their visual attributes and construct visually grounded distributional models that can represent image features with visual attributes.
Zhu et al.
(2020) examine the “language prior” problem in Visual Question Answering models, where models tend to answer based on word frequencies in the data, ignoring the image contents.
In this work, we explore to what extent such a language prior is recruited absent a visual input.
2. Reporting Bias
Pretrained language models such as BERT (Devlin et al., 2019) are trained on billions of tokens of text, capturing statistical regularities present in the training corpora.
However, their textual training data can suffer from reporting bias, where the frequency distribution of specific events and properties in text may not reflect the real-world distribution of such properties (Gordon and Van Durme, 2013).
For example, while grass is typically green, this may be under-reported in web corpora (as it is assumed to be true), and while motorcycle crashes may be more common in the real world, plane crashes are mentioned far more in news text (Gordon and Van Durme, 2013).
Misra et al.
(2016) highlight the reporting bias in “human-centric” image annotations and find that the noise in annotations exhibits a structure that can be modeled.
Dataset: ViComTe 3. Dataset Mining
For each relation color, shape, material, size, and object co-occurrence, our data take the form of (subject, object) tuples extracted from object distributions per subject. The goal is to predict the object and its distribution from the subject and relation.
Table 1 summarizes the number of classes and subject-object pairs for each relation.
Color, Shape, Material
For color, shape, and material, the subject is a noun and the object is the color, shape, or material property of the noun, mined from attributes of Visual Genome (VG) (Krishna et al., 2016).
3 We manually create a list of single-word attributes for each relation, and only
VG subjects that are matched with a specific attribute for more than a threshold number of times are recorded, in order to avoid noise in the dataset.
The thresholds for color, material, and shape are 5, 2, and 1, respectively, chosen based on the availability of attributes of each relation in VG.
VG attributes are filtered with the following steps: (1) attribute “Y colored / made / shaped” is treated as “Y”; (2) select only the last word for compound attributes (e.g. treat “forest green” as “green”); (3) similar attributes are merged into a main attribute class (e.g. “maroon” and “crimson” become “red”).
The above procedure produces a distribution over the set of attributes for each subject noun.
From that distribution, a (subject, object) data instance is generated for each subject where the object is the attribute that associates with it the most.
See the first three rows of Table 1 for examples.
Size
Size is separated into size_smaller and size_larger , where the subject is a noun and the object is another noun that is smaller or larger, respectively, than the subject.
To form the size dataset, we obtain a set of concrete nouns that appears in VG, which we manually classify into 5 size categories ( tiny , small , medium , large , and huge ).
Typical objects in each category includes pill , book , table , lion , mountain , respectively.
We randomly pick two nouns from different categories to form a (subject, object) pair.
Visual Co-occurrence
The visual co-occurrence dataset is generated in a similar way to the color, shape, and material datasets.
Co-occurrence distribution is extracted from Visual Genome where two objects that occur in the same scene graph together for more than 8 times are recorded, and a (subject, object) instance is generated for each subject, where the object is the noun that co-occurs with the subject the most.
3. Data Grouping
Following Paik et al.
(2021), we split the color, shape, and material datasets each into three groups:
S INGLE , M ULTI , and A NY .
The S INGLE group is for subjects whose most common attribute covers more than 80% of the probability, e.g., the color of snow is almost always white.
The M ULTI group is defined as subjects not in the S INGLE group where more than 90% of the probability falls in the top 4 attribute classes, e.g., the color of a penguin in Fig.
The rest of the subjects are in the A NY group.
Lower model performance for the S INGLE group would indicate the influence of reporting bias.
For example, if the model is unable to correctly capture the distribution of the color of snow , it is likely because the color of snow has low probability of being reported in the training corpus, as people know it is white by default.
[Table]
[Table]
3. Templates
In order to elicit model response and extract target objects and distributions from text, we manually design a set of templates for each relation.
There are 7 templates for color, shape, and material each, 8 for size, and 4 for visual co-occurrence.
See
Table 1 for example templates.
3. Wikipedia Data
In order to compare text-based and visuallygrounded data, we mine the color, shape, and material datasets from Wikipedia data, which is typically used in model pretraining.
To mine these text-based datasets, we combine the sets of subjects in VG, take the manual list of attributes as objects again, and extract (subject, object) pairs if the pair matches any of the pre-defined templates.
In Section 3.5 we will show the advantages of the
VG-mined dataset over this text-based dataset.
3. Dataset Evaluation
To ensure the validity of ViComTe, we compare our color dataset with the human-annotated CoDa dataset (Paik et al., 2021), which we assume is close to real-world color distributions and has minimal reporting bias.
We see a reasonably strong correlation with CoDa, indicating that the ViComTe dataset is a good and cost-effective approximation to human annotations.
Metrics
We report the Spearman’s rank-order correlation between the two distributions in comparison, averaged across all subjects.
The Spearman correlation is used instead of the Pearson correlation since for our purpose the rank of the object distributions is more important than the exact values, which may change due to data variability.
The top-1 accuracy (Acc@1) is the percentage of the objects with the highest probability in the source distributions matching those in the target distributions.
These two metrics are also used in later sections when evaluating model distributions.
Analysis
Table 2 shows the detailed results of the evaluation of the ViComTe and Wikipedia color datasets by comparing with the human-annotated dataset, CoDa.
We can see that ViComTe has much higher Spearman correlation with CoDa, as well as substantially higher top-1 accuracy for the S INGLE group.
The correlation is expected to be low for the A NY group, because objects in the A NY group can have many possible colors.
Reporting bias is present in both datasets, as the average number of occurrences of S INGLE group subjects are much fewer than that of the M ULTI and A NY group subjects.
Counter-intuitively, for ViComTe, the highly-correlated S INGLE group subjects have fewer average occurrences than the ones with low correlations.
This is contrary to our expectation that more frequent objects would better reflect the human-perceived distribution and can be explained by S INGLE subjects being easier to represent even without a large amount of data.
One example where the Wikipedia distribution diverges from the CoDa distribution is “penguin”, whose most likely color in CoDa is black, followed by white and gray; however, its top color in Wikipedia is blue, because “blue penguin” is a specific species with an entry in Wikipedia, even if it is not as common as black and white penguins.
One example where the VG distributions diverge from CoDa is “mouse”, because in VG, most occurrences of “mouse” are computer mice, which are most commonly black, whereas when asked about the word “mouse”, human annotators typically think about the animal, so that the most likely colors in CoDa are white and gray. 3. Dataset splits Each of the color, shape, material, size, and cooccurrence datasets is split into 80% training data and 20% test data. All evaluation metrics are reported on the test set.
The training set is used for the logistic regression and the soft prompt tuning algorithm (Section 4.2).
Probing Visual Commonsense 4. Models
We examine 7 pretrained transformer-based models and 2 variations of them, trained on a variety of data.
BERT (Devlin et al., 2019), ALBERT (Lan et al., 2020), and RoBERTa (Liu et al., 2019) are trained on text only using a masked language modeling objective (MLM).
Oscar (Li et al., 2020) is a vision-language model based on the BERT architecture, trained with an combined MLM and contrastive loss on text-image pairs.
VisualBERT (Li et al., 2019) is another vision-language model based on BERT that learns joint representation of images and text. Tan and Bansal (2020) introduce the “vokenization” method, which aligns language tokens to their related images, mitigating the shortcomings of models trained on visually-grounded datasets in text-only tasks. Since our task is purely text-based, we also experiment with a pretrained vokenization model (BERT + VLM on Wiki).
Finally, we use representations from CLIP (ViT-B/32)
(Radford et al., 2021), which is trained with a contrastive image-caption matching loss. Distilled Oscar
As our experiments involve exclusively textual inputs, we develop a knowledgedistilled version of Oscar (“Distilled”) which corrects for the lack of image input in our task. Knowledge distillation (Hinton et al., 2015; Sanh et al., 2019) is the process of transferring knowledge from one model to another, where the student model is trained to produce the output of the teacher model. Here, we use Oscar as the teacher and BERT as the student. The training data is part of the Oscar pretraining corpus: COCO (Lin et al., 2014),
Flickr30k (Young et al., 2014), and GQA (Hudson and Manning, 2019), and the Distilled Oscar model has access to the text data only.
We use the
Kullback-Leibler loss to measure the divergence between the output logits of BERT and Oscar, and optimize the pretrained BERT on that loss to match the outputs of Oscar.
Configurable parameters are set the same as for Oscar pretraining.
CaptionBERT
Since VL models are trained largely on caption data, it could be that the differences between a text-only model and a VL model come not from a difference in modalities – text vs. images and text – but from a difference in domain – webtext vs. image captions.
In order to disentangle the effects of the domain difference from those of visual inputs, we train a BERT model from scratch (“CaptionBERT”) on Oscar’s caption-based text data (the same data as for the Distilled model).
If
CaptionBERT, which does not have exposure to visual inputs, performs better than BERT and similarly to VL models (which are trained with visual inputs), it would suggest that the training domain matters more than the modality.
If, on the other hand, CaptionBERT performs worse than VL models, it would highlight the importance of modality.
4. Evaluation Methods
We compare the visual commonsense abilities of pretrained unimodal and multimodal models.
Given a list of prompts and a subject word, each model outputs the distribution of the target word.
Following Paik et al.
, we apply zero-shot probes to models that are trained on a language modeling objective, and conduct representation probes for those that are not.
We report the prediction accuracy and the Spearman correlation of the output distribution with the true distribution.
We use models trained with an MLM objective (BERT, Distilled, etc) directly for zero-shot predic- tion of masked tokens.
For Oscar we add a wordprediction head on top of it.
The results across templates are aggregated in two modes.
In the “best template” mode, for each example, the highest Spearman correlation among all templates is reported, and the top-1 result is regarded as correct if the true target object is the same as the topresult of any of the templates.
In the “average template” mode, the output distribution is the mean of the distributions across all templates.
Since CLIP is not trained on a token-prediction objective, we implement logistic regression on top of the frozen encoder output, to predict the target attribute or object.
The input is each of the templates with the subject [X] filled with an input in the dataset.
Like Paik et al.
(2021), to give the model ample chance of success, we take the template that results in the best test accuracy score, report that accuracy and the Spearman correlation associated with that template.
For the classification head, we use the Scikit-Learn implementation of Logistic Regression (random_state=0, C=0.316, max_iter=2000) (Pedregosa et al., 2011).
Soft prompt tuning
In order to overcome the limitation of self-designed prompts, we incorporate prompt tuning technique that learns soft prompts by gradient descent, from Qin and Eisner (2021).
The algorithm minimizes the log loss:
[Equation]
for a set of example pairs E r and template set T r .
4. Size Evaluation
The size dataset differs from the other datasets in that we use relative sizes (X is larger/smaller than
Y), as absolute size information is hard to obtain.
Thus, we use two evaluation strategies for size.
Rank partition
First, as in the previous prediction task, given a template such as “[X] is larger than [Y]” and an object [X], we ask the model to predict the distribution of [Y], taking only the distribution D of nouns in the size dataset.
For the current object [X], we take the nouns in size categories that are smaller than the category of [X] ( N sm ), and those that are in larger categories ( N lg ).
Let the length of N sm be m and the length of N lg be n .
Then for the “larger” templates, we compute the average percentage of overlap between the top n objects in D and N lg and that between the bottom m objects in D and and N sm .
For the “smaller” templates, the “top” and “bottom” are reversed.
Adjective projection
The second approach follows that of van Paridon et al.
(2021), which projects the word to be evaluated onto an adjective scale.
In this case, we compute the word embeddings of the adjectives “small” and “large” and the nouns from models, so the scale is −−→ large − −−−→ small and the projection is calculated by cosine similarity.
For instance, for the example noun “bear”, the projection score is given by:
[Equation]
With good word embeddings, larger nouns are expected to have higher projection scores.
The validity of the adjective scales from word representations is shown by Kim and de Marneffe (2013).
4. Measuring Model Reporting Bias
We measure the reporting bias of our models by comparing model performance on datasets with different levels of reporting bias and on the S INGLE , M ULTI , A NY groups of the ViComTe dataset.
We assume that CoDa contains no reporting bias, in which case we can interpret Table 2 as showing that ViComTe contains a relatively small amount of it, and Wikipedia contains a relatively large amount. Thus, a larger correlation of model outputs with
ViComTe and a smaller one with Wikipedia would indicate less model reporting bias.
Also, since the S INGLE group subjects are those whose attribute distribution concentrates on a single attribute, these subject-attribute pairs are less likely to be reported in text corpora or even image annotations. Therefore, lower model correlation on the S INGLE group than the M ULTI and the A NY groups would be a sign of model reporting bias.
Results
The experimental results show that multimodal models outperform text-only models, suggesting their advantage in capturing visual commonsense.
However, all models are subject to the influence of reporting bias, as they correlate better with the distributions from Wikipedia than those from CoDa [Table] and ViComTe.
Prompt tuning and knowledge distillation substantially enhance model performance, while increasing model size does not.
5. Results with MLM Objective Color, Shape, Material
The resulting model performance for the “average template” mode is shown in Table 3.
Prompt tuning is done in this mode only.
Note that because the top-1 accuracy is taken among all possible classes of each relation, it should be interpreted together with the number of classes (Table 1).
We can see from Table 3 that Oscar does better than BERT in almost all cases.
Significant difference between Oscar (base) and BERT (base) is seen in most cases.
Also, after soft prompt tuning, both the Spearman correlation and the accuracy substantially improved.
Although there is considerable variation of the Spearman correlations, we find consistent improvement per example with both prompt tuning and multimodal pretraining (Appendix A.2).
Table 3 also shows that knowledge distillation helps improve the performance of BERT in all cases, and the distilled model can sometimes even outperform the teacher model, Oscar.
Moreover, the large version of each model does not always outperform its base counterpart, suggesting that increasing the size of the model does not enhance the model’s ability to understand visual commonsense. Instead, training with visually grounded data does. Fig. 2 illustrates the Spearman correlations of different models with the color distributions from
CoDa, ViComTe and Wikipedia, under the “best template” mode.
7 All models correlate moderately
CoDa
VG
Wiki
BERT
Oscar
Distilled
RoBERTa
ALBERT
Vokenization
VisualBERT
CaptionBERT 46.
Sp e a r m a n c o rr e l a t i o n with all three datasets, with the highest correlations to Wikipedia, indicating text-based reporting bias in all model types.
BERT has the largest correlation gap between Wikipedia and CoDa, whereas the visually-grounded models have smaller gaps, indicating less reporting bias in VL models.
Visual Co-occurrence
Table 3 also contains the results on visual co-occurrence before and after prompt tuning.
Only the Spearman correlations are reported, because the top-1 accuracy is meaningless due to the large number of possible co-occurring objects with any noun.
Before prompt tuning, BERT has small Spearman correlations, suggesting that it may contain little knowledge about the visual co-occurrence relationship.
Oscar demonstrates more such knowledge under the zero-shot setting.
After prompt tuning, all model performances improve.
[Table]
[Table]
5. Results with Classification Head
Table 4 shows the results of BERT, CLIP, and Oscar when topped with a classification head.
We observe that Oscar and CLIP achieve similar performance and both outperform BERT.
Note that, while Visual Genome is part of Oscar’s pretraining corpus and one might suspect that that gives it an advantage, CLIP is trained on a large corpus from web search that is unrelated to Visual Genome.
Therefore, we can conclude that multimodal models pretrained on both images and text outperform text-only models.
Table 5 breaks down the results in Table 4 into three subject groups.
Oscar and CLIP outperform
BERT in almost all cases.
The top-1 accuracy is higher for the S INGLE group than for the M ULTI and A NY groups, perhaps because the S INGLE group subjects have only one most likely target attribute, which may be easier to predict.
Note that the Spearman correlations for all three models become higher from group S INGLE to M ULTI to A NY .
Paik et al.
(2021) argue that higher correlation for the A NY and M ULTI groups is a sign of model reporting bias, as objects in those two groups are more often reported.
Thus, the results here indicate that reporting bias is still present in multimodal models.
5. Results: Size Relation
Table 6 shows results of the rank partition method (Section 4.3), before and after prompt tuning. Sur- [Table]
prisingly, prompt tuning does not help in this case.
Moreover, the performance for the “larger” templates is higher than that of the “smaller” templates, suggesting that the models contain inherent preference towards the “larger” templates
[Table]
Fig.
3 shows the results of the adjective projection method.
8 For BERT and Oscar, we use the average embedding of the subword tokens of the nouns projected onto that of the adjectives “large” and “small”.
For CLIP, we take the textual encoder outputs as the embeddings, resulting in a different score range from that of BERT and Oscar.
The results show the following trend: larger objects are projected onto the “large” end of the spectrum, although the trend is sometimes broken towards the “huge” end.
This may be due to the “huge” group tiny small medium large huge 0.
c o s _ s i m bert oscar clip including nouns such as “pool” and “house” which can be modified by a relative size indicator “small”.
In Table 3, the accuracy of BERT for shape is particularly low (only 6.7%), despite that shape has only 12 classes.
We hypothesize that this is due to reporting bias on shape in the text corpora
that
BERT is trained on.
This hypothesis is supported by mining sentences from Wikipedia that contain (noun, attribute) pairs, where we see that the relation shape has fewer number of occurrences than material and color (Appendix A.3).
We also investigate whether the advantage of the visually-grounded models over pure-language models comes from the domain difference between web corpora and image captions, or the presence of actual visual input.
Although its teacher is trained with visual inputs, the Distilled model is trained only on captions data and its performance matches that of Oscar, so we hypothesize that grounded training data enhance models’ ability to capture visual commonsense.
The CaptionBERT results support the hypothesis in favor of domain difference, since it performs better than BERT in both
CoDa and VG (Fig. 2).
Nevertheless, the visual inputs also have an effect, as Oscar has a higher correlation than CaptionBERT on CoDa.
Thus, it seems that both domain and modality affect the ultimate model performance. Finally, although multimodal models show improvement on the task, sometimes the improvement is not significant and the resulting correlations are still weak.
Further work is needed to enhance the visual commonsense abilities of the models and mitigate reporting bias, and our datasets can serve as an evaluation method.
Conclusion
In this paper, we probe knowledge about visually salient properties from pretrained neural networks.
We automatically extract dataset of five visual relations: color, shape, material, size, and cooccurrence, and show that our ViComTe dataset has a much higher correlation with human perception data for color than data mined from Wikipedia.
We then apply several probing techniques and discover that visually-supervised models perform better than pure language models, which indicates that they can better capture such visual properties.
Distilling the knowledge from a visually-supervised model into a pure language model results in comparable performance with the teacher model. We also observe less reporting bias in both visually-grounded text (VG-mined datasets) than Wikipedia text and visually-grounded models (Oscar, DistilledOscar, VisualBERT, and CLIP) than pure language models.
However, visuallygrounded models are still subject to the influence of reporting bias, as seen in the per-group analysis, where both types of models perform better for the M ULTI group than the S INGLE group. Acknowledgments We would like to thank the reviewers for their comments and suggestions.
Chenyu Zhang is supported by the Pistritto Research Fellowship. Elias Stengel-Eskin is supported by an NSF Graduate Research Fellowship.
Zhuowan Li is supported by
NSF 1763705.
A Appendix A. List of Objects
Table 7 shows the list of all possible attributes for relations color, shape, and material.
Table 8 shows the list of objects in the five categories of relation size.
Visual co-ocurrence has a large number of objects that are not listed here for space reasons.
[Table] [Table]
A. Additional Probing Best template mode
Table 9 contains zero-shot results under the “best template” mode, for BERT (base), Oscar (base), BERT distilled from Oscar,
RoBERTa (base), ALBERT (base), Vokenization, and VisualBERT (base).
These results demonstrate similar trends as the ones in the “average template” mode
[Figure]
Per-object analysis
Fig.
4 illustrates the finegrained Spearman correlation ± standard deviation per object group for BERT and CLIP.
[Figure]
Size per-object
Fig.
5 shows how the per-object projection scores on the size spectrum from BERT and Oscar are correlated
[Figure]
[Table]
Per-Subject Comparison
Fig. 6 and Fig.
7 show how the Spearman correlations of 10 individual subjects improve after soft prompt tuning and after multimodal pretraining.
Consistent improvement can be seen in color, material, and cooccurrence.
Although we report average Spearman correlations in Table 3 and there are large standard deviations, here we show that when improvement is observed collectively, it is also consistent across subjects.
With shape, the improvement is less obvious (45. to 50.4 for prompt tuning and 49.2 to 50.4 for multimodal pretraining).
A. Error Analysis Data
The three subjects with the highest and lowest Spearman correlation are shown in Fig.
8 and
Fig.
Wikipedia
Table 10 shows the number of (noun, attribute) pairs of the three relation types in
Wikipedia.
Shape has fewer occurrences than material and color.
[Table]
Model
Table 11 shows the errors made by BERT and Oscar in the “average template” mode before prompt tuning.
Overall, subjects with low correlation are those that are less often reported in Visual
Genome as well as in textual data.
A. Resources BERT
, RoBERTa, ALBERT
We use the Huggingface implementations of BERT, RoBERTa, and
ALBERT.
Oscar
See the GitHub repository for the code and pretrained Oscar: https://github.com/ microsoft/Oscar .
CLIP
We use the CLIP model released by
OpenAI: https://github.com/openai/
CLIP .
Vokenization
See the GitHub repository for the pretrained model: https://github.com/ airsplay/vokenization .
[Figure]
a r m a n c o rr e l a t i o n
[Figure]
[Figure]
r m
[Figure]
n c o rr e l a t i o n
[Figure] [Figure]
Oscar
[Table]
[Figure] [Figure]