What do Models Learn From Training on More Than Text? Measuring Visual Commonsense Knowledge (2022)

Abstract: There are limitations in learning language from text alone. Therefore, recent focus has been on developing multimodal models. However, few benchmarks exist that can measure what language models learn about language from multimodal training. We hypothesize that training on a visual modality should improve on the visual commonsense knowledge in language models. Therefore, we introduce two evaluation tasks for measuring visual commonsense knowledge in language models (code publicly available at: github.com/lovhag/measure-visual-commonsense-knowledge) and use them to evaluate different multimodal models and unimodal baselines. Primarily, we find that the visual commonsense knowledge is not significantly different between the multimodal models and unimodal baseline models trained on visual text data.

## introduction language models (lms) trained on large amounts of textual data have shown great performance on several textual tasks @xcite . however, recent work has illuminated limitations with text-only training of lms. these limitations arise from a lack of meaning @xcite and experience @xcite , together with the problem of reporting bias @xcite . multimodal training has been identified as one way to create models that do not suffer from the aforementioned limitations @xcite . while several multimodal models have been developed @xcite @xcite , few evaluation methods exist that can tell us whether multimodal training mitigates text-only training limits. if we wish to successfully create multimodal lms that learn from more than text, we need a way to evaluate them for what we expect them to have learned from their multimodal training. one hypothesis is that multimodal training should aid lms in learning commonsense knowledge @xcite . there are several textonly evaluation tasks that aim to measure the commonsense knowledge in lms @xcite , but none of them focus explicitly on the commonsense knowledge learned through training on more than text. in this work, we focus on models trained on images and text, denoted vision-and-language models. we reason that if there is any additional information to be learnt from a visual modality it should firstly be basic visual commonsense knowledge. that is, visual conceptual knowledge that is viewed as commonsense by humans, and thus not attainable from text alone due to reporting bias. we propose a simple method for measuring the visual commonsense knowledge of a model using two zero-shot masked language text-only tasks, depicted in figure 1 . the first task is the memory colors evaluation task @xcite and the second we create based on the visual features in the centre for speech, language and the brain (cslb) concept property norms dataset @xcite . we refer to the latter task as the visual property norms evaluation task. we complement our work with the results of four vision-and-language models and four baselines on these two tasks. ## evaluation tasks our aim is to evaluate models for visual commonsense knowledge. to do this we make use of the existing memory colors evaluation task described in section 2.1, and introduce a new evaluation task, visual property norms in section 2.2. memory colors is smaller than visual property norms and specifically focuses on visual information related to the color of different concepts, so it is potentially easier. we include both tasks to get a performance curve over increasing difficulty. common for both tasks is that they contain queries in english relating to visual properties of tangible concepts and that these queries are based on the knowledge of multiple human participants. therefore, the tasks can be considered to evaluate a basic aspect of visual commonsense knowledge. also common for both tasks is that they use textual templates containing a [mask] token to be predicted by a model in a cloze-style fashion, similarly to the method used by @xcite and @xcite . the advantages with querying the models in this fashion is that most lms foot_1 already have been exposed to this type of query format, including most multimodal models. we can then evaluate any model in a masked language modelling fashion on these tasks without additional training or having to make modelspecific adaptations, enabling easy evaluation for researchers who wish to use these evaluation tasks. this form of cloze-style evaluation is also referred to as prompt-based retrieval. the reliability of this method has recently been questioned by @xcite and @xcite due to the query format sensitivity of lms. to alleviate this issue, we evaluate the models using several different prompts for each of the two tasks. ## models we evaluate four multimodal pre-trained models for their visual commonsense knowledge. these are clip-bert both with and without imagination foot_2 @xcite , a lxmert base uncased @xcite and visualbert @xcite . we also evaluate four unimodal baseline models. these are a bert base uncased pretrained on english wikipedia and bookcorpus, a bert base uncased further trained on the pure-text part of the clip-bert training data (bert-clip-bert-train) and two bert base uncased models trained on the pure-text part of the lxmert training data, one from scratch and one initialized from pre-trained bert weights (bert-lxmert-trainscratch and bert-lxmert-train). all models are to some extent based on the bert base architecture and consequently share the same vocabulary and tokenizer. they are also of similar sizes with ∼ 110m trainable weights, the exception being lxmert with ∼ 230m trainable weights. additional information about the models can be found in the appendix. adapting the models for pure-text queries the majority of current multimodal models have not been developed to be queried only with text. in this case, both clip-bert and visualbert should work well with only removing their visual features input, since they are single-stream models. however, lxmert is a dual-stream model that requires a visual feature input. we handle the removal of visual information by simply removing the visual processing chain in lxmert, making the language input the only input given to the cross-modality encoder in the model. this would not work if we still wanted to use the model in a multimodal fashion, but we can make this adaption since we are only interested in querying the model for visual commonsense knowledge via language. ## results the results of the models on our two evaluation tasks can be seen in figure 2 . we format the analysis of the results around a set of questions. do the multimodal models display more memory colors knowledge? the multimodal clip-bert-explicit model has the best performance on this task. so to some extent, yes. but it is worth noting that the unimodal bert model trained on lxmert training data is second best on the task, outperforming both lxmert and visual-bert, indicating a small multimodal advantage. ## limitations our work is limited to a subset of vision-andlanguage models, so the results found may not translate to all such model types. also, since our evaluation utilizes prompt-based retrieval, its measurement accuracy depends on how well this method works for lms. additionally, as previously mentioned, we do not investigate how well the multimodal models adapt to a unimodal input. thus, our results depend on whether the models were functioning adequately with our method of adapting them to a unimodal input. ## ethical considerations our work should not have any direct ethical implications, since we mainly introduce evaluation tasks and evaluate different models on them. we do however investigate visual conceptual perceptions based on data from a potentially small group of people whose world-view may be culturally different from that of other individuals. this means that we may encourage knowledge that benefits some people more than others. similar issues are discussed by @xcite . our investigation is limited to english-language models and datasets, limiting the generality of our conclusions. ## conclusions we introduce new evaluation methods for measuring the visual commonsense knowledge in lms and evaluate a number of multimodal lms on these benchmarks. we find that there are no significant differences in performance between models trained on pure text and models trained on images and text. most prominently, we find that a unimodal lm trained on image captions and vqa queries can attain a visual commonsense knowledge on par with that of a multimodal model. we also confirm the results by jiang et al. (2020) and @xcite , that lms are sensitive to query format even when querying for commonsense knowledge. this casts some doubts on what is really measured in a model for a cloze task and whether we can reason about lms as having knowledge. an interesting future step would be to investigate this further and see if it would be more applicable to use e.g. probing or some other evaluation method. nonetheless, this is a first step towards measuring the visual commonsense knowledge in multimodal as well as unimodal lms. we hope that the evaluation tasks introduced here may aid other researchers in their aim to create models that learn language from more than text. model text visual text images+text backbone training objectives bert 80m mlm, nsp clip-bert-implicit 80m 4.7m 400m mlm clip-bert-explicit 80m 4.7m 400m mlm bert-clip-bert-train 80m 4.7m mlm lxmert 9.2m 0.1m mlm, rfr, dlc, itm, iqa bert-lxmert-train 80m 9.0m mlm bert-lxmert-train-scratch 9.0m mlm visualbert 80m 1.7m 0.1m mlm, itm