Are Visual-Linguistic Models Commonsense Knowledge Bases? (2022)

Abstract: Despite the recent success of pretrained language models as on-the-fly knowledge sources for various downstream tasks, they are shown to inadequately represent trivial common facts that vision typically captures. This limits their application to natural language understanding tasks that require commonsense knowledge. We seek to determine the capability of pretrained visual-linguistic models as knowledge sources on demand. To this end, we systematically compare language-only and visual-linguistic models in a zero-shot commonsense question answering inference task. We find that visual-linguistic models are highly promising regarding their benefit for text-only tasks on certain types of commonsense knowledge associated with the visual world. Surprisingly, this knowledge can be activated even when no visual input is given during inference, suggesting an effective multimodal fusion during pretraining. However, we reveal that there is still a huge space for improvement towards better cross-modal reasoning abilities and pretraining strategies for event understanding.

## introduction commonsense knowledge is essential in human life for task solving and communication. being aggregated knowledge acquired from past experiences and communications, it is usually left implicit in human communication, and used subconsciously for reasoning and drawing inferences. this puts a challenge on natural language understanding systems, and a large body of work has put forward approaches to provide commonsense knowledge to models for downstream tasks @xcite @xcite . while knowledge bases have been a popular way to provide relevant knowledge for a task at hand, more recently, pretrained language models (ptlms) have become a popular mechanism to extract knowledge in free-form text on demand. tasks range from, e.g., persona-grounded dialog @xcite , narrative story generation @xcite , to metaphor generation @xcite . @xcite and @xcite , however, suggest that text corpora alone may be insufficient for knowledge acquisition due to reporting bias found in them @xcite . this has led to analyzing the knowledge that ptlms possess through dedicated probing studies @xcite @xcite existing works on probing ptlms have used a loose categorization of commonsense, which limits a comprehensive understanding of the types of commonsense they possess and lack, respectively. at the same time, the literature proposes vision as a promising knowledge source @xcite @xcite . it seems therefore straightforward to leverage visuallinguistic (vl) models for knowledge extraction on demand-these representation models are extensions of ptlms to the visual-linguistic domain by pretraining lms and image recognition models jointly on multimodal data @xcite @xcite . yet, the question on their capability to capture commonsense knowledge that can be activated through language only @xcite ) is yet to be explored systematically. in this work, we address this research gap by conducting a controlled comparison of text-only and vl models. specifically, we extend a synthetic commonsense question answering (qa) dataset based on @xcite 's work, which structures knowledge relations into abstract types (called dimensions henceforth), and transform it to a qa inference task. we use the task to compare the models by applying them in a zero-shot manner, and in their natural setting-masked language modeling. the overarching question of our study is: do vl models learn to encode commonsense knowledge through multimodal pretraining, that can be activated during inference from textual input only? in particular, we seek to empirically answer: (q1) which dimensions of commonsense do vl models possess compared against text-only ptlms? (q2) during pretraining, does explicit visual information (i.e., images) benefit commonsense knowledge encoding? (q3) during inference, is explicit visual observation (i.e., images) necessary for recalling commonsense knowledge? (q4) do commonsense acquisition and retrieval depend on the architecture of vl models? we address the questions by performing a range of experiments using various pretrained models and ablated variants. we find that existing vl models do complement ptlms on certain commonsense dimensions, which are related to the visual world (part-whole, spatial , i.a.), and that they can be activated through language input only, making them promising for their use in natural language tasks that do not require explicit visual context. we also identify a range of limitations opening up several avenues for future work, including enhanced pretraining and modality integration strategies, and improved multimodal prompting @xcite @xcite . ## the qa dataset cwwv img to compare vl models against purely textual models with respect to the commonsense knowledge they capture, we extend @xcite 's procedure for creating a synthetic dataset of promptanswer candidate instances (cwwv) to that of a multimodal commonsense dataset (cwwv img ). it provides a set of qa instances for various knowledge relations, structured into 10 dimensions of commonsense knowledge (e.g., spatial ). questions are in the form of filled prompts @xcite : an instance in cwwv img has three natural language statements, each associated with a set of images. each statement is a pair of a prompt (e.g., shade is not) and one of three candidate answers (e.g., sunny). ## experiments: qa task and inference our goal is to assess the benefit of pretrained vl models for purely linguistic tasks underlying commonsense (cs) knowledge. specifically, we seek to answer the questions (q1) -(q4) that we put forward in §1. to this end, we use our derived image-grounded dataset, cwwv img , and evaluate pretrained vl models against language-only ptlms in a prompt-based qa task setting ( §4.1). we stress that in order to solve cwwv img , only natural language understanding and commonsense knowledge is required, but no explicit visual input (images). we perform our experiments in a zero-shot setting, i.e., without fine-tuning the models on the task, since our goal is to study the ability of task-agnostic pretrained models to capture commonsense knowledge @xcite . ## results to address the overarching question we put forward in §1, namely if vl models can serve as commonsense knowledge base, and to what extent they complement pure linguistic model, we first examine the encoded knowledge of the vl models on individual commonsense dimensions, i.e., (q1) in §1. we report the models' effectiveness by measuring their mean accuracy in selecting the correct answer out of the three statement candidates of each qa instance in cwwv img . we declare outperformance if @xmath0.05 according to the paired student's t-test @xcite for statistical significant differences between any two accuracy scores. 9 ## analysis to answer our questions (q2)-(q4) we put forward in §1, we first examine (q2), the benefit of visual information during pretraining. we then investigate (q3), the role of explicit visual input during inference. lastly, we look into (q4), whether the process of knowledge acquisition and retrieval act consistently across vl models. ## conclusion regarding our research questions put forward in §1, our findings strongly suggest that the vl models learn to encode certain visual knowledge in their textual streams during multimodal pretrain-ing, in particular for concrete concepts (see also @xcite , which they can activate from purely textual input during inference, i.e., visual information is not required, or not even beneficial. the fact that the textual stream is the driving force for inference is promising, given that we examined the benefit of vl models for a purely linguistic task. regarding the dependence of the architecture for commonsense acquisition and activation, we conclude that the examined single-stream seems to be better suited for text-only qa tasks, while double-stream seems to require some form of signal in the visual stream during inference (but cannot leverage it properly to the extent that it would be better than text-only input). in summary, we find vl models to be promising regarding their potential use for natural language tasks requiring commonsense knowledge. we also identified a range of limitations for future work: the ability to handle visual noise, to understand events and verbs, and to integrate inconsistent modalities towards metaphorical, rather than situated understanding. future work lies also on multimodal prompt-engineering for improved knowledge retrieval on commonsense intensive tasks.