Beyond Language: Learning Commonsense from Images for Reasoning (2020)

Abstract: This paper proposes a novel approach to learn commonsense from images, instead of limited raw texts or costly constructed knowledge bases, for the commonsense reasoning problem in NLP. Our motivation comes from the fact that an image is worth a thousand words, where richer scene information could be leveraged to help distill the commonsense knowledge, which is often hidden in languages. Our approach, namely Loire, consists of two stages. In the first stage, a bi-modal sequence-to-sequence approach is utilized to conduct the scene layout generation task, based on a text representation model ViBERT. In this way, the required visual scene knowledge, such as spatial relations, will be encoded in ViBERT by the supervised learning process with some bi-modal data like COCO. Then ViBERT is concatenated with a pre-trained language model to perform the downstream commonsense reasoning tasks. Experimental results on two commonsense reasoning problems, i.e.commonsense question answering and pronoun resolution, demonstrate that Loire outperforms traditional language-based methods. We also give some case studies to show what knowledge is learned from images and explain how the generated scene layout helps the commonsense reasoning process.

## introduction commonsense reasoning is an important yet challenging task in artificial intelligence and natural language processing. take commonsense question answering as an example, given a question and multiple choices, some commonsense knowledge is usually required to make the correct answer from the provided choices. motivated by the fact that images usually contain richer scene information, which can be viewed as an important supplementary resource to perceive for commonsense knowledge, this paper proposes to learn commonsense from images and incorporate such knowledge into the commonsense reasoning process. take the question 'where is a good idea but not required to have a fire extinguisher?' shown in to the best of our knowledge, we are the first to propose learning commonsense knowledge from images to facilitate the commonsense reasoning in nlp. the proposed model of using scene layout generation as the supervision demonstrates a preliminary exploration in this direction. other methods like learning commonsense from retrieved relevant images could also be investigated. we believe this novel approach may provide a new perspective for commonsense reasoning in nlp. ## visual commonsense knowledge images are made up of individual pixels, which are detailed but sometimes noisy. therefore, how to extract useful commonsense knowledge from images remains a challenging problem. inspired by the knowledge base in nlp, where knowledge is usually represented as a triple to demonstrate the relation between two entities, we focus on the attributes and relations of the objects in images. clearly, such information can be well captured by the scene layout. take the sentence 'a woman eats in the restaurant' as an example. images related to this sentence are shown in the figure 1 . we can see that the scene layouts of these images, including bounding boxes and labels, contains a lot of useful information for commonsense reasoning: from the above analysis, images usually contain rich scene information, such as size, position and co-occurrence relations, which are useful for understanding the commonsense knowledge hidden in language. so we propose to learn such visual commonsense knowledge and incorporate them into the commonsense reasoning models in nlp. ## conclusion in this paper, we propose a novel two-stage pipeline approach loire to learn commonsense from images. in the first stage, a text representation model vibert is trained in the bi-modal sequence-tosequence approach for scene layout generation on coco. therefore, visual commonsense knowledge like spatial relations will be encoded in vibert by the supervision of caption and image layout. after that, vibert is concatenated with a pre-trained language model to perform a knowledge-augmented reasoning process. experimental results show that loire outperforms the current state-of-the-art language models bert and roberta on two nlp commonsense reasoning tasks, i.e. commonsense question answering data commonsenseqa and pronoun resolution data winogrande. the ablation and case study further show that the improvements are truly owing to the learned visual commonsense knowledge, and how this knowledge helps the nlp reasoning process. the current approach is a preliminary study on the proposed direction of using images to automatically learn commonsense knowledge to facilitate the nlp reasoning tasks, which could be modified from the following aspects to further improve the empirical performances. firstly, larger bi-modal data could be employed to learn more commonsense required in the reasoning task. secondly, other bi-modal methods instead of training vib-ert by the supervision of scene layout generation may be investigated. thirdly, how to design intrinsic evaluation to help to understand what is learned by lorie is still challenging and will be considered in the future.