Seeing the World through Text: Evaluating Image Descriptions for Commonsense Reasoning in Machine Reading Comprehension (2020)

Abstract: Despite recent achievements in natural language understanding, reasoning over commonsense knowledge still represents a big challenge to AI systems. As the name suggests, common sense is related to perception and as such, humans derive it from experience rather than from literary education. Recent works in the NLP and the computer vision field have made the effort of making such knowledge explicit using written language and visual inputs, respectively. Our premise is that the latter source fits better with the characteristics of commonsense acquisition. In this work, we explore to what extent the descriptions of real-world scenes are sufficient to learn common sense about different daily situations, drawing upon visual information to answer script knowledge questions.

## introduction the recent advances achieved by large neural language models (lms), such as bert @xcite , in natural language understanding tasks like question answering @xcite and machine reading comprehension @xcite are, beyond any doubt, one of the most important accomplishments of modern natural language processing (nlp). these advances suggest that a lm can match a human's stack of knowledge by training on a large text corpora like wikipedia. consequently, it has been assumed that through this method, lms can also acquire some degree of commonsense knowledge. it is difficult to find a unique definition, but we can think of common sense as something we expect other people to know and regard as obvious @xcite . however, when communicating, people tend not to provide information which is obvious or extraneous (as cited in @xcite ). if common sense is something obvious, and therefore less likely to be reported, what lms can learn from text is already being limited. @xcite and more recently @xcite and @xcite have tried to alleviate this problem by collecting crowdsourced annotations of commonsense knowledge around frequent phrasal events (e.g., personx eats pasta for dinner, personx makes persony's coffee) extracted from stories and books. from our perspective, the main limitation of this approach is that even if we ask annotators to make explicit information that they will usually omit for being too obvious, the set of commonsense facts about the human world is too large to be listed. then, what other options are there? as the name suggests, common sense foot_0 is related to perception, which the oxford english dictionary defines as the ability of becoming aware of something through our senses: sight (e.g., the sky is blue), hearing (e.g., a dog barks), smell (e.g., trash stinks), taste (e.g., strawberries are sweet), and touch (e.g., fire is hot). among those, vision (i.e., sight) is one of the primary modalities for humans to learn and reason about the world @xcite . therefore, we hypothesize that annotations of visual input, like images, are an option to learn about the world without actually experiencing it. this paper explores to what extent the textual descriptions of images about real-world scenes are sufficient to learn common sense about different human daily situations. to this end, we use a large-scale image dataset as knowledge base to improve the performance of a pre-trained lm on a commonsense machine reading comprehension task. we find that by using image descriptions, the model is able to answer some ## approach we evaluate image descriptions through a mrc task for which commonsense knowledge is required, and assume that answering a question incorrectly means the reader lacks such knowledge. most of what humans consider obvious about the world is learned from experience, and we believe there is a fair amount of them written down in an image's description. we will test this idea by using image descriptions as external knowledge. out of the different types of common sense, the text passages in the selected mrc dataset focus on script knowledge (schank and abelson, 2013), which covers everyday scenarios like brushing teeth, as well as the participants (persons and objects) and the events that take place during them. since scenarios represent activities that we do on a regular basis, we expect to find images of it. ideally, for each passage, we would automatically query an image dataset to retrieve descriptions related to what the passage is about. retrieval is a key step in our approach and for the time being, such process was done manually so we can focus on the image's description content rather than in the retrieval process itself. there is a considerable number of crowdsourced image datasets whose image descriptions are available, which means they can be collected (and extended, if needed) for a reasonable cost. the motivation behind our approach is that once such descriptions are proven to contain useful commonsense knowledge that it is not easily obtained from text data, one can think of extending the description collection. ## results for most of the questions in the unanswerable set, we did find related region descriptions. figure 3 shows some of the images retrieved and the regions that matched what the question is asking. besides its size, one of the main advantages of visual genome annotations is that they cover several regions that compose the scene in an image. thanks to this, we were able to find region descriptions that not only mention an object (e.g., a towel, scissors, a dollar-bill), but also add a description of how the object can be used (e.g., towel used for drying off, scissors for cutting string) or what does it represent (e.g., five dollar tip on table). this suggests that our hypothesis mentioned in section 1 about annotations of visual input might be correct. as shown in in a classic reading comprehension task, word matching usually helps to find the correct answer. however, mcscript2.0 evaluates beyond mere understanding of the text and as such, it was designed to be robust against it. out of the 56 questions in our dev set, we observed that the number of times a passage mentions the correct and the incorrect answer candidates is similar (42 and 36, respectively) and in either case this seemed to have influenced bert's predictions. this stayed roughly the same after we appended the region descriptions. ## conclusion and future work pre-trained large lms have significantly closed the gap between human and computer performance in a wide range of tasks, but the commonsense knowledge they capture is still limited. in this work, we presented a controlled experimental setup to explore the plausibility of acquiring commonsense knowledge from dense image descriptions. our preliminary results on a commonsense-mrc task suggest that such descriptions contain simple but valuable information that humans naturally build through experiencing the world. in future work, our aim is to automate the retrieval process and explore better ways of using region descriptions than the presented approach of modifying bert's input format.