Using Commonsense Knowledge to Answer Why-Questions (2022)

Abstract: Answering questions in narratives about why events happened often requires commonsense knowledge external to the text. What aspects of this knowledge are available in large language models? What aspects can be made accessible via external commonsense resources? We study these questions in the context of answering questions in the TellMeWhy dataset using COMET as a source of relevant commonsense relations. We analyze the effects of model size (T5 and GPT3) along with methods of injecting knowledge (COMET) into these models. Results show that the largest models, as expected, yield substantial improvements over base models. Injecting external knowledge helps models of various sizes, but the amount of improvement decreases with larger model size. We also find that the format in which knowledge is provided is critical, and that smaller models benefit more from larger amounts of knowledge. Finally, we develop an ontology of knowledge types and analyze the relative coverage of the models across these categories.

## introduction humans reason about events in narratives by making inferences about why those events happen. the recently introduced tellmewhy dataset tests for this capability by posing why questions over events in simple narratives @xcite ). answering these often requires commonsense knowledge (csk) that is not explicitly stated as part of the narratives. indeed, qa models built over standard base sized models fare poorly, especially where the answer is not stated in the narrative. there are two broad avenues for incorporating the necessary commonsense knowledge for this task -using larger language models (e.g. t5-11b @xcite ) and leveraging external knowledge resources. the former can be seen as an implicit approach, where we tap knowledge that is acquired via language modeling and general qa task pretraining. the latter is an explicit approach where we inject knowledge from a resource as part of the context. we start by asking three questions that can inform future research along these avenues: (1) what aspects of commonsense knowledge are already accessible to larger language models? (2) what aspects can be made accessible by injecting information from relevant knowledge sources? (3) what kinds of knowledge remains inaccessible? for the tellmewhy task, we explore the utility of comet foot_0 @xcite as a knowledge source. comet is a transformer-based model that generates commonsense inferences about events that it has learned from atomic @xcite and conceptnet @xcite . however, the automatically generated knowledge may contain incorrect or irrelevant inferences. we start by exploring multiple ways of integrating this kind of knowledge into a qa model. first, we experiment with the best way of selecting relations from comet that should be added to model input. we find that adding diverse types of relations helps the most. next, we investigate various ways to integrate this knowledge into a model's input. while comet relations are usually words or phrases, converting them to sentences using simple verbalization templates works the best. finally, we analyze the amount of external knowledge needed by models of various sizes. smaller models benefit more from a larger amount of knowledge while larger models do well with less external knowledge. these findings are used to build models of multiple sizes that can use external commonsense knowledge for the tellmewhy task. we use diverse types of information from comet converted into fluent sentences as part of the model inputs. for small models, we supply more commonsense knowledge to boost model performance while larger models are given less knowledge. to analyze the relative merits of all these approaches, we manually categorized the why questions according to the types of knowledge needed to answer them. we find that most questions target consequence, goal seeking, desire, and reactionary knowledge types. we categorize the rest as other and analyze the performance of different models across these knowledge categories. our analysis shows that models seem to particularly lack the ability to understand and utilize 'goal seeking' knowledge. in summary, our contributions are: this section gives an overview of the data and evaluation scheme, and defines a formulation to describe the model configurations we investigated. ## empirical insights into knowledge integration we instantiate the abstract model formulation described in equation 1 with various knowledge integration approaches. we ask three questions about injecting external knowledge into models to improve why question answering. our findings influence the choices we make when building the best possible model. we use the small and large models for our investigations in this section. examples of each variant are shown in ## distilling the empirical insights: the knowwhy approach finally, we combine our findings to create the knowwhy approach. we use it to build the best possible models of all sizes -small, large and very large -for the tellmewhy task. from findings 1-2, we use @xmath0, g verbal and ω = diverse-comet. from finding 3, we use @xmath1, @xmath2, and @xmath3. ## analysis to better understand the strengths and weaknesses of these models, we defined an ontology for the types of knowledge that are required to answer tellmewhy questions. we identified five cate-matt and sarah were pregnant. they wanted to announce it in a fun way. they wrote it on a cake. the, they invited their friends over. when their friends saw the cake, they were excited. question: why were matt and sarah pregnant? human answer: the two had previously been intimate together. model answer: matt and sarah were pregnant because they wanted to have a baby. maggie was drinking some green juice. she left the cup out awhile. when she went to get another sip it tasted odd. she realized that it had separated weirdly. she threw the juice out. ## conclusion answering why questions requires several forms of commonsense knowledge. this paper investigates different aspects of incorporating external knowledge to improve this process. we discover several empirical insights on how to incorporate external knowledge. by incorporating these insights, our approach, knowwhy, successfully uses external knowledge to help models of all sizes (small, large, and very large) answer why questions better; but they still fall short of human performance. questions that involve implicit inferences are harder for all the models, and require modeling innovations. our investigation opens up interesting questions, such as learning when and how to add external knowledge, in order to further close the gap with humans.