W ino W hy: A Deep Diagnosis of Essential Commonsense Knowledge for Answering W inograd Schema Challenge (2020)

Abstract: In this paper, we present the first comprehensive categorization of essential commonsense knowledge for answering the Winograd Schema Challenge (WSC). For each of the questions, we invite annotators to first provide reasons for making correct decisions and then categorize them into six major knowledge categories. By doing so, we better understand the limitation of existing methods (i.e., what kind of knowledge cannot be effectively represented or inferred with existing methods) and shed some light on the commonsense knowledge that we need to acquire in the future for better commonsense reasoning. Moreover, to investigate whether current WSC models can understand the commonsense or they simply solve the WSC questions based on the statistical bias of the dataset, we leverage the collected reasons to develop a new task called WinoWhy, which requires models to distinguish plausible reasons from very similar but wrong reasons for all WSC questions. Experimental results prove that even though pre-trained language representation models have achieved promising progress on the original WSC dataset, they are still struggling at WinoWhy. Further experiments show that even though supervised models can achieve better performance, the performance of these models can be sensitive to the dataset distribution. WinoWhy and all codes are available at: https://github.com/HKUST-KnowComp/WinoWhy .

## introduction commonsense reasoning, as an important problem of natural language understanding, has attracted much more attention in the nlp community recently @xcite @xcite ., * equal contribution. 2019). among all developed commonsense reasoning tasks, the winograd schema challenge (wsc) @xcite , which is a hard pronoun coreference resolution task, is one of the most influential ones. all questions in wsc are grouped into pairs such that paired questions have minor differences (mostly one-word difference), but reversed answers. for each question, we denote the other question in the same pair as its reverse question. one pair of the wsc task is shown in figure 1 . based on the design guideline of wsc, all commonly used features (e.g., gender, plurality, and co-occurrence frequency) do not have any effect. human beings can solve these questions because of their shared commonsense knowledge. for example, ordinary people can know that the pronoun 'it' in the first sentence refers to 'fish' while the one in the second sentence refers to 'worm' because 'hungry' is a common property of something eating things while 'tasty' is a common property of something being eaten. conventionally, people tried to leverage crowdsourced commonsense knowledge bases @xcite or search engines @xcite to solve the wsc task, but performances of these models are not satisfying. recently, pretrained language representation models @xcite @xcite have demonstrated significant improvements in both unsupervised and supervised settings. however, as these approaches treat the concept 'commonsense knowledge' as a black box, we are not clear about why they can do better (e.g., can these models understand commonsense or they just capture the statistical bias of the dataset) and do not know how to further improve them. to answer these two questions, in this work, we present the first deep diagnosis of essential commonsense knowledge for answering wsc questions. specifically, we invite annotators to first provide reasons for why they choose the answers when they answer the questions, and then group all the wsc questions by different types of used commonsense knowledge (e.g., the property of entities, temporal knowledge, or spatial knowledge). by doing so, we can then analyze what kinds of commonsense knowledge can be well represented and understood by current models and more importantly, we can be clear about what kinds of commonsense knowledge are still challenging for current models, which could be an important future research direction for solving not only the wsc task but also the general commonsense reasoning problem. after the diagnosis, based on the collected reasons, we also create a new task winowhy, which aims at better evaluating models' abilities to understand commonsense knowledge. for each question in the wsc task, we pair it with several reasons. models are required to distinguish the correct reasons from all very similar but wrong candidates. from examples in figure 2 , we can see that even though all candidates are highly related to the original question, only one of them is the correct reason for resolving the coreference relation. experimental results show that even though state-of-the-art models can achieve about 90% accuracy on the original wsc task, they are still struggling on winowhy questions, which shows that current models are still far away from un-derstanding the commonsense knowledge. moreover, by conducting experiments on both wsc and winowhy tasks, we prove that even though supervised models can achieve better performance, these models can be sensitive to the dataset distribution, which indicates that the improvement is probably coming from better capturing the statistical bias of the dataset rather than better understanding the required commonsense knowledge. the rest of the paper is organized as follows. in section 2, we present the diagnosis of essential commonsense knowledge for answering wsc questions, which includes the reason collection and categorization. after that, we show how we create winowhy in section 3. in sections 4 and 5, we introduce the detailed experiments and analysis on both the original wsc and the proposed winowhy tasks. we introduce the related work about commonsense reasoning in section 6. in the end, we conclude this paper with section 7. ## commonsense knowledge diagnosis commonsense reasoning is often viewed as one of the most challenging ai tasks and we still do not have a principled way of solving it. one important reason behind this is that, due to the vague definition of commonsense knowledge, we are not clear about what the essential knowledge types are and thus we are unclear about how to represent, acquire, and use them. as a result, we can only treat commonsense knowledge as a black box and try to learn it from limited training data. to explore a principled way of representing commonsense knowledge and solving commonsense reasoning problems, we take the winograd schema challenge as the breaking point to conduct a detailed diagnosis of what kinds of knowledge are essential for answering these questions. to be specific, we first ask human beings to provide reasons why they make the correct decisions for all wsc questions. after that, we categorize these reasons by the involved knowledge types (e.g., the property of objects, temporal knowledge, or spatial knowledge). by doing so, we are more clear about how to acquire, represent, and apply such knowledge. details are introduced as follows. ## winowhy in this section, we introduce details about the creation of winowhy. ## wsc experiments in this section, we present the performance of current models on wsc. by doing so, we can better understand their strengths and limitations. ## winowhy experiments in this section, we conduct experiments to investigate whether current models can understand how human beings solve wsc questions. ## related work as an important knowledge resource for many artificial intelligence systems, commonsense knowledge covers various knowledge categories like causality @xcite , reasoning (schubert, 2015), property @xcite , and quantity @xcite , and has been proven crucial in many downstream tasks like question answering (lin et al., 2019), dialogue system @xcite , reading comprehension @xcite , and pronoun coreference resolution @xcite . among all these tasks, winograd schema challenge (wsc) @xcite is viewed as one of the most challenging ones because solving wsc questions typically requires inference over various kinds of commonsense knowledge. conventionally, people tried to solve wsc questions in an unsupervised way by leveraging either search engines @xcite , linguistic knowledge @xcite , or language representation models @xcite . experimental results showed that these models still cannot fully solve the problem but we are not clear about how to further improve them. one important reason behind this is that the conventional definition of commonsense knowledge is too vague and thus we are not clear about what kinds of knowledge are still challenging for current commonsense reasoning models. in this paper, we use the wsc task as the breaking point to conduct a deep diagnosis of essential commonsense knowledge types, which sheds some light on how to achieve a better commonsense reasoning system in the future. ## conclusion in this paper, we presented the first deep diagnosis of essential commonsense knowledge for answering winograd schema challenge questions. by doing so, we better understand the strengths and limitations of current commonsense reasoning models. more importantly, we better know about what kinds of commonsense knowledge are required to be acquired for better commonsense reasoning. on top of the collected reasons, we develop a new task called winowhy, which requires models to select the plausible reasons for answering wsc questions. experiments show that even though current models have gained significant improvement over the original wsc task, they still cannot fully understand the reasons behind.