This paper proposes a general framework to combine context and commonsense knowledge for solving the Winograd Schema (WS) and Pronoun Disambiguation Problems (PDP). In the proposed framework, commonsense knowledge bases (e.g. cause-effect word pairs) are quantized as knowledge constraints. The constraints guide us to learn knowledge enhanced embeddings (KEE) from large text corpus. Based on the pre-trained KEE models, this paper proposes two methods to solve the WS and PDP problems. The first method is an unsupervised method, which represents all the pronouns and candidate mentions in continuous vector spaces based on their contexts and calculates the semantic similarities between all the possible word pairs. The pronoun disambiguation procedure could then be implemented by comparing the semantic similarities between the pronoun (to be resolved) and all the candidate mentions. The second method is a supervised method, which extracts features for all the pronouns and candidate mentions and solves the WS problems by training a typical mention pair classification model. Similar to the first method, the features used in the second method are also extracted based on the KEE models. Experiments conducted on the available PDP and WS test sets show that, these two methods both achieve consistent improvements over the baseline systems. The best performance reaches 62% in accuracy on the PDP test set of the first Winograd Schema Challenge. In recent years, many AI challenges or competitions have been proposed to help evaluate the cognitive levels of machines (Levesque, Davis, and Morgenstern 2011; Weston et al. 2015; Clark 2015) . Among those challenges, the Winograd Schema Challenge (WSC) has been proposed as an alternative to the Turing Test (Levesque, Davis, and Morgenstern 2011). Turing first introduced the notion of testing a computer system's intelligence by assessing whether it could make a human judge think that she was conversing with a human rather a computer (Turing 1950). However, some recent efforts have merely on engaged surface-level conversation tricks to fool humans who do not delve deeply enough into a conversation, and make them think they are speaking to another human being (Veselov, Demchenko, and Ulasen ; Warwick and Shah 2014). To fix this issue, WSC is claimed to be a more suitable task which does not rely on human's Copyright c 2017 subjective assessment. A Winograd schema (WS) question is a pair of sentences that differ only in one or two words which results in a different resolution of coreference. A wellknown example is "The city council refused the demonstrators a permit because they feared violence," where "they" refers to the council. But if one changes the verb "feared" to "advocated", the computer needs to know that "they" refers to "demonstrator", if it possesses real intelligence. To solve the problem, common sense knowledge is essential. Towards solving the final solution for WS problem, a similar test set, called pronoun disambiguation problem (PDP) is designed as the first round of the Winograd Schema Challenge (Morgenstern, Davis, and Ortiz Jr 2016). A typical example is "Mrs. March gave the mother tea and gruel, while she dressed the little baby as tenderly as if it had been her own." One way to reason that she in she dressed refers to Mrs. March and not the mother, is to realize that the phrase "as if it were her own" implies that it (the baby) is not actually her own; that is, she is not the mother and must, by process of elimination, be Mrs. March. Similar to the Winograd schemas, a substantial amount of commonsense knowledge appears to be necessary to disambiguate pronouns. This paper proposes neural network models to combine context and commonsense knowledge to solve both the WS and PDP problem. The main ideas are two-fold: 1) The first is to leverage context effectively. We believe that since context is a key information for learning word meanings, it should be useful for disambiguating the Winograd schema problems. In general, modeling context and learning word meanings could be very efficient through unsupervised learning that leverages large amounts of free texts. 2) In the common sense respect, we describe a simple but effective method utilizing commonsense knowledge. To jointly consider those two aspects, this paper proposes to combine context and commonsense knowledge through neural networks through a knowledge enhanced embeddings (KEE) framework. This paper further proposes two methods: 1) The first is the an unsupervised semantic similarity method (USSM), which represents all the pronouns and candidate mentions by composing their contexts based on the pre-trained knowledge enhanced embeddings. We then calculate the semantic similarities between the embedding vectors of the pronoun under concern and all candidate mentions. The candidate with largest semantic similarity with respect to the arXiv:1611.04146v1 [cs.AI] 13 Nov 2016 pronoun will be predicted as the answer. 2) The second method is a neural knowledge activated method (NKAM), which extracts features based on the KEE models and trains a mention pair classifier with neural networks. For the experiment section, this paper conducts experiments on the official datasets of the first WSC challenge, including the PDP test set provided by commonsensereasoning.org, as well as a set of Winograd schemas manually created by (Levesque, Davis, and Morgenstern 2011). Experimental results all indicate that the proposed KEE method (combing context and knowledge) performs better comparing with the baseline models. The remainder of the paper will start with introducing the main motivation. After that, we introduce the main methods proposed to solve the WS and PDP problems. We then present all the experiments, including setup, datasets, and results, before we conclude this work. In this section, we introduce the main motivation of this work. We will firstly present the main problems we aim to solve, i.e., the Winograd schema (WS) and the Pronoun Disambiguation Problem (PDP). After that, detailed descriptions would be given to illustrate our motivation. The Winograd schema (WS) evaluates a system's commonsense reasoning ability based on a traditional, very specific natural language processing task: coreference resolution (Saba 2015). The WS problems are carefully designed to be a task that cannot be easily solved without commonsense knowledge. In fact, even the solution of traditional coreference resolution problems rely on semantics or world knowledge (Strube 2016). As described in (Levesque, Davis, and Morgenstern 2011), a WS is a small reading comprehension test involving a single binary question. • Joan made sure to thank Susan for all the help she had given. Who had given the help? -Answer A: Joan -Answer B: Susan -Correct Answer: B The correct answers to the above question are obvious for human beings. In each of the questions, the corresponding WS has the following four features: 1. Two parties are mentioned in a sentence by noun phrases. They can be two males, two females, two inanimate objects or two groups of people or objects. 2. A pronoun or possessive adjective is used in the sentence in reference to one of the parties, but is also of the right sort for the second party. 3. The question involves determining the referent of the pronoun or possessive adjective. Answer A is always the first party mentioned in the sentence (but repeated from the sentence for clarity), and Answer B is the second party. 4. There is a word (called the special word) that appears in the sentence and possibly the question. When it is replaced by another word (called the alternate word), everything still makes perfect sense, but the answer changes. In this example, if we change the word given to received, the answer changes to A (i.e., Joan) since in our commonsense knowledge, we think that a person who receives help should makes sure to thank the person who provides help to him. The pronoun disambiguation problems (PDP) are complex coreference resolution problems, which are taken directly or modified from examples found in literature, biographies, autobiographies, essays, news analyses, and news stories; they may need some manual processing (Morgenstern, Davis, and Ortiz Jr 2016) . Here is one typical PDP example: • The Dakota prairie lay so warm and bright under the shining sun that it did not seem possible that it had ever been swept by the winds and snows of that hard winter. -Snippet: it had ever been swept -Answer A: the prairie -Answer B: the sun -Correct Answer: A In the PDP problem, the pronoun to be resolved is highlighted in bold. It is repeated again, with a snippet of context, and with several candidate answers, in the line following the passage. In the example shown here, we know that the prairie (rather than the sun) would be more likely to be swept by the winds. A difference between PDP and WS problems is that, the number of candidate noun phrases in each PDP problem would not always be two, but can be three, four, or even more. Therefore, the random-guess accuracy in the PDP problems will be less than 50% while the accuracy of a random-guess for WS is 50%. Solving WS or PDP problems is not easy since it requires commonsense knowledge. In the paper, we propose to combine context and commonsense knowledge through neural networks for solving both problems. The main motivation is described as follows. First, since context is key for learning word meaning (Harris 1954; Miller and Charles 1991) , we represent the pronoun and all the candidate mentions by their contexts. For instance, in the aforementioned PDP example, we represent the word prairie by The Dakota and lay so warm and bright. Meanwhile, the word sun could be represented by and bright under the shining and that it did not seem. Based on this, the pronoun disambiguation problems is solved by calculating the semantic similarities between the representations of the pronoun and all the corresponding candidates. Second, relying only on context is not good enough for tackling the WS and PDP problems. It is essential to find an effective strategy to combine context and commonsense knowledge. Therefore, in this paper, we propose a knowledge enhanced embedding (KEE) model. As shown in Figure 1,
The current best approach makes use of the skip-gram model to learn word representations REF .
Combing Context and Commonsense Knowledge Through Neural Networks for Solving Winograd Schema Problems.
ArXiv
ArXiv
Computer Science