W ino L ogic: A Zero-Shot Logic-based Diagnostic Dataset for W inograd S chema C hallenge (2021)

Abstract: The recent success of neural language models (NLMs) on the Winograd Schema Challenge has called for further investigation of the commonsense reasoning ability of these models. Previous diagnostic datasets rely on crowd-sourcing which fails to provide coherent commonsense crucial for solving WSC problems. To better evaluate NLMs, we propose a logic-based framework that focuses on high-quality commonsense knowledge. Specifically, we identify and collect formal knowledge formulas verified by theorem provers and translate such formulas into natural language sentences. Based on these true knowledge sentences, adversarial false ones are generated. We propose a new dataset named WinoLogic with these sentences. Given a problem in WinoLogic, NLMs need to decide whether the plausible knowledge sentences could correctly solve the corresponding WSC problems in a zero-shot setting. We also ask human annotators to validate WinoLogic to ensure it is human-agreeable. Experiments show that NLMs still struggle to comprehend commonsense knowledge as humans do, indicating that their reasoning ability could have been overestimated.

## introduction recently, large-scale neural language models (nlms) have shown promising results on many challenging tasks, including the winograd schema challenge (wsc), a multiple-choice coreference resolution problem that is designed to test natural language understanding and reasoning with commonsense knowledge @xcite . each problem depicts a daily situation with an ambiguous pronoun to be resolved. for example, in the wsc sentence "the trophy doesn't fit into the brown suitcase because it is too large", the pronoun "it" could plausibly refer to either "the trophy" or "the suitcase". the lack of training examples in wsc has driven researchers to fine-tune nlms on similar datasets, which achieved near-human performance @xcite . still, the opacity of the nlms has raised questions about whether they truly capture common sense or merely exploit biases. such concerns were confirmed for the natural language inference task, as @xcite discovered an lm could provide correct answers using fallible heuristics. to investigate if nlms understand the reasons for solving wsc, @xcite crowd-sourced reasons for the wsc problems and built a new dataset winowhy. while in the correct direction, crowd-sourcing is far from perfect for collecting reliable explanations. consider the following winowhy example. example 1 (winowhy). the trophy doesn't fit into the brown suitcase because it is too large. the it is more likely to refer to the trophy than the brown suitcase because (a) the brown suitcase is too large. (b) it is a game. (c) the trophy is not fit into the suitcase. given a wsc sentence with its correct answer, crowd-workers and nlms are prompted to write a piece of text as the "reason" to justify the answer, e.g., (a), (b), and (c). another group of crowdworkers was tasked to decide if each reason explains the answer. in example 1, all three reasons are labeled as correct. however, upon closer inspection, all of them are incorrect: reason (a) contradicts with the correct answer; (b) is irrelevant; (c) circularly repeats the situation. these characteristics, namely, the lack of coherence, the lack of relevance, and circularity render the explanations useless @xcite . the discouraging quality of these reasons has led us to the logic-based path. compared to casually crowd-sourced justifications, we regard reliable commonsense knowledge as the key to solving wsc, and thus the key to constructing a diagnostic dataset. we employ first-order logic (fol) to en-code commonsense knowledge for two purposes: in this paper, we propose winologic, a diagnostic dataset to evaluate nlms' ability to understand and reason with commonsense knowledge for wsc. common sense in winologic is encoded in natural language sentences named knowledge sentences, based on their counterparts in fol formulas. consider the following example. example 2 (knowledge sentence). when someone is trying to fit an object x into a container y, if x is too large then it wouldn't be possible to fit x into y. this knowledge sentence is translated from a logical formula whose reliability is verified by a theorem prover. it is coherent, relevant and unambiguous compared to those in example 1. moreover, such knowledge also applies to other similar situations due to its abstract nature, while justifications such as those in winowhy are bound to specific entities in the problem. translating formulas into sentences not only improves the readability of the commonsense knowledge but also paves the way for evaluation. since nlms might not be exposed to variable symbols (x, y, etc.) in pre-training, we provide two additional variants with variable-free knowledge sentences. to generate reliable knowledge sentences, we use fol to encode the knowledge and perform verifications before converting it into readable text. specifically, we follow these steps: (1) provide formalizations and suitable commonsense knowledge formulas for wsc problems; (2) verify these knowledge formulas using a theorem prover; (3) translate these formulas into knowledge sentences in natural language; (4) generate adversarial false knowledge sentences with only subtle differences. winologic is then constructed as a text classification dataset, where each problem is composed of a wsc sentence, its answer, and a plausible knowledge sentence. the task is to decide whether the knowledge supports the correct answer. additionally, we ask human annotators to validate the dataset to ensure that the knowledge sentences are human-agreeable. therefore the reliability of winologic is guaranteed by both the formal verification in fol and human validation. winologic, with its high-quality knowledge, is suitable for diagnosing nlms in a zero-shot setting. just as educators don't need large-scale examinations to test students, the small scale of winologic doesn't invalidate its usage. what matters in tests and evaluation datasets is the quality of the problems. focusing on the small amount of knowledge that is key for solving wsc, wino-logic provides more reliable diagnostic problems. this also aligns with the few-shot or zero-shot evaluation settings, where large-scale task-specific fine-tuning is intentionally avoided. after all, people could solve wsc273 tasks without large-scale training or fine-tuning @xcite . experiments with three high-performing nlm architectures show that they still struggle to understand knowledge in winologic. even when they are fine-tuned on winowhy, we observe no improvement on winologic, suggesting that they may learn little from the crowd-sourced reasons. ## related work since its inception, wsc is gaining more and more attention. in 2016, a competition was held at the ijcai-16 conference, but no systems qualified for the second round, as accuracies were below 60% @xcite . @xcite collected 285 wsc problems available online, the first 273 of them are commonly referred to as wsc273. rahman and ng (2012) proposed the definite pronoun resolution (also known as wscr) dataset, containing 1886 problems that are considered easier than those in wsc273. @xcite crowd-sourced winogrande, a large-scale wsclike dataset with 44k problems. an nlm-based filtering algorithm was used to identify "unbiased" problems among them. wsc problems are also incorporated into nlp benchmarks. the super-glue benchmark includes a subtask superglue-wsc (804 problems), where wsc problems are cast into binary classification problems @xcite . while wsc273 and superglue-wsc have significant overlap, they are not entirely the same. recent sota results are achieved with finetuned nlms. two such examples are the bert and roberta models @xcite , which have shown improvements after being fine-tuned on similar datasets. specifically, the roberta models have over 90% accuracy on superglue-wsc when fine-tuned on winogrande @xcite . there are also analyses about the problems in wsc273 and human performance. the very first human baseline evaluation reports 92% accuracy @xcite . they noticed that for certain problems, the correct answers are only evident after the pair of questions are revealed together. @xcite discovered the associative subset of wsc problems where statistical correlations between the problem and the candidates could reveal the answer. @xcite asked crowd-workers to classify wsc problems into different knowledge types, and they discovered spatial knowledge is particularly difficult for nlms. @xcite experimented with several formulations of wsc problems and reported that task-framing has an impact on the performance of nlms, e.g., multiple-choice setting leads to better performance. ## logic-based commonsense knowledge for wsc273 in this section, we introduce how explicit commonsense knowledge could be written in fol and how verification is done using theorem provers. we first formalize each wsc problem into a set of logical formulas, then we provide the necessary commonsense knowledge. to ensure the correctness and validity of such knowledge, we use z3 (de moura and bjørner, 2008), a theorem prover, to ensure that the formulas of knowledge could be used to derive correct answers. situation calculus. we use the situation calculus (sc), a variant of fol, as the representation language @xcite . sc provides suitable constructs for modeling dynamic worlds: entities in sc belong to either objects, actions or situations. both the wsc scenario and the commonsense knowledge are represented using sc. example 3. "the father couldn't lift his son because he was too weak." the scene in example 3 before the father tried to lift his son involves the following entities: • objects: f ather and son; to describe this scenario in sc, we use special predicates where the last argument is always a situation. for example, we could say that the son is not lifted in the initial situation s 0 : ¬lif ted(son, s 0 ). ## knowledge sentences and winologic in this section, we present the creation of positive and negative knowledge sentences as in figure 1 , then we describe the resulting winologic. "the father" ## evaluation in this section, we describe baseline implementations for winologic and present the evaluation results. the dataset, code, and hyper-parameters are available in the supplementary materials. ## discussion although crowd-sourcing is a valuable tool to put collective intelligence to work, it may not be an ideal method to produce reliable explanations for wsc. firstly, generating explanatory text for even simple problems is already immensely more difficult than classifying its correctness. people often apply common sense subconsciously, and thus could potentially trivialize its significance. for example, we often take the closed-world assumption for granted. secondly, it requires a deep reasoning process for solving many wsc problems. in this case, precision and unambiguity would be vital, which is not exactly the strong suit of crowdsourcing. last but not least, from our prior experience in collecting expert-provided justifications similar to those in winowhy, we notice the same issues (e.g., circularity and incoherence) in the collection, even though these nlp-experts were asked to meticulously explain the reason for wsc problems. in this paper, we leverage the unambiguity of fol which enables us to precisely represent the commonsense knowledge, while theorem provers are used to verify its validity, addressing the major issues in crowd-sourcing explanations. ## conclusion we show a knowledge-based perspective for evaluating recent improvements on the winograd schema challenge. instead of crowd-sourcing justifications, we take a more reliable logic-based route. by formalizing the wsc problems and the needed commonsense knowledge into fol formulas, we verify the capability of using such knowledge for solving the problems. to test recent neural language models, we translate the commonsense knowledge into natural language sentences. simple transformations create negative versions of these sentences. we propose wino-logic, a novel evaluation dataset of 562 wscrelated knowledge classification problems that are also human-validated. this new task requires systems to determine whether a knowledge sentence could be used to solve the corresponding wsc problem. the experiments show that recent language models do not have a correct understanding of the required knowledge, even though they are already fine-tuned on the similar winowhy dataset. our exploration suggests that the challenge of commonsense reasoning in wsc is still bottlenecked by the lack of machine common sense.