The Sensitivity of Language Models and Humans to W inograd Schema Perturbations (2020)

Abstract: Large-scale pretrained language models are the major driving force behind recent improvements in perfromance on the Winograd Schema Challenge, a widely employed test of commonsense reasoning ability. We show, however, with a new diagnostic dataset, that these models are sensitive to linguistic perturbations of the Winograd examples that minimally affect human understanding. Our results highlight interesting differences between humans and language models: language models are more sensitive to number or gender alternations and synonym replacements than humans, and humans are more stable and consistent in their predictions, maintain a much higher absolute performance, and perform better on non-associative instances than associative ones.

## introduction large-scale pre-trained language models have recently led to improvements across a range of natural language understanding (nlu) tasks @xcite @xcite , but there is some scepticism that benchmark leaderboards do not represent the full picture @xcite @xcite . an open question is whether these models generalize beyond their training data samples. in this paper, we examine how pre-trained language models generalize on the winograd schema challenge (wsc). named after terry winograd, the wsc, in its current form, was proposed by @xcite as an alternative to the turing test. the the man couldn't lift his son because he was so heavy. the man couldn't lift his son because he was so weak. the men couldn't lift their sons because they were so heavy. the men couldn't lift their sons because they were so weak. task takes the form of a binary reading comprehension test where a statement with two referents and a pronoun (or a possessive adjective) is given, and the correct antecedent of the pronoun must be chosen. examples are chosen carefully to have a preferred reading, based on semantic plausibility rather than co-occurrence statistics. wsc examples come in pairs that are distinguished only by a discriminatory segment that flips the correct referent, as shown in figure 1a . @xcite define a set of qualifying criteria for instances and the pitfalls to be avoided when constructing examples (see §3.2). these combine to ensure an instance functions as a test of what they refer to as 'thinking' (or common sense reasoning). recent work has reported significant improvements on the wsc @xcite . as with many other nlu tasks, this improvement is primarily due to largescale language model pre-training, followed by fine-tuning for the target task. we believe that further examination is warranted to determine whether these impressive results reflect a funda-mental advance in reasoning ability, or whether our models have learned to simulate this ability in ways that do not generalize. in other words, do models learn accidental correlations in our datasets, or do they extract patterns that generalize in robust ways beyond the dataset samples? in this paper, we conduct experiments to investigate this question. we define a set of lexical and syntactic variations and perturbations for the wsc examples and use altered examples (figure 1b ) to test models that have recently reported improved results. these variations and perturbations are designed to highlight the robustness of human linguistic and reasoning abilities and to test models under these conditions. contributions we introduce a new winograd schema dataset for evaluating generalization across seven controlled linguistic perturbations. 1 we use this dataset to compare human and language model sensitivity to those perturbations, finding marked differences in model performance. we present a detailed analysis of the behaviour of the language models and how they are affected by the perturbations. finally, we investigate the effect of fine-tuning with large task-specific datasets, and present an error analysis for all models. ## related work probing datasets previous studies have explored the robustness of ml models towards different linguistic phenomena @xcite , e.g., by creating challenge datasets such as the one introduced here. when predicting subjectverb agreement, @xcite found that inserting a relative clause hurt the performance of recurrent networks. 2 a large body of research has since emerged on probing pre-trained (masked) language models for linguistic structure @xcite @xcite and analysing them via comparison to psycholinguistic and brain imaging data @xcite @xcite gauthier and 1 code and dataset can be found at: https://github. com/mhany90/enhanced_wsc/ 2 this contrasts with our results with transformer-based architecture and is probably explained by memory loss in recurrent networks trained on short sequences. similarly, gulordava et al. (2018) tested whether a recurrent neural network can predict long-distance number agreement in various constructions comparing natural and nonsensical sentences where rnns cannot rely on semantic or lexical cues. levy, 2019 ). other recent work has attempted to probe these models for what is referred to as common sense or factual knowledge @xcite . their findings show that these models do indeed encode such knowledge and can be used for knowledge base completion or common sense mining from wikipedia. clever hans a considerable amount of work has also been devoted to what might be described as the clever hans effect. this work has aimed to quantify the extent to which models are learning what we expect them to as opposed to leveraging statistical artifacts. this line of work has to date revealed significant problems (and some possible solutions to those problem) with reading comprehension datasets @xcite , natural language inference datasets @xcite @xcite @xcite , and the story cloze challenge @xcite , among others. winograd schema challenge trinh and le (2018) first proposed using neural language models for the wsc, achieving an accuracy of 63.7% using an ensemble of 14 language models. @xcite and @xcite fine-tune bert @xcite on the pdp (rahman and ng, 2012) and an automatically generated maskedwiki dataset, reaching an accuracy of 71.1% and 72.5% respectively. meanwhile, @xcite report an accuracy of 70.7% without fine-tuning using the gpt-2 language model. most recently, @xcite present an adversarial filtering algorithm which they use for crowd-sourcing a large corpus of wsc-like examples. fine-tuning roberta @xcite on this, they achieve an accuracy of 90.1%. in an orthogonal direction, @xcite presented a timely critical treatment of the wsc. they classified the dataset examples into associative and non-associative subsets, showing that the success of the lm ensemble of trinh and le (2018) mainly resulted from improvements on the associative subset. moreover, they suggested switching the candidate referents (where possible) to test whether systems make predictions by reasoning about the "entirety of a schema" or by exploiting "statistical quirks of individual entities". in a similar spirit, our work is a controlled study of robustness along different axes of linguistic variation. this type of study is rarely possible in nlp due to the large size of datasets used and the focus on obtaining improved results on said datasets. like a carefully constructed dataset which is thought to require true natural language understanding, the wsc presents an ideal testbed for this investigation. ## perturbations we define a suite of seven perturbations that can be applied to the 285 wsc examples, which we refer to as the original examples. these perturbations are designed to test the robustness of an answer to semantic, syntactic, and lexical variation. each of the perturbations is applied to every example in the wsc (where possible), resulting in a dataset of 2330 examples, an example of each type is shown in tense switch (ten) most wsc instances are written in the past tense and thus are changed to the present continuous tense (247 examples). the remaining 34 examples are changed from the present to the past tense. number switch (num) referents have their numbers altered: singular referents (and the relevant pronouns) are pluralised (223 examples), and plural referents are modified to the singular (30 examples). sentences with names have an extra name added via conjunction; eg. "carol" is replaced with "carol and susan". possessives only mark possession on the second conjunct ("john and steve's uncle" rather than "john's and steve's uncle"). gender switch (gen) each of the referents in the sentence has their gender switched by replacing their names with other randomly drawn frequent english names of the opposite gender. foot_0 92% of the generated data involved a gender switch for a name. though humans may be biased towards gender @xcite @xcite , the perturbations do not introduce ambiguity concerning gender, only the entity. 101 examples were switched from male to female, and 55 examples the other way around. voice switch (vc) all wsc examples, except for 210 and 211, are originally in the active voice and are therefore passivized. 210 and 211 are changed to the active voice. 65 examples could not be changed. passive voice is known to be more difficult to process for humans @xcite . relative clause insertion (rc) a relative clause is inserted after the first referent. for each example, an appropriate clause was constructed by first choosing a template such as "who we had discussed" or "that is known for" from a preselected set of 19 such templates. an appropriate ending, such as "who we had discussed with the politicians" is then appended to the template depending on the semantics of the particular instance. relative clauses impose an increased demand on working memory capacity, thereby making processing more difficult for humans @xcite . ## experimental protocol our experiments are designed to test the robustness of language models to the winograd schema perturbations described in the previous section. evaluation models are evaluated using two types of measures. the first is accuracy. for each of the perturbations, we report (a) the accuracy on the perturbed set (perturbation accuracy), (b) the difference in accuracy on the perturbed set and on the equivalent subset of original dataset: foot_3 ∆ acc. = perturbation accuracy -original subset accuracy, and (c) pair accuracy, defined as the number of pairs for which both examples in the pair are correctly answered divided by the total number of pairs. the second measure is stability, s. this is the proportion of perturbed examples p for which the predicted referent is the same as the original prediction p: since the perturbations do not alter the correct referent, this provides a strong indication of robustness towards them. baseline we take the unigram pmi between candidates and discriminatory segments (see §3.2) as a baseline. we expect that this simple baseline will perform well for instances with a high level of associativity but not otherwise. language models our analysis is applied to three out-of-the-box language models (lms): bert @xcite , roberta @xcite , and xlnet @xcite . these models are considered to be the state-of-the-art for the wide variety of natural language understanding tasks found in the glue @xcite and superglue @xcite benchmarks. we use the large pre-trained publicly available models @xcite . 7 fine-tuned language models we also examine the effect of fine-tuning language models. bert+ww uses bert fine-tuned on the maskedwiki and wscr datasets which consist of 2.4m and 1322 examples (kocijan et al., 2019), and roberta+wg is fine-tuned on winogrande xl, which consists of 40,938 adversarially filtered examples @xcite . both finetuned models have been reported by recent work to achieve significant improvements on the wsc. ## conclusion we presented a detailed investigation of the effect of linguistic perturbations on how language models and humans perform on the winograd schema challenge. we found that compared to out-of-the-box models, humans are significantly more stable to the perturbations and that they answer non-associative examples with higher accuracy than associative ones, show sensitivity to wsc pair complementarity, and are more sensitive to sentence-level (as opposed to wordlevel) perturbations. in an analysis of the behaviour of language models, we observe that there is a preference for referents in the object role and that the models do not always consider the discriminatory segments of examples. finally, we find that fine-tuning language models can lead to much-improved accuracy and stability. it remains an open question whether this task-specific approach to generalisation constitutes a true advancement in "reasoning". fine-tuning a model on a rather large number of examples similar to the wsc leads to increased robustness, but this stands in stark contrast to humans, who are robust to the perturbations without having been exposed to similar examples in the past. adv gen num org rc syn/na ten vc perturbation 50 60 70 80 90 100 accuracy model bert bert+ww roberta roberta+wg xlnet humans pmi scheme a n figure 7: perturbation accuracy on the associative (a) and non-associative (n) subsets of the data.