An Analysis of Dataset Overlap on W inograd-Style Tasks (2020)

Abstract: The Winograd Schema Challenge (WSC) and variants inspired by it have become important benchmarks for common-sense reasoning (CSR). Model performance on the WSC has quickly progressed from chance-level to near-human using neural language models trained on massive corpora. In this paper, we analyze the effects of varying degrees of overlaps that occur between these corpora and the test instances in WSC-style tasks. We find that a large number of test instances overlap considerably with the pretraining corpora on which state-of-the-art models are trained, and that a significant drop in classification accuracy occurs when models are evaluated on instances with minimal overlap. Based on these results, we provide the WSC-Web dataset, consisting of over 60k pronoun disambiguation problems scraped from web data, being both the largest corpus to date, and having a significantly lower proportion of overlaps with current pretraining corpora.

## introduction the original purpose of the winograd schema challenge was to serve as an alternative turing test to evaluate an automatic system's capacity for common-sense inference @xcite . as an example: while recent work has included some analysis on the effects of 13-gram overlaps between pretraining and test instances for the wsc @xcite , a deeper look into how the degree of overlap (and how this can be defined) affects language models' performance is critical to revealing models' reasoning and inference functions. for example, in 1b), useful knowledge instances in the pretraining corpora may occur as: studying how models make use of these training instances, in correlation with their overlap with test instances, can provide insight on the roles and downstream influence of exact duplicates (which may be useful if memorized) or highly relevant but distinctly expressed knowledge (useful by retrieval and analogy). in turn, this insight could be used to improve training approaches for models meant to exhibit common-sense reasoning. contributions: in this work, we address the above issues in csr modeling by devising a mechanism to score train-test overlap according to a schematization based on bm25, a popular information retrieval function for text matching @xcite . we use the mechanism to sub-divide test-set instances according to these overlaps. we find that a significant drop in classification accuracy occurs when models are evaluated on the subset with no overlap (we see drops between 3% and 10%, depending on the model, test set, and degree of overlap). based on this result, we develop the knowref-60k dataset, consisting of 64,301 difficult pronoun disambiguation problems. it is the largest corpus to date for wsc-style common sense reasoning and exhibits a significantly lower proportion of overlaps with current pretraining corpora. 1 2 related work previous work on the difficulty of instances in the wsc and its variants includes the study by @xcite , who classified data points into various meaningful subsets. they showed that the success of a then-state-of-the-art lm ensemble (trinh and le, 2018) resulted mainly from improvements on simpler "associative" instances. similarly, experiments by @xcite show that models are sensitive to linguistic perturbations of winograd-style examples. new datasets have been proposed to circumvent issues of unintentionally easy test instances, including winogrande @xcite , a scaled wsc-variant debiased against roberta, and knowref @xcite , which consists of naturally occurring sentences that are free of wsc-specific stylistic quirks. given the recent popularity of large, internet-scale datasets for pretraining neural language models, there is an increasing concern that test instances in a downstream task may inadvertently appear in the pretraining corpus. this is a form of data contamination. one of the earliest works that trained a language model on common crawl data identified and removed a training documents that overlapped with one of their evaluation datasets @xcite . other work, such as gpt-2 @xcite and @xcite , conducted post-hoc overlap analysis on csr benchmarks based on a conservative threshold for contamination-specifically, instance pairs that have a 13-gram overlap. they found the effects of this 13-gram contamination to be negligible. on the other hand, a recent work in computer vision found a significant effect of near-duplicates on test performance in an important benchmark, leading to the proposal of a duplicate-free and demonstrably more difficult dataset @xcite . to our knowledge, no work has investigated the effect of varying degrees of overlap between pretraining and csr test instances for the state-of-the-art transformer-based models (bert and roberta). any such investigation must include formulating a more precise definition of contamination. methods for purging easy instances from csr benchmarks have been developed recently: for example, the algorithmic bias reduction of test sets proposed by @xcite removes instances from the test set with exploitable annotation artifacts. these techniques depend on pre-computed neural network embeddings of a particular model, and so may be difficult for that model alone but not for previous or up-and-coming models. as the work of @xcite and the follow-up by @xcite have shown, adversarial filtering must be iteratively re-adapted to newer models that may be immune to previous filtering. this may be costly. adversarial filtering and related debiasing techniques also do not provide much insight on why certain test instances are filtered out. our proposed method for data purging is interpretable and model-independent, and can be further supplemented with existing debiasing algorithms like aflite @xcite to ensure that benchmarks remain challenging. ## hunting for overlaps our procedure for identifying train-test overlaps consists of three main steps: (1) parsing a test instance into its core components, (2) formulating a query using a schema derived from the parse, and (3) quantifying the degree of overlap between a train-test pair using an overlap scoring mechanism. ## conclusion we proposed an automatic method of scoring the degree of overlap between test-train instances in csr benchmarks and demonstrated that models generally perform better on test instances with high degrees of overlap in pretraining corpora. in response to our findings, we released a more difficult and largestto-date wsc-style test set called knowref-60k. we ensured that its overlaps with pretraining data are minimal by using a text source not contained in the suite of common pretraining corpora and by basing its construction on naturally-occurring sentences with no direct influence from wsc-like patterns. our findings suggest that, for better or worse, highly similar pretraining instances have a significant influence on the performance of state-of-the-art transformer-based architectures. coupled with the large fraction of exact copies or highly-overlapping instances that currently exist in csr test sets, this effect may bias the evaluation and development of deep learning approaches for common-sense reasoning. on a positive note, models still performed significantly better than random on non-overlapping test instances, and their relative rankings did not change. this suggests that the community's efforts do not seem to have overfit to the presence of overlaps yet: transformer-based language models still provide higher accuracy on the subset of non-overlapping instances as well as on our knowref-60k dataset. we therefore encourage researchers to be cognizant of such overlaps as important factors affecting the performance of csr-based models, and to use this knowledge to have a clearer picture regarding the true capabilities of machine commonsense across these benchmarks. an important limitation of our analysis is that there is no guarantee that the overlapping subset is drawn from the same distribution as the original dataset, meaning that it is entirely possible that an emergent statistical bias (and neither retrieval nor memorization) caused the overlapping subset to be easier. accordingly, our work also raises some open questions worthy of pursuit; firstly, how do we more precisely identify the cases of memorization/retrieval? secondly, what constraints does the retrieval and/or memorization of models as a means of acquiring common sense present in terms of their capabilities/robustness?