Solving the Challenge Set without Solving the Task: On W inograd Schemas as a Test of Pronominal Coreference Resolution (2024)

Abstract: Challenge sets such as the Winograd Schema Challenge (WSC) are used to benchmark systems’ ability to resolve ambiguities in natural language. If one assumes as in existing work that solving a given challenge set is at least as difficult as solving some more general task, then high performance on the challenge set should indicate high performance on the general task overall. However, we show empirically that this assumption of difficulty does not always hold. In particular, we demonstrate that despite the strong performance of prompted language models (LMs) on the WSC and its variants, these same modeling techniques perform relatively poorly at resolving certain pronominal ambiguities attested in OntoNotes and related datasets that are perceived to be easier. Motivated by these findings, we propose a method for ensembling a prompted LM with a supervised, task-specific system that is overall more accurate at resolving pronominal coreference across datasets. Finally, we emphasize that datasets involving the same linguistic phenomenon draw on distinct, but overlapping, capabilities, and evaluating on any one dataset alone does not provide a complete picture of a system’s overall capability.

## introduction the winograd schema challenge (wsc; @xcite is a challenge set of ambiguous pronominal coreference resolution (pcr) problems, one of many popular challenge sets used to evaluate nlp systems (e.g., @xcite @xcite . challenge sets are constructed to consist of relatively difficult instances of some more general task. in many cases, systems' performance on challenge sets is considered in isolation of performance on the broad range of ambiguous expressions attested in natural corpora on which the general task being studied … mrs. long says that netherfield is taken by a young man of large fortune from the north of england; that he came down on monday as so much delighted with it, that he agreed with mr. morris immediately; … jim yelled at kevin because he was so upset. ## related work pcr is broadly the task of determining which linguistic expressions refer to the same discourse entity as a given pronominal expression @xcite . see @xcite and @xcite for related surveys. proposed systems for resolving pronominal coreference have traditionally relied on heuristic rules often in combination with unsupervised statistical patterns of handcrafted features @xcite @xcite . more recently, lm-based approaches have been proposed including: lms finetuned on supervised training data @xcite , weakly supervised lms @xcite , and prompting lms by formatting pcr as either a cloze task @xcite ) or question answering @xcite @xcite . the ability of a system to perform pcr has been evaluated generally on: 1) collections of ambiguous pronouns attested in natural text @xcite @xcite , 2) the subsets of larger coreference resolution datasets that include pronominal coreference @xcite @xcite , and 3) challenge sets composed of wsc-like instances @xcite @xcite . the wsc and inspired datasets have been adopted by researchers studying the more general task of coreference resolution to be used as challenge sets in addition to more canonical evaluations such as ontonotes @xcite @xcite . such work has shown that systems designed for coreference resolution perform poorly on the wsc. we adopt the perspective of this line of work and view wsc-like datasets as challenge sets of pcr. recent advances in language modeling have proven accurate at resolving wsc instances when compared to earlier approaches, in some cases nearing approximates of human accuracy @xcite @xcite . however, similar techniques have been shown to be less accurate than supervised models when evaluated on established evaluations of the general task of coreference resolution @xcite @xcite . our work diverges from these studies by focusing specifically on pcr rather than the broader concept of coreference which has multiple competing definitions @xcite . ## method in this section, we formulate the problem of pronominal coreference resolution (pcr) and provide a high-level description of how system accuracy is evaluated. we also formalize the assumptions commonly made when evaluating on challenge sets so that we can explicitly test if these assumptions hold. ## experiments in this section, we describe our experimental setup in detail which tests whether the challenge set assumption holds empirically. we compare prompted lms that are known to be accurate at the wsc against task-specific systems known to be accurate at resolving certain attested pronouns ( §4.1). systems are evaluated across 11 datasets spanning both attested and wsc-like instances ( §4.2). ## results we first present results comparing zero-shot prompting methods with the unsupervised dcoref system. we then compare the best performing prompting method against the supervised maverick coreference resolution system and a supervised llama 3.1 baseline. across all figures, error bars represent 90% confidence intervals. results are presented on the corresponding test splits using the best model configuration. additional details are presented in app. d. comparing prompted lms against earlier unsupervised systems, the challenge set assumption does hold. results for the fully unsupervised systems are presented in figure 4 . we observe that generally prompted lms, which outperform dcoref on the wsc variants, also outperform dcoref on datasets of attested pronominal expressions. we also see that model performance is sensitive to the prompt format. exceptions are on the pdp dataset, whose small size makes it difficult to draw generalizable conclusions, and in the case of the doc prompt, which has high variance across datasets. le and ritter (2023) similarly found that llama models did not consistently generalize with the doc prompt template. in figure 5 , we compare accuracies of various lms using the qa prompt template. our conclusion, that prompted lms outperform dcoref on both constructed and attested instances, is consistent across lm families. however, when comparing prompted lms against a supervised coreference resolution system, the challenge set assumption does not hold. in figure 6 , we present the accuracy of llama 3.1 we find that lms generally outperform dcoref on both attested and constructed instances. for this experiment, we consider the instructiontuned version of llama 3.1 with 32 in-context examples as a prototypical example of an lm as evaluated on wsc-like datasets. when we compare against a supervised llama 3.1 base model, trained on the winogrande and dpr training sets for 5k steps, the difference in accuracies across datasets is even more extreme. the exceptional case of knowref may be due to the fact that this dataset is constructed by perturbing attested pronominal expressions and may be overall more similar to collections of attested rather than constructed linguistic expressions. ## analysis our results thusfar do not answer the question of why the challenge set assumption does not hold. heuristic estimates of features such as number and animacy are typically required to agree between an antecedent and a pronoun in order for the two to be coreferring, but these features are never required to resolve wsc instances per their design. therefore, it may be the case that prompted lms are not sufficiently considering these features for the attested pcr problems. to test this hypothesis, we analyze to what extent lms could benefit from, or already incorporate implicitly, the use of such 1 70b instruct (32-shot) against the supervised maverick coreference resolution system. we observe that the challenge set assumption does not hold; that is, despite being generally more accurate on wsc-like datasets, the prompted lm is less accurate on datasets of pronominal expressions attested in natural corpora. from left to right, the first six datasets consist of attested examples, and the remaining five are wsc-like challenge sets. ## ensembling systems for better performance finally, we present results for our proposed ensembling method. this method is motivated as follows: because the challenge set assumption does not hold, prompted lms and task-specific systems have distinct strengths at pcr. ## discussion coreference resolution systems have traditionally struggled at resolving pronouns when the resolution depends on semantic knowledge related to high-order predicate-argument relations @xcite @xcite . meanwhile, our results suggest that resolving wsc instances, which are designed to explicitly rely on such knowledge, is in some ways relatively easier than other cases for prompted lms. therefore, our intuitions as a research community regarding what constitutes challenging examples may not always be aligned with the actual failure cases of newer modeling paradigms. consequently, we must be careful as a community to not interpret high performance on challenge sets as indicating that the more general task being studied can consistently be solved by a given system. the solvability of pcr. our experiments and results are not intended to make claims regarding the solvability of the task of pcr. it may be that alternative prompting formats exist for which llama models are relatively more accurate at resolving attested pronominal coreferences, and one would expect accuracy to increase with lm size. what our results do show, rather, is that existing approaches that are successful on the wsc and variants cannot generalize to all attested pcr problems. coreference and substitutability. by their design, wsc instances can be formatted as a clozestyle task where the correct antecedent is that which is most likely to be substituted for the ambiguous pronoun. substitutability and coreference are related but distinct concepts, however. while wsc instances are difficult in that they cannot be solved with the agreement of features between a pronoun and a candidate antecedent, they differ from some attested pcr problems in that for wsc instances the concept of coreference is aligned with substitutability. one possible hypothesis to explain our results is that this alignment is useful for solving the wsc. this hypothesis is based on the idea that substitutability can be formatted as a cloze-style task and is therefore closely aligned with the lm pretraining objective. ## conclusion the ability to disambiguate pronominal expressions is necessary for interpreting natural language and has been used extensively as a benchmark to evaluate models of semantics and discourse. in this work, we study several possible approaches to modeling pronominal coreference. across evaluations, we find that prompting a large language model (lm) outperforms other approaches on the wsc, but underperforms on certain attested occurrences of pronouns annotated for coreference in ontonotes and related datasets. yilun zhu, sameer pradhan, and amir zeldes. 2021. ontogum: evaluating contextualized sota coreference resolution on 12 more genres. in proceedings of the 59th annual meeting of the association for computational linguistics and the 11th international joint conference on natural language processing (volume 2: short papers), pages 461-467, online. association for computational linguistics.