Investigating associative, switchable and negatable W inograd items on renewed F rench data sets (2022)

Abstract: The Winograd Schema Challenge (WSC) consists of a set of anaphora resolution problems resolvable only by reasoning about world knowledge. This article describes the update of the existing French data set and the creation of three subsets allowing for a more robust, fine-grained evaluation protocol of WSC in French (FWSC) : an associative subset (items easily resolvable with lexical co-occurrence), a switchable subset (items where the inversion of two keywords reverses the answer) and a negatable subset (items where applying negation on its verb reverses the answer). Experiences on these data sets with CamemBERT reach SOTA performances. Our evaluation protocol showed in addition that the higher performance could be explained by the existence of associative items in FWSC. Besides, increasing the size of training corpus improves the model’s performance on switchable items while the impact of larger training corpus remains small on negatable items.

## introduction a winograd schema @xcite consists of two anaphora resolution problems (items) differing by two keywords (successful/available in (1)) which change the answer to a question targeting the referent (paul and george) of an ambiguous anaphor (he). a winograd item is supposed to be google-proof or non associative, meaning that it should be insensitive to simple statistics such as lexical co-occurrence 1 . the idea behind this challenge is that if a system is capable of solving these schemas, it should be capable of commonsense reasoning and hence, could be called "intelligent". paul tried to call george on the phone, but he wasn't successful/available. question : who wasn't successful/available ? response : paul/george since the launch of the first winograd schema challenge @xcite , the reference data set in english has evolved from 273 (wsc273) to 285 items (wsc285). 2 a large variety of methods have been explored to resolve the wsc, including logical formalisms (e.g. @xcite , information retrieval approaches (e.g. @xcite , neural networks (e.g. @xcite and neural language models (e.g. @xcite . the recent literature is dominated by the use of large pretrained language models such as gpt @xcite . sota performance has been reached by @xcite , who fine-tuned roberta @xcite on a large data set they crowd-sourced (winogrande) and achieved 90.1% accuracy on wsc273, vs. 92.1% for humans @xcite and 50% at random. despite the accuracy of models which is now close to human performance, it can be questioned whether systems have become truly capable of commonsense reasoning. @xcite found that not all items are equally robust and categorized items into two subsets : associative and switchable. an associative item is an item where the correct answer can be deduced by solely looking at the clause containing the pronoun/possessive adjective (but he wasn't successful/available in (1)). in a switchable item, the referents can be switched (paul and george in (1)), causing the correct answer to shift accordingly. it was demonstrated that the then-state-of-the-art performance by @xcite was mainly due to the simpler associative subset and on the other hand, insensitive to the switching operation. the evaluation on separate subsets makes it possible to investigate the performance of a system on difficult items and, moreover, tests the robustness of a model's decisions when items are slightly modified. when the french version of wsc (214 items, henceforth fwsc214) was developed @xcite , it was found that some items were associative. however, it was considered that this would not be of much influence, as a system trying to exploit this feature could obtain at most 55% accuracy, vs. a human baseline of 93.6% @xcite . a more recent approach @xcite using small pretrained language models (without fine-tuning on wsc-problems) also pointed at some associativity in the data set, but failed on a large number of other items, performing only at 52% accuracy. the aim of the present work is to get a better understanding of winograd items by dividing fwsc into three subsets. after transforming fwsc214 to fwsc285 based on the most recent version of wsc 3 , we identified an associative and a switchable subset inspired by @xcite . moreover, 1. even though nowadays google-proofness and associativity are often taken to refer to the same property, there is still a difference in the methods used to ensure google-proofness vs. non associativity. in the first case, the idea is that counting co-occurrences in a corpus shouldn't suffice to choose the appropriate answer -for instance in the schema a tree fell on the roof, we'll have to remove/fix it, any search in a corpus will give a higher co-occurrence count to the pair roof/fix vs. roof/remove ; in the second case the idea is that a speaker hearing only the question and the possible answers will be biased towards one of the answers. the bias may come from lexical co-occurrence or from world knowledge. in this section, we will explain how we first adapted the existing fwsc214 to fwsc285 and how we created the associative, switchable and negatable subsets. ## experiments with camembert on fwsc285 and its subsets we fined-tuned camembert large on the machine-translated version of winogrande. concretely, we reframed the wsc task as a sentence classification problem by replacing the pronoun with the two candidate answers. the correct sentence is marked 1 and the wrong one 0 (10) : we used deepl to translate winogrande into french. the first issue of this approach is that the second occurrence of the answer candidate (claire or sylvie in ( 10 )) is sometimes translated back to a pronoun. it is also important to note that the translated corpus does not comply strictly with the definition of wsc items, because deepl may not translate the same word consistently. for instance, because can be translated parce que in one sentence and car in another sentence. since winogrande training corpora come in five sizes, from xs (160 items) to xl (40 938 items), included into one another, we used all these partitions in various experiments on our data sets ( as shown by we evaluated further the model's robustness against perturbations on the switchable and negatable subsets. it can be seen from ## discussion although we achieved sota performances on fwsc214 (66%) and fwsc285 (68%), these performances are expected since we used a sota pretrained language model fine-tuned on a large corpus while previous studies on fwsc used models trained on much smaller corpora and could not leverage the power of transfer learning. besides, our performances were partly due to the existence of associative items. the main contribution of the present work is thus the update of current winograd items in french and the creation of three subsets allowing for a more robust evaluation protocol of the winograd schema challenge. more advanced metrics could be derived using our subsets, such as the group-scoring method used by @xcite which assigns a point only if both items of one schema get solved. in our case, an item could be considered as solved only if its switched and negated versions (if they exist) get solved as well. the negatable subset of winograd items seems particularly challenging, it would be interesting to see if models enhanced with information about the syntactic structure of items @xcite .g.) can perform better. it is also worth noting that both operations (switching and negation) proposed in this study lead to an answer switch. @xcite proposed seven perturbations (tense switch, number switch, etc.) which didn't alter the answer and showed that language models were more likely to switch the answer in case of some perturbations (e.g., number or gender alternations) than humans. similar data sets could be created in french to allow further investigations into humans' and language models' sensitivity to linguistic perturbations. one major limitation related to our training process is the quality of the fine-tuning corpus. it is difficult to know if the insensitivity to negation observed in a final note concerns the wsc task itself. with the quasi-human performance achieved on wsc, a debate has been raised on whether the challenge has been defeated or not @xcite . however, the same performance is far from being reached on fwsc (the best performance is 68% in our work, vs. 93.6% achieved by humans @xcite ). also, @xcite raised the question of whether the commonsense reasoning ability is inherent to the language model or learned during the fine-tuning process and called for more studies using a zero-shot setting. we'd like to point out that using fine-tuning or not, the same evaluation protocol is always necessary to test the robustness of a model's decisions.