Wino- X : Multilingual W inograd Schemas for Commonsense Reasoning and Coreference Resolution (2021)

Abstract: Winograd schemas are a well-established tool for evaluating coreference resolution (CoR) and commonsense reasoning (CSR) capabilities of computational models. So far, schemas remained largely confined to English, limiting their utility in multilingual settings. This work presents Wino-X, a parallel dataset of German, French, and Russian schemas, aligned with their English counterparts. We use this resource to investigate whether neural machine translation (NMT) models can perform CoR that requires commonsense knowledge and whether multilingual language models (MLLMs) are capable of CSR across multiple languages. Our findings show Wino-X to be exceptionally challenging for NMT systems that are prone to undesirable biases and unable to detect disambiguating information. We quantify biases using established statistical methods and define ways to address both of these issues. We furthermore present evidence of active cross-lingual knowledge transfer in MLLMs, whereby fine-tuning models on English schemas yields CSR improvements in other languages.

## introduction originally introduced in @xcite , winograd schemas (schemas from here on) have become an established tool for probing the ability of computational models to reason about natural language. either viewed through the lens of coreference (cor) as in @xcite or, more recently, framed as a gap-filling task @xcite , schemas are assumed to require commonsense knowledge to be resolved correctly. consider the following schema: the trophy doesn't fit into the brown suitcase because it is too [large / small]. here, the pronoun it has two possible antecedents (trophy / suitcase), with the choice of the antecedent determined by the trigger word (large / small). to connect the pronoun to its true antecedent, a model must 'know' that objects that are too large cannot fit into containers and that containers that are too small cannot house objects. when translating an instance of a schema (i.e. the schema with a fixed trigger word) into languages such as german, where pronouns and their antecedents must agree in their grammatical gender, translation models must implicitly perform the cor step to produce accurate translations. a competent translation model is, therefore, expected to identify the correct antecedent as reflected by the target pronoun choice. in the first part of this work, we construct cross-lingual instances by aligning english instances with their translations into morphologically rich languages, so as to probe the robustness of cor in current nmt models, as illustrated in figure 1 (top half). in doing so, we show that models follow simplistic heuristics when attempting to resolve coreference, while failing to detect disambiguating information. a second category of models that is expected to correctly identify coreference in multiple languages are multilingual language models. where translation models learn to map their input to semantically equivalent sequences in the target language, mllms are trained on a mask-filling objective and learn to encode sentences drawn from different languages into a shared semantic space. accordingly, schema instances correctly solved by mllms in one language should be equally solvable in other languages, by leveraging the same, language-agnostic representations. similarly, improvements to model performance in one language should transfer to other languages via the shared latent space. in the second part of our work, we empirically put these assumptions to the test with multilingual schema instances, as shown in figure 1 (bottom half), finding evidence of active commonsense knowledge transfer across languages. our primary contributions are as follows: based on the chosen noun, the resulting sentence either satisfies or violates commonsense constraints. schemas are divided into two domains -social and physical. those belonging to the former category predominantly feature names of individuals (e.g. mary or tom) as fillers, whereas physical samples feature objects or entities (e.g. vase or cat). constructing cross-lingual schemas suitable for evaluating translation models requires replacing the gap with the ambiguous pronoun it, which is not possible for the social domain. consequently, we focus our attention on the physical subset of wino-grande that contains 19,260 unique samples (9,630 schemas), with each sample representing a single instance of a monolingual, english schema. ## testing cross-lingual transfer in mllms having thus probed the capacity and limitations of nmt models for solving cross-lingual wino-x samples, we now turn to mllms. ## conclusion and outlook in this work, we introduced wino-x, a dataset containing cross-lingual and multilingual winograd schemas. based on this resource, we showed that nmt models struggle to correctly resolve coreference that presupposes commonsense knowledge, due to over-reliance on dataset artifacts and general inability to detect disambiguating information. we defined methods to quantify biases and trigger word importance in a principled way, and proposed strategies for reducing the former while increasing the latter. for mllms, we presented evidence of commonsense knowledge transfer, showing that transferring knowledge from english to another language can lead to similar (or greater) improvements as directly fine-tuning on transfer languages. overall, our study identifies existing difficulties in cross-lingual cor and csr, discusses potential causes, and offers initial ways to mitigate them. in future work, we intend to further improve the handling of coreference in nmt by reducing undesirable biases and introducing useful ones. for mllms, future efforts can be directed towards identifying categories of knowledge that do not benefit from cross-lingual transfer, to effectively guide data collection in lower-resourced languages.