XCOPA : A Multilingual Dataset for Causal Commonsense Reasoning (2020)

Abstract: In order to simulate human language capacity, natural language processing systems must be able to reason about the dynamics of everyday situations, including their possible causes and effects. Moreover, they should be able to generalise the acquired world knowledge to new languages, modulo cultural differences. Advances in machine reasoning and cross-lingual transfer depend on the availability of challenging evaluation benchmarks. Motivated by both demands, we introduce Cross-lingual Choice of Plausible Alternatives (XCOPA), a typologically diverse multilingual dataset for causal commonsense reasoning in 11 languages, which includes resource-poor languages like Eastern Apurímac Quechua and Haitian Creole. We evaluate a range of state-of-the-art models on this novel dataset, revealing that the performance of current methods based on multilingual pretraining and zero-shot fine-tuning falls short compared to translation-based transfer. Finally, we propose strategies to adapt multilingual models to out-of-sample resource-lean languages where only a small corpus or a bilingual dictionary is available, and report substantial improvements over the random baseline. The XCOPA dataset is freely available at github.com/cambridgeltl/xcopa.

## introduction commonsense reasoning is a critical component of any natural language understanding system @xcite . contrary to textual entailment, commonsense reasoning must bridge between premises and possible hypotheses with world knowledge that is not explicit in text @xcite . such world knowledge encompasses, among other aspects: temporal and spatial relations, causality, laws of nature, social conventions, politeness, emotional responses, and multiple modalities. ultimately, it shapes the individuals' expectations about typical situations @xcite . 1 a seminal work on the quantitative evaluation of commonsense reasoning is the choice of plausible alternatives dataset (copa; @xcite , which focuses on cause-effect relationships. in recent years, more datasets have been dedicated to other facets of world knowledge @xcite @xcite @xcite . unfortunately, the extensive efforts related to this thread of research have so far been limited only to the english language. foot_1 such a narrow scope not only curbs the development of natural language understanding tools in other languages @xcite , but also exacerbates the anglocentric bias in modeling commonsense reasoning. in fact, the expectations about typical situations do vary across cultures @xcite . datasets that cover multiple languages for other natural understanding tasks, such as language inference @xcite , question answering @xcite @xcite , and paraphrase identification @xcite have received increasing attention. in fact, the requirement to generalise to new languages encourages the development of more versatile language understanding models, which can be ported across different grammars and lexica. these efforts have recently culminated in the integration of several multilingual tasks into the xtreme evaluation suite @xcite . however, a compre-premise choice 1 choice 2 qu sipasqa cereal mikhunanpi kuruta tarirqan. r payqa pukunman ñuqñuta churakurqan. payqa manam mikhuyta munarqanchu. en the girl found a bug in her cereal. she poured milk in the bowl. she lost her appetite. th ตาของฉั นแดงและบวม c ฉั นร้ องไห้ ฉั นหั วเราะ en my eyes became red and puffy. i was sobbing. i was laughing. in order to address this gap, we develop a novel dataset, xcopa (see examples in we leverage xcopa to benchmark a series of state-of-the-art pretrained multilingual models, including xlm-r @xcite , mbert @xcite , and multilingual use @xcite . two xcopa languages (i.e., southern quechua and haitian creole) are out-of-sample for the pretrained models: this naturally raises the question of how to adapt the pretrained models to such unseen languages. in particular, we investigate the resource-lean scenarios where either some monolingual data or a bilingual dictionary with english (or both) are available for the target language. in summary, we offer the following contributions. 1) we create the first large-scale multilingual evaluation set for commonsense reasoning, spanning 11 languages, and discuss the challenges in accounting for world knowledge across different cultures and languages. 2) we propose quantitative metrics to measure the internal variety of a language sample, which can guide the design of any multilingual dataset in the future. 3) we benchmark pretrained state-of-the-art models in cross-lingual transfer of commonsense knowledge, and 4) investigate how to (post-hoc) improve transfer for languages unseen at pretraining time. in order to rise to the challenge of this dataset, models must be able not only to combine textual evidence with world knowledge -which makes commonsense reasoning challenging per se @xcite , but they must also transfer the acquired causal reasoning abilities across languages. the results we obtain on xcopa thus indicate the limitations of current state-of-theart multilingual models in cross-lingual transfer settings for complex reasoning tasks. ## annotation design design objectives. the principal objectives in devising xcopa were: 1) to create a genuinely typologically diverse multilingual dataset, aligned across target languages to make performance scores comparable, and 2) to ensure high quality, naturalness and idiomacity of each monolingual dataset. while the commonly used translation approach addresses the former objective, it is prone to compromise the latter goal, bending the target language to the structural and lexical properties of the source language: the resulting evaluation benchmarks thus fail to measure system performance adequately @xcite @xcite . to avoid these pitfalls, we: (i) entrusted the translation task to a single (carefully selected) translator per target language, foot_2 and (ii) offered enough leeway for necessary target-language adjustments (e.g., substitutions with culture-specific concepts and multi-word paraphrases, wherever the original text eluded direct translation). detailed translation guidelines are available in the appendix. language sampling. multilingual evaluation benchmarks assess the expected performance of a model across languages. however, should such languages be sampled according to the distribution of digital texts or rather based on the distribution over the languages spoken around the world? the inspired by @xcite and @xcite , we propose a series of simple and interpretable metrics that quantify diversity of a language sample independent of its size: 1) a typology index based on 103 typological features of each language from uriel @xcite , originally sourced from the world atlas of language structures (wals; @xcite . each feature is binary and indicates the presence or absence of an attribute in a language. we estimate the entropy of the distribution of values in a sample. afterwards, we average across all 103 feature-specific entropies. intuitively, if all values are equally represented, the entropy is high. if all languages have identical features, the entropy is 0; 2) the family index is simply the number of distinct families divided by the sample size. 3) the geography index is the entropy of the distribution over macro-areas in a sample. 4 the sample of languages for xcopa aims at maximising these indices. in particular, xcopa includes estonian (et), haitian creole (ht), indonesian (id), italian (it), eastern apurímac quechua (qu), kiswahili (sw), tamil (ta), thai (th), turkish (tr), vietnamese (vi), and mandarin chinese (zh). these languages belong to distinct families, respectively: uralic, creole, austronesian, indo-4 six macro-areas, as defined by wals @xcite , are: africa, australia, eurasia, north america, papunesia, and south america. whenever a language spans multiple macro-areas, we select that of the standard variety. european, yuman-cochimí, niger-congo, dravidian, kra-dai, turkic, austroasiatic, and sino-tibetan. moreover, ht and qu are spoken in north and south america, respectively, which are both underrepresented macro-areas. we report the 3 metrics in final dataset. as from the 11 sets of annotation labels we obtain the majority labels (i.e., 6+ translators agree). we observe perfect agreement between our majority labels and the english copa labels for development data. we then compute the percentage of annotated labels which agree with the majority label for each language individually, reported in ## qualitative analysis as highlighted in §2, our guidelines anticipated that the adopted translation approach may entail language-specific challenges, e.g., the lack of equivalent concepts or the grammatical expression of tense and aspect. we now analyse the main design challenges and the adopted solutions. cultural context. the scenarios included in english copa were authored by american english speakers with a particular cultural background. it is therefore inevitable that some concepts, intended as commonplace, sound unusual or even completely foreign in the target language. examples include: (i) concrete referents with no language-specific term available (e.g., bowling ball, hamburger, lottery); (ii) systems of social norms absent in the target culture, e.g., traffic regulations (e.g., parallel parking, parking meter); (iii) social, political, and cultural institutions and related terminology (e.g., e.g., mortgage, lobbyist, gavel); (iv) idiomatic expressions (e.g., put the caller on hold). in such cases, the translators were advised to resort (in the following order of preference) to (i) paraphrasing; (ii) substitutions with similar concepts, e.g., 'faucet' is replaced with 'pipe' in tamil (!ழா@xmath0% (cōppu, 'soap'). ## experiments and results xcopa is a multiple-choice classification task: given a premise and a prompt (cause or result), the goal is to select the more plausible of the two answer choices (see ## conclusion and future work we presented the cross-lingual choice of plausible alternatives (xcopa), a multilingual evaluation benchmark for causal commonsense reasoning. all xcopa instances are aligned across 11 languages, which enables cross-lingual comparisons. the language selection was informed by variety sampling, in order to maximise diversity in terms of typological features, geographical macro-area, and language family. this allows us to test the robustness of machine learning models for an array of rare phenomena displayed by the chosen languages. we also ran a series of cross-lingual transfer experiments, evaluating state-of-the-art transfer methods based on multilingual pretraining and finetuning on english. we observed that, although these methods perform better than chance, they still lag significantly behind the monolingual setting where test data are translated into english, due to the 'curse of multilinguality.' in addition, we verified that spurious correlations are insufficient to solve this new task by creating 'adversarial' variants of the dataset, where premises or prompts are masked, thus showing that robust causal reasoning is indeed required to solve xcopa. finally, we investigated resource-lean adaptation of pretrained multilingual models to out-of-sample languages, exploiting only small monolingual corpora and/or bilingual dictionaries, reporting notable gains. we hope that this new challenging evaluation set will foster further research in multilingual commonsense reasoning and cross-lingual transfer.