Common Sense Beyond E nglish: Evaluating and Improving Multilingual Language Models for Commonsense Reasoning (2021)

Abstract: Commonsense reasoning research has so far been limited to English. We aim to evaluate and improve popular multilingual language models (ML-LMs) to help advance commonsense reasoning (CSR) beyond English. We collect the Mickey corpus, consisting of 561k sentences in 11 different languages, which can be used for analyzing and improving ML-LMs. We propose Mickey Probe, a language-general probing task for fairly evaluating the common sense of popular ML-LMs across different languages. In addition, we also create two new datasets, X-CSQA and X-CODAH, by translating their English versions to 14 other languages, so that we can evaluate popular ML-LMs for cross-lingual commonsense reasoning. To improve the performance beyond English, we propose a simple yet effective method — multilingual contrastive pretraining (MCP). It significantly enhances sentence representations, yielding a large performance gain on both benchmarks (e.g., +2.7% accuracy for X-CSQA over XLM-R_L).

## introduction understanding natural language relies heavily on commonsense reasoning (csr), which is the process of making inferences with commonsense knowledge. commonsense knowledge is the set of general facts that reflect our natural understanding of the physical world and human behavior, which are usually seen as an implicit background when people communicate with each other using languages. it is thus of vital importance to evaluate and improve the commonsense reasoning capability of language models (lms), towards building general natural language understanding (nlu) systems @xcite . ## background and related work in this section, we introduce important concepts, background knowledge, and related work before we present our work in following sections. ## the mickey probe the challenges of using the lama probe for probing common sense in ml-lms motivate us to propose a more suitable method for analyzing ml-lms, one that can fairly compare across a diverse set of languages. we present mickeyprobe, a multilingual task for probing commonsense knowledge and analysis. we design a languageagnostic probing task with a sentence-selection objective for analyzing common sense of a ml-lm: given a set of assertions (i.e., declarative sentences) that have similar words and syntactic features, select the one with highest commonsense plausibility. we present the task formulation in this section and then introduce how we collect the dedicated dataset in section 4. notations. we define a mickey probe m as a set of k assertions in the same language, where one and only one of them (say, m i ) is the truth assertion with better commonsense plausibility than the other k 1 ones. each mickey probe m has multiple semantically equivalent versions in different languages. let us denote a language by l 2 l where @xmath0, f r, ru, zh, . . . } and |l| is the number of languages of interest. then, m l is the probe m in the language l. for example, m en and m fr denote the probes with the same meaning but in english (en) and french (fr) respectively. we use m to denote a multilingual parallel dataset for mickeyprobe, which consists of t ⇥|l|⇥k assertions. t is the number of mickeyprobe items and each item has k assertions and |l| language. finally, we can formally describe a multilingual parallel dataset m for mickeyprobe: we use the notation ./ to indicate two assertions in different languages (e.g., l x and l y ) are semantically equivalent to each other. we leave the details of creating such an m in section 4. commonsense probing task. given a micky probe m in the dataset m, and suppose the index of the truth assertion to be t, a perfect multilingual language model would produce sentence probabilities such that it always gives the truth assertion m l t the highest probability among other candidates for every language. ## the mickey corpus and evaluation we present a procedure for automatically creating a multilingual parallel dataset m for the probing task mickeyprobe. our collected corpus, named mickeycorpus , has 561k sentences in 11 languages (@xmath1.2k, @xmath2, |l|=11). ## multilingual contrastive pre-training in this section, we reformat the mickeyprobe so that we can reuse the mickeycorpus for improving the pre-trained ml-lms for commonsense reasoning beyond english. we propose a multilingual contrastive pre-training (mcp) task that focuses on enhancing the sentence-level representation of ml-lms. mcp improves a ml-lm in a multilingual, contrastive environment, where the model learns to select the assertion with the best commonsense plausibility from a set of contrastive sentences in different languages. each mcp example is a set of multilingual assertions while each mickey probe is a monolingual set. mcp dataset creation from m. we create pretraining examples for the mcp task by converting mickeyprobe examples, as shown in the steps illustrated in algorithm 1. simply put, we reformat a k-way mickey probe m (k ⇥ |l| assertions) to a mcp example by sampling a set of v candidate assertions in v different languages. we convert all examples in the mickeycorpus m to build a new cross-lingual sentence-selection dataset c for learning the mcp task. mcp learning. given a mcp example c 2 c, we append one dense linear layer f on top of a ml-lm with parameters denoted as ⇥ ml-lm for learning to predict the commonsense plausibility score of each assertion c i 2 c as follows: we first get the logit o i of each assertion by projecting its [cls] embeddings h i to a logit o i via a dense layer f with parameters ⇥ f ; then, we use softmax to normalize the logits as plausibility scores z i ; finally, we compute the cross-entropy loss ⇢ where 1 @xmath3. we fine-tune {⇥ ml-lm , ⇥ f } to minimize the overall loss over the mcp dataset c. ## evaluation for cross-lingual csr in this section, we introduce the datasets, experimental setup, results, and our analysis. algorithm 1: convert a mickey probe m to an example for the mcp task. in: m 2 m /* is a probe that has |l| sub-sets; each sub-set m lx is a set of k assertions in the same language lx 2 l. m lx t is always the truth. */ out: to evaluate ml-lms for commonsense reasoning in a cross-lingual zero-shot transfer setting, we create two benchmark datasets, namely x-csqa and x-codah. ## conclusion we evaluate and improve popular multilingual language models (ml-lms) for advancing commonsense reasoning beyond english. we propose the mickeyprobe, a language-agnostic probing task for analyzing common sense of ml-lms in a zero-shot manner. with our proposed new benchmark datasets via automatic translation, x-csqa and x-codah, we evaluate ml-lms in a crosslingual transfer setting for commonsense reasoning. we also improve the state-of-the-art ml-lm with a simple yet effective method -multilingual contrastive pre-training, which uses a sentencelevel objective to enhance sentence representations, yielding a significant performance gain. all above work is based on mickeycorpus, which can be used as both a probing dataset and a pretraining corpus for analyzing and improving ml-lms. we hope our resources and pre-training method for ml-lms can help the community advance commonsense reasoning beyond english.