Leveraging Knowledge in Multilingual Commonsense Reasoning (2022)

Abstract: Commonsense reasoning (CSR) requires models to be equipped with general world knowledge. While CSR is a language-agnostic process, most comprehensive knowledge sources are restricted to a small number of languages, especially English. Thus, it remains unclear how to effectively conduct multilingual commonsense reasoning (XCSR) for various languages. In this work, we propose to use English as a pivot language, utilizing English knowledge sources for our our commonsense reasoning framework via a translate-retrieve-translate (TRT) strategy. For multilingual commonsense questions and answer candidates, we collect related knowledge via translation and retrieval from the knowledge in the source language. The retrieved knowledge is then translated into the target language and integrated into a pre-trained multilingual language model via visible knowledge attention. Then we utilize a diverse of four English knowledge sources to provide more comprehensive coverage of knowledge in different formats. Extensive results on the XCSR benchmark demonstrate that TRT with external knowledge can significantly improve multilingual commonsense reasoning in both zero-shot and translate-train settings, consistently outperforming the state-of-the-art by more than 3% on the multilingual commonsense reasoning benchmark X-CSQA and X-CODAH.

## introduction commonsense reasoning (csr) is one of the key challenges in natural language understanding. it requires a model to integrate world knowledge into language modeling to produce answers. a large number of tasks have been proposed to evaluate commonsense reasoning in english, such as copa @xcite and csqa @xcite . most recently, multilingual commonsense reasoning (xcsr) extends a model's commonsense the statistics are generated from wiktionary 2021-10-01 dump. there are 55 languages with 10,000 or more definitions and we list top 20 languages by the definition count here. to solve commonsense reasoning tasks, it is essential to fuse human created knowledge into pre-trained language models (plm) @xcite @xcite . for example, dekcor @xcite integrates knowledge from conceptnet @xcite and wiktionary foot_0 into the albert model @xcite for commonsense question answering. however, most existing knowledge sources are crafted in a few popular languages, especially english. for example, figure 1 shows the number of total definitions in english is much greater than any other language based on the statistics from a recent 2021-10-01 dump of wiktionary. thus, it remains an open question how to tackle xcsr with a lack of curated knowledge in the figure 2 : an overview of our framework for multilingual commonsense reasoning. given the question and candidate answers in the target language (chinese), we first translate it into english, then retrieve related knowledge from four english knowledge sources and translate the retrieved knowledge back into the target language. the retrieved knowledge, along with question and candidate answer, are fed into the multilingual pretrained language model for answer prediction. target language. in this paper, we propose a translate-retrievetranslate (trt) solution to utilize english knowledge sources for xcsr. specifically, given a commonsense reasoning question (possibly concatenated with a candidate answer) in the target language, we first translate it into english. next, we retrieve related knowledge from english knowledge sources. the retrieved knowledge is then translated back into the target language. finally, the knowledge is integrated into a multilingual language model via an visible knowledge attention mechanism to answer the question. another contribution of our work is that the utilization of a diverse set of knowledge sources to provide a more comprehensive coverage of knowledge in different formats. specifically, we utilize unstructured text corpus (open mind common sense (singh, 2002)), structural knowledge graph (conceptnet @xcite ), dictionary (wiktionary) and large-scale language model (gpt-3 @xcite ). given an input query, we utilize information retrieval, entity linking, and model inference to obtain knowledge from the corresponding sources. we conduct extensive evaluation of our model on the multilingual commonsense reasoning benchmark x-csqa and x-codah @xcite . the results demonstrate the effectiveness of our proposed translate-retrieve-translate solution with multiple knowledge sources. for example, in the zero-shot transfer setting, trt with wiktionary can improve 1.9 and 2.7 points over the baselines. for translate-train setting, trt with wiktionary and omcs outperform 1.6 and 1.0 over the baselines. compared with previous state-of-the-art results on the xcsr leaderboard, trt improve them by more than 3 points. we summarize the main contributions of this work as follows. (i) we propose a translateretrieve-translate (trt) solution to utilize english knowledge sources for multilingual commonsense reasoning. (ii) we comprehensively explore four knowledge sources in different formats and demonstrate their utility on a pair of xcsr benchmarks (x-csqa, x-codah). (iii) we achieve the stateof-the-art results on the on xcsr leaderboard, outperforming the previous best methods by more than 3.3 points. ## related work multilingual commonsense reasoning evaluating the commonsense reasoning abilities of trained models has been explored through a variety of tasks and problem settings. an early work in this space was the winograd scheme challenge @xcite , where the goal is to disambiguate the reference of a pronoun (levesque knowledge source knowledge format query format retrieved knowledge retrieval method wiktionary dictionary content word definition string matching conceptnet entity-relation triplets entity pair entity-relation triplet entity linking omcs text in sentences sentences sentences bm25 gpt-3 parameters unstructured text unstructured text conditional generation multilingual language model large scale multilingual pretrained language models (mplm) @xcite @xcite have always been the most important backbone for solving multilingual tasks including commensense reasoning tasks. knowledge bases have also been integrated into the pretraining process @xcite . as shown in @xcite , there exist multilingual knowledge base. but they still lack the components or contexts to explicitly integrate knowledge and commonsense. and lin et al. gpt-3 prompt learning large scale pretrained language models like gpt-3 @xcite have shown tremendous success on few-shot learning. there exists a large body of work on prompting to leverage the implicit knowledge from it @xcite @xcite . in this work, we focus on leveraging gpt-3 to generate three diverse knowledge formats and then fusing them into fine-tuning stage. ## approach in this section, we first formalize the multilingual commonsense reasoning (xcsr) task (section 3.1). then we describe more details about our commonsense knowledge resources (section 3.2). next, we introduce our proposed translate-retrieve-translate (trt) solution to obtain the multilingual knowledge (section 3.3). finally, we introduce how to fuse the obtained knowledge into multilingual pretrained language models by employing the visible attention mechanism (section 3.4). an overview of our framework is illustrated in figure 2 . ## experiments in this section, we perform extensive experiments to explore the aforementioned trt solution with four knowledge sources on the multilingual commonsense reasoning benchmark xcsr @xcite . ## conclusion in this work, we first present the translate-retrievetranslate (trt) strategy for multilingual commonsense reasoning that collects related knowledge via translation and then retrieval from the knowledge sources. then we conduct extensive experiments by utilizing a diverse of four english knowledge sources, including wiktionary, conceptnet, omcs and gpt-3. by using trt with different knowledge sources, we achieve state-of-the-art results on xcsr leaderboard which demonstrates the effectiveness of our proposed methods. future work includes more effective ways to incorporate the diverse knowledge sources into pre-training and fine-tuning stage for commonsense reasoning.