Breaking the Language Barrier: Improving Cross-Lingual Reasoning with Structured Self-Attention (2023)

Abstract: In this work, we study whether multilingual language models (MultiLMs) can transfer logical reasoning abilities to other languages when they are fine-tuned for reasoning in a different language. We evaluate the cross-lingual reasoning abilities of MultiLMs in two schemes: (1) where the language of the context and the question remain the same in the new languages that are tested (i.e., the reasoning is still monolingual, but the model must transfer the learned reasoning ability across languages), and (2) where the language of the context and the question is different (which we term code-switched reasoning). On two logical reasoning datasets, RuleTaker and LeapOfThought, we demonstrate that although MultiLMs can transfer reasoning ability across languages in a monolingual setting, they struggle to transfer reasoning abilities in a code-switched setting. Following this observation, we propose a novel attention mechanism that uses a dedicated set of parameters to encourage cross-lingual attention in code-switched sequences, which improves the reasoning performance by up to 14% and 4% on the RuleTaker and LeapOfThought datasets, respectively.

## introduction recent studies show that language models (lms) are capable of logically reasoning over natural language statements @xcite , reasoning with their implicit knowledge @xcite , and performing multi-step reasoning via chain-ofthought prompting when the model size is large enough @xcite @xcite . a separate line of work has focused on pre-training language models on multilingual corpora to enable knowledge transfer across different languages. these efforts led to multilingual language models (multilm) such as mbert @xcite , mt5 @xcite , and xlm-r @xcite , which have been shown to generalize in a zero-shot cross-lingual setting @xcite . the cross-lingual transfer is often enabled by fine-tuning the multilm on a high-resource language (typically english) and then evaluating it on other target languages. however, as most of the recent efforts on reasoning-related tasks have been centered around english, our knowledge of the multilingual reasoning capabilities of language models remains limited. in this work, we investigate the logical reasoning capabilities of multilms, especially in monolingual and structured code-switched foot_0 settings (figure 1 ). in the monolingual setting, the context and the question are in the same language. in the structured code-switched setting, we refer to a setting where the context and question are in two different languages. the code-switched setting can be found in many realistic scenarios, such as when non-english speakers may ask questions about information that is unavailable in their native language @xcite . for both reasoning settings, we conduct experiments using the ruletaker dataset @xcite , which contains artificial facts and rules, and the leapofthought dataset @xcite , which incorporates real-world knowledge into the reasoning context. our results show that although multilms perform well when fine-tuned in different languages (i.e., high in-language performance when fine-tuning and testing on the same language), their cross-lingual transfer can vary considerably, especially in the code-switched setting. we posit that the lack of code-switched data in multilm pre-training data makes fine-tuning on code-switched data inconsistent with pre-training. to improve the code-switched reasoning capabilities of multilms, we propose two methods. first, we propose a dedicated cross-lingual query matrix (section 4.1) to better model cross-lingual attention when the multilms receive code-switched sequences as input. this query matrix is pre-trained on unsupervised code-switched data, either shared across all language pairs or specific to a single one. then, we propose a structured attention dropout (see section 4.1), in which we randomly mask attention between tokens from different languages (i.e., context-question attentions) during training. this masking makes the fine-tuning phase more consistent with the pre-training by regularizing crosslingual attention. by mixing the two methods, we also experiment with an interfered variant of the cross-lingual query, which considerably improves cross-lingual generalization, especially in code-switched settings. we evaluate our methods for the code-switched setting and show they improve the cross-lingual transfer of multilms by 14% and 4% for the ruletaker and leapofthought datasets, respectively. ## motivation most prior work on reasoning with language models remains limited to monolingual (english) systems @xcite @xcite . in this work, we investigate the reasoning abilities of multilms, formulating an analysis in formal reasoning that evaluates multilms on their ability to resolve logical statements. given a set of facts and rules as context (in natural language sentences), the task is to predict whether a given statement is true. in our multilingual reasoning setting, we assume a given set of @xmath0, l 2 , ..., l n }, and define l c and l q as the context and statement languages, respectively. typically, multilms are evaluated in a monolingual setup where l @xmath1. however, if multilms are truly multilingual, we posit that they should also be able to reason in a scenario where l c ̸ = l q . thus, to evaluate the multilingual reasoning ability of multilms, we first define four different evaluation setups based on the language of context or statement: (1) both the context and statement are always in one language (monolingual reasoning); (2) the context is always in one language, and the statement can be in any language; to have a reasonable baseline to compare with the code-switched setups, we first focus on the monolingual evaluation (1), in which we evaluate the reasoning ability of multilms for nine typologically different languages. then, by fine-tuning the models on code-switched data, we evaluate their performance for setups (2) and (3) where either the language of the context or the language of the question is different from the training data. this evaluation aims to study the possibility of teaching models to reason across languages in a codeswitched setting, and to investigate the extent they can transfer their reasoning to other code-switched data formats. finally, we hypothesize that in order to succeed in setup (4), the model would have to be strong in setups (2) and (3). since our experimental results show that the multilms struggle in these two setups, we focus on improving their performance for setups (2) and (3). ## multilingual reasoning in this section, we describe our evaluation of the logical reasoning capabilities of multilms for monolingual and code-switched settings. ## cross-lingual-aware self-attention although multilms have been pre-trained on multilingual corpora, individual inputs to the model stay mostly monolingual @xcite . when these models are finetuned on a code-switched downstream task, unlike the pre-training phase, tokens from different languages can attend to each other, which, as demonstrated in tables 2 and 8 , results in poor generalization to other code-switched language pairs. we also observe that self-attention patterns considerably change when we compare code-switched in-language and cross-lingual samples' attention patterns foot_1 (see figure 4 ). ## discussion in this study, we explored the effectiveness of multilms in a code-switched setting and found that while these models exhibit strong reasoning capabilities in monolingual settings, they struggle when it comes to code-switching. to address this, we first proposed the structured attention dropout, which encourages the model to rely less on cross-lingual attention when dealing with code-switched data. this simple method considerably improved cross-lingual transfer to other code-switched languages, demonstrating the importance of structured attention for this setting. we then proposed a novel structured attention mechanism, incorporating the cross-lingual query, that helps the model to better handle cross-lingual attention in the code-switched setting. the proposed cross-lingual query matrix, pre-trained on unsupervised code-switched data, significantly improved the cross-lingual transfer to other codeswitched language pairs in all studies settings, demonstrating the importance of code-switched alignment for multilms. we also observed better cross-lingual code-switched performance for the leapofthought dataset (real-world knowledge contexts) compared to ruletaker (utilizing artificial facts and rules). we attribute leapofthought's better code-switched performance to the usage of real-world knowledge in the reasoning context (compared to artificial facts and rules in ruletaker), in line with @xcite observation that language models perform better when provided with commonsense-consistent context, and struggle with artificial ones. ## limitations in this work, we evaluate our proposed method on encoder-only language models, and the impact of this method on autoregressive models and encodedecoder-only models has not been explored, leaving room for further investigation and evaluation. moreover, our experiments are limited to relatively small language models (less than one billion parameters), and the results and our findings do not necessarily extend to large language models. furthermore, we should highlight that the scope of our experiments is constrained by the availability of multilingual data and computational resources. consequently, our evaluation is limited to two specific datasets and covers only nine languages. while we strive for diversity in our selection, it is important to recognize that broader and more extensive datasets encompassing a wider range of languages could offer additional perspectives and potentially reveal new insights. a. seza dogruöz, sunayana sitaram, barbara e. bullock, and almeida jacqueline toribio. 2021. a survey of code-switching: linguistic and social perspectives for language technologies. in proceedings of the 59th annual meeting of the association for computational linguistics and the 11th international joint conference on natural language processing (volume 1: long papers), pages 1654-1666, online. association for computational linguistics. christiane fellbaum. 1998. wordnet: an electronic lexical database. mit press. negar foroutan, mohammadreza banaei, rémi lebret, antoine bosselut, and karl aberer. 2022. discovering language-neutral sub-networks in multilingual language models. in proceedings of the 2022 conference on empirical methods in natural language processing, pages 7560-7575, abu dhabi, united arab emirates. association for computational linguistics. rujun han, i-hung hsu, jiao sun, julia baylon, qiang ning, dan roth, and nanyun peng. 2021. ester: a machine reading comprehension dataset for reasoning about event semantic relations. in proceedings of the 2021 conference on empirical methods in natural language processing, pages 7543-7559. junjie hu, sebastian ruder, aditya siddhant, graham neubig, orhan firat, and melvin johnson. 2020. xtreme: a massively multilingual multi-task benchmark for evaluating cross-lingual generalisation. in international conference on machine learning, pages 4411-4421. pmlr. navya jose, bharathi raja chakravarthi, shardul suryawanshi, elizabeth sherly, and john p mc-crae. 2020. a survey of current datasets for codeswitching research. in 2020 6th international conference on advanced computing and communication systems (icaccs), pages 136-141. ieee. simran khanuja, sandipan dandapat, anirudh srinivasan, sunayana sitaram, and monojit choudhury. 2020. gluecos: an evaluation benchmark for code-switched nlp. in proceedings of the 58th annual meeting of the association for computational linguistics, pages 3575-3585, online. association for computational linguistics. takeshi kojima, shixiang shane gu, machel reid, yutaka matsuo, and yusuke iwasawa. 2022. large language models are zero-shot reasoners. arxiv preprint arxiv:2205.11916. patrick lewis, barlas oguz, ruty rinott, sebastian riedel, and holger schwenk. 2019. mlqa: evaluating cross-lingual extractive question answering. arxiv preprint arxiv:1910.07475. thomas wolf, lysandre debut, victor sanh, julien chaumond, clement delangue, anthony moi, pierric cistac, tim rault, remi louf, morgan funtowicz, joe davison, sam shleifer, patrick von platen, clara ma, yacine jernite, julien plu, canwen xu, teven le scao, sylvain gugger, mariama drame, quentin lhoest, and alexander rush. 2020. transformers: state-of-the-art natural language processing. in proceedings of the 2020 conference on empirical methods in natural language processing: system demonstrations, pages 38-45, online. association for computational linguistics. shijie wu and mark dredze. 2019. beto, bentz, becas: the surprising cross-lingual effectiveness of bert. arxiv preprint arxiv:1904.09077. linting xue, noah constant, adam roberts, mihir kale, rami al-rfou, aditya siddhant, aditya barua, and colin raffel. 2021. mt5: a massively multilingual pre-trained text-to-text transformer. in proceedings of the 2021 conference of the north american chapter of the association for computational linguistics: human language technologies, pages 483-498, online. association for computational linguistics. elad ben zaken, shauli ravfogel, and yoav goldberg. 2021. bitfit: simple parameter-efficient fine-tuning for transformer-based masked languagemodels. arxiv preprint arxiv:2106.10199.