Analyzing the Effects of Reasoning Types on Cross-Lingual Transfer Performance (2021)

Abstract: Multilingual language models achieve impressive zero-shot accuracies in many languages in complex tasks such as Natural Language Inference (NLI). Examples in NLI (and equivalent complex tasks) often pertain to various types of sub-tasks, requiring different kinds of reasoning. Certain types of reasoning have proven to be more difficult to learn in a monolingual context, and in the crosslingual context, similar observations may shed light on zero-shot transfer efficiency and few-shot sample selection. Hence, to investigate the effects of types of reasoning on transfer performance, we propose a category-annotated multilingual NLI dataset and discuss the challenges to scale monolingual annotations to multiple languages. We statistically observe interesting effects that the confluence of reasoning types and language similarities have on transfer performance.

## introduction recent work has shown that masked language models such as xlm @xcite , mbert @xcite and xlm-roberta @xcite can achieve efficient crosslingual transfer, in both zero-shot and few-shot settings. such results have motivated researchers @xcite @xcite to investigate the underlying influencing factors behind transfer efficiency along different dimensions such as model capacity, language similarity and learning objective. however, such transfer efficiency is measured for natural language understanding (nlu) tasks which often cover a wide range of linguistic phenomena. natural language inference is one such representative task which is known to require different types of reasoning and linguistic phenomena. in the monolingual context, recently, authors in @xcite extended previous work @xcite and listed a comprehensive list of types of reasoning capabilities required to solve the nli examples in large public nli datasets. authors show that nli requires a mix of 15 different types of reasoning capabilities (taxinli), categorized broadly into language, logic and knowl-edge. they observe that both bert @xcite and roberta @xcite perform poorly in reasoning tasks such as causal and coreference, whereas they pick up negation easily; and observe that some categories are intrinsically harder for these models (both transformers-based and pre-transformers lstm-based) to understand. our central hypothesis of this work is, crosslingual transfer gap @xcite and few-shot performance may depend on the type of reasoning required. to this end, we propose multilingual extension of the taxinli dataset. our zero-shot analysis strongly suggests that reasoning types play a critical role in transfer performance. in few-shot, some categories (such as negation), show steady improvement in accuracy along-with increased cross-lingual alignment. for others (syntactic, logic) improvement varies across language. for causal, few-shot models fail to generalize well on a diagnostic gold set. ## experimental setup for our experiments, we fine-tune multilingual bert and xlm-r (base) foot_2 on multinli training @xcite data and select the best model using the multinli validation set. we repeat the fine-tuning with 5 different seed values and for learning-rate in {2e-5, 3e-5}. for zero-shot experiments, we evaluate the best model on taxixnli datasets for each language and taxonomic category. we measure the crosslingual transferability using transfer gap @xcite which is defined as the difference between the performance on english and target language. whereas, for few-shot learning, for each of the target language and inference type pair, we continue to fine-tune the best multinli fine-tuned xlm-r and m-bert model using a random few tax-ixnli (translated) training examples in that particular pair and then test the model on the same language and inference type pair. we repeat all our experiments with 5 different seeds (different examples each time) and report the average. for our analysis, we measure relative error reduction (@xmath0 ## crosslingual cross-capability insights in this section, we discuss some important insights that we observe through the zero-shot and few-shot experiments. we highlight phenomena which are repeated across taxixnli (translated) and tax-ixnli (diagnostic) datasets. figure 5: few shot accuracy (taxixnli (diagnostic), xlm-r): for each taxinli category and language, we plot accuracy on taxixnli (diagnostic) test, when xlm-r multinli model is fine-tuned on n examples in that language and category is transferability uniform across categories? does more pre-training help? from figure 1 and 2, we observe that the transferability of different inference types are non-uniform, especially in the case of xlm-r. in particular, transferability of negation (except for swahili) is a lot better than lexical, syntactic, boolean, or knowledge inference types. the results on the taxixnli (diagnostic) test confirms these observations. further, from the results on taxixnli (translated), in ## findings and future work we proposed a multilingual extension of a reasoning type-annotated nli dataset (taxinli) in english -taxixnli (translated) and a gold test set taxixnli (diagnostic) to analyze transfer efficiency of multilingual language models. our analysis strongly indicates that reasoning types play an important role in transfer capability. our few-shot results demonstrate that for types such as negation, improvement in transfer accuracy co-occurs with improved crosslingual alignment. for causal, few-shot models fail to generalize well on the diagnostic set; and some categories hardly improve. in summary, nli being central to nlu and nlp, this dataset and our ensuing analysis provides a way to broadly quantify the gaps in zero-shot transfer across various linguistic and logical dimensions.