A mericas NLI : Evaluating Zero-shot Natural Language Understanding of Pretrained Multilingual Models in Truly Low-resource Languages (2022)

Abstract: Pretrained multilingual models are able to perform cross-lingual transfer in a zero-shot setting, even for languages unseen during pretraining. However, prior work evaluating performance on unseen languages has largely been limited to low-level, syntactic tasks, and it remains unclear if zero-shot learning of high-level, semantic tasks is possible for unseen languages. To explore this question, we present AmericasNLI, an extension of XNLI (Conneau et al., 2018) to 10 Indigenous languages of the Americas. We conduct experiments with XLM-R, testing multiple zero-shot and translation-based approaches. Additionally, we explore model adaptation via continued pretraining and provide an analysis of the dataset by considering hypothesis-only models. We find that XLM-R’s zero-shot performance is poor for all 10 languages, with an average performance of 38.48%. Continued pretraining offers improvements, with an average accuracy of 43.85%. Surprisingly, training on poorly translated data by far outperforms all other methods with an accuracy of 49.12%.

## introduction pretrained multilingual models such as xlm @xcite , multilingual bert (mbert; @xcite , and xlm-r @xcite achieve strong cross-lingual transfer results for many languages and natural language processing (nlp) tasks. however, there exists a discrepancy in terms of zero-shot performance between languages present in the pretraining data and those that are not: performance is generally highest for well-represented languages and decreases with less representation. yet, even for unseen languages, performance is generally above chance, and model adaptation approaches have been shown to yield in this work, we introduce americasnli, an extension of xnli @xcite ) -a natural language inference (nli; cf. §2.3) dataset covering 15 high-resource languages -to 10 indigenous languages spoken in the americas: asháninka, aymara, bribri, guaraní, nahuatl, otomí, quechua, rarámuri, shipibo-konibo, and wixarika. all of them are truly low-resource languages: they have little to no digitally available labeled or unlabeled data, and they are not typically studied by the mainstream nlp community. the goal of this work is two-fold: first, we hope to increase the visibility of these languages by providing a portion of the resources necessary for nlp research. second, we aim to allow for a more comprehensive study of multilingual model performance on unseen languages, where improvements will help extend the reach of nlp techniques to a larger set of languages. we are specifically interested in the following research questions: (1) do pretrained multilingual models still perform above random chance for a high-level, semantic task in an unseen language? (2) do methods aimed at adapting models to unseen languages -previously exclusively evaluated on low-level, syntactic tasks -also increase performance on nli? (3) are translationbased approaches effective for truly low-resource languages, where translation quality is typically very poor? 1 we experiment with xlm-r, both with and without model adaptation via continued pretraining on monolingual corpora in the target language. our results show that the performance of xlm-r outof-the-box is moderately above chance, and model adaptation leads to improvements of up to 5.86 percentage points. training on machine-translated training data, however, results in an even larger performance gain of 11.13 percentage points over the corresponding xlm-r model without adaptation. we further perform an analysis via experiments with hypothesis-only models, to examine potential artifacts which may have been inherited from xnli and find that performance is above chance for most models, but still below that for using the full example. americasnli is publicly available foot_1 and we hope that it will serve as a benchmark for measuring the zero-shot natural language understanding abilities of multilingual models for unseen languages. additionally, we hope that our dataset will motivate the development of novel pretraining and model adaptation techniques which are suitable for truly low-resource languages. ## experiments in this section, we detail the experimental setup we use to evaluate the performance of various approaches on americasnli. ## analysis 6.1 hypothesis-only models as shown by @xcite , snli and mnli -the datasets americasnli is based on -contain artifacts created during the annotation process which models exploit to artificially inflate performance. to analyze whether similar artifacts exist in americasnli and if they can also be exploited, we train and evaluate models using only the hypothesis, and present results in ## limitations and future work while the case study above provides strong evidence for the validity of our bribri examples, we cannot currently generalize this claim to the remaining languages. for future work, we plan on extending our human evaluation to more languages and provide a more detailed analysis. additionally, due to the limited availability of annotators and the difficulties of translation for languages that are less frequently studied, the size of the americasnli test set is relatively small. as such, care must be taken to carefully evaluate conclusions drawn using the dataset; following @xcite we present a power analysis of our results in section d.1. future work expanding the dataset size will help create a stronger baseline. furthermore, while we do not make any modelspecific assumptions in our experiments, our results are based on only one pretrained model and adaptation method. methods using vocabulary extension or adapters may offer additional improvements. similarly, other pretrained models could perform differently, depending on, e.g., the model size or the set of languages in their pretraining data. in table f .3, we present results using xlm-r large, and find that, while the relationship between the approaches differs from the main experiments, the overall highest average performance is still achieved by the translate-train approach with xlm-r base. we provide a longer discussion in section f.3. ## conclusion to better understand the zero-shot abilities of pretrained multilingual models for semantic tasks in unseen languages, we present americasnli, a parallel nli dataset covering 10 low-resource lan-guages indigenous to the americas. we conduct experiments with xlm-r, and find that the model's zero-shot performance, while better than a majority baseline, is poor. however, it can be improved by model adaptation via continued pretraining. additionally, we find that translation-based approaches outperform a zero-shot approach, which is surprising given the low quality of the employed translation systems. we hope that this work will not only spur further research into improving model adaptation to unseen languages, but also motivate the creation of more resources for languages not frequently studied by the nlp community.