Cross-lingual Few-Shot Learning on Unseen Languages (2022)

Abstract: Large pre-trained language models (LMs) have demonstrated the ability to obtain good performance on downstream tasks with limited examples in cross-lingual settings. However, this was mostly studied for relatively resource-rich languages, where at least enough unlabeled data is available to be included in pre-training a multilingual language model. In this paper, we explore the problem of cross-lingual transfer in unseen languages, where no unlabeled data is available for pre-training a model. We use a downstream sentiment analysis task across 12 languages, including 8 unseen languages, to analyze the effectiveness of several few-shot learning strategies across the three major types of model architectures and their learning dynamics. We also compare strategies for selecting languages for transfer and contrast findings across languages seen in pre-training compared to those that are not. Our findings contribute to the body of knowledge on cross-lingual models for low-resource settings that is paramount to increasing coverage, diversity, and equity in access to NLP technology. We show that, in few-shot learning, linguistically similar and geographically similar languages are useful for cross-lingual adaptation, but taking the context from a mixture of random source languages is surprisingly more effective. We also compare different model architectures and show that the encoder-only model, XLM-R, gives the best downstream task performance.

## introduction the availability of large-scale multilingual pretrained language models has enabled a more effective transfer of knowledge across languages @xcite @xcite @xcite , thus limiting the need to gather taskspecific annotated data for a given target language. recent research into few-shot learning approaches proposed methods that explicitly aim to improve performance when few annotated data points are available to perform a task @xcite @xcite , semantic parsing @xcite , topic modeling @xcite . further, cross-lingual few-shot learning uses multilingual models and few-shot learning methods to perform a task given limited training data in another language and has shown promise on several downstream tasks @xcite @xcite . these studies have only looked at relatively resource-rich target languages, as they are part of the pre-training data for the multilingual language model, and even for these languages, the representation quality is not equal due to imbalanced corpus size @xcite ). representation quality is expectedly lower for the vast majority of the spoken languages in the world, most of which are not part of the pre-training data in multilingual models, albeit being spoken by large populations. for example, ngaju is the native language of over 890,000 people, yet there is no wikipedia available for this language, which is a common source of data for pre-training. cross-lingual few-shot learning methods are a promising avenue of research for enabling nlp technologies for such languages, especially as we can assume both unlabeled and, especially, labeled data for a given task are difficult to obtain at scale @xcite @xcite @xcite . this paper is the first to study cross-lingual fewshot learning methods in unseen languages at the pre-training stage. we focus mainly on how to most effectively train a model for a downstream classification task in an unseen language without having access to any labeled data in that language. we experiment with all three major types of multilin-gual pre-trained language model architectures, including the encoder-only xlm-r @xcite , the decoder-only xglm @xcite and the encoder-decoder mt5 @xcite models. we combine these with different strategies for few-shot learning for a new language, including in-context learning, prompt-based fine-tuning, and encoder-based fine-tuning. we evaluate the effectiveness of these approaches under varying levels of available training data. we perform several analyses to understand aspects such as the performance gap between languages seen in pre-training compared to those unseen and which source languages are best suited for a target language. we perform this study on the downstream task of sentiment analysis across 12 languages spoken in indonesia plus english from the nusax corpus @xcite . this dataset contains parallel sentences annotated for sentiment, which conveniently allows control for content drift when comparing transfer capabilities across languages. our contributions are as follows: in sum, our work contributes new insights to the growing body of work in cross-lingual nlp for extremely low-resource languages, a critical step in increasing coverage and access to nlp technology. ## methodology we define our task as follows: let θ be the lm and t l be the dataset for language l consisting of n sentence and label pairs {x 1 , y 1 }, {x 2 , y 2 }, ..., {x n , y n }, where x i , y i , are the inputs and labels, respectively. in the crosslingual setting, we take the source language l src from a pool of languages l that does not include the target language l tgt . in this work, we categorize languages as seen and unseen. the unseen languages, are those languages that were not present in the data used to pre-train the multilingual models, while the seen languages were included during pre-training. our goal is to investigate what are the most successful strategies for cross-lingual transfer learning under extremely limited data settings. with this in mind, we want to answer the following questions: ## conclusion we present the first comprehensive study to measure the effectiveness of few-shot in-context learning and fine-tuning approaches with multilingual lms on languages that have never been seen during pre-training. we investigate the effectiveness of utilizing few-shot examples and present strategies and insights depending on the amount of labeled training data available. we find that fine-tuning the multilingual encoder model (i.e., xlm-r) is generally the most effective method when we have more than 15 samples; otherwise, zero-shot crosstask is preferable. we also observe that in-context learning has a relatively higher variance than finetuning, and mixing multiple source languages is a promising approach when the number of training examples in each language is limited.