A Closer Look at Few-Shot Crosslingual Transfer: The Choice of Shots Matters (2021)

Abstract: Few-shot crosslingual transfer has been shown to outperform its zero-shot counterpart with pretrained encoders like multilingual BERT. Despite its growing popularity, little to no attention has been paid to standardizing and analyzing the design of few-shot experiments. In this work, we highlight a fundamental risk posed by this shortcoming, illustrating that the model exhibits a high degree of sensitivity to the selection of few shots. We conduct a large-scale experimental study on 40 sets of sampled few shots for six diverse NLP tasks across up to 40 languages. We provide an analysis of success and failure cases of few-shot transfer, which highlights the role of lexical features. Additionally, we show that a straightforward full model finetuning approach is quite effective for few-shot transfer, outperforming several state-of-the-art few-shot approaches. As a step towards standardizing few-shot crosslingual experimental designs, we make our sampled few shots publicly available.

## introduction multilingual pretrained encoders like multilingual bert (mbert; @xcite ) and xlm-r @xcite are the top performers in crosslingual tasks such as natural language inference @xcite , document classification @xcite , and argument mining @xcite . they enable transfer learning through language-agnostic representations in crosslingual setups @xcite . a widely explored transfer scenario is zero-shot crosslingual transfer @xcite @xcite , where a pretrained encoder is finetuned on abundant task data in the source language (e.g., english) and then directly evaluated on target-language test data, achieving surprisingly good performance @xcite . however, there is evidence that zero-shot performance reported in the literature has large variance and is often not reproducible @xcite ; the results in languages distant from english fall far short of those similar to english @xcite . @xcite stress the importance of few-shot crosslingual transfer instead, where the encoder is first finetuned on a source language and then further finetuned with a small amount (10-100) of examples (few shots) of the target language. the few shots substantially improve model performance of the target language with negligible annotation costs @xcite . in this work, however, we demonstrate that the gains from few-shot transfer exhibit a high degree of sensitivity to the selection of few shots. for example, different choices for the few shots can yield a performance variance of over 10% accuracy in a standard document classification task. motivated by this, we propose to fix the few shots for fair comparisons between different crosslingual transfer methods, and provide a benchmark resembling the standard "n -way k-shot" few-shot learning configuration @xcite . we also evaluate and compare several stateof-the-art (sota) few-shot finetuning techniques, in order to understand their performance and susceptibility to the variance related to few shots. we also demonstrate that the effectiveness of few-shot crosslingual transfer depends on the type of downstream task. for syntactic tasks such as named-entity recognition, the few shots can improve results by up to ≈20 f 1 points. for chal-lenging tasks like adversarial paraphrase identification, the few shots do not help and even sometimes lead to worse performance than zero-shot transfer. to understand these phenomena, we conduct additional in-depth analyses, and find that the models tend to utilize shallow lexical hints @xcite in the target language, rather than leveraging abstract crosslingual semantic features learned from the source language. our contributions: 1) we show that few-shot crosslingual transfer is prone to large variations in task performance; this property hinders unbiased assessments of the effectiveness of different fewshot methods. 2) to remedy this issue, we publish fixed and standardized few shots to support fair comparisons and reproducibility. 3) we empirically verify that few-shot crosslingual transfer has different performance impact on structurally different tasks; we provide in-depth analyses concerning the source of performance gains. 4) we analyze several sota few-shot learning methods, and show that they underperform simple full model finetuning. we hope that our work will shed new light on the potential and current difficulties of few-shot learning in crosslingual setups. recently, @xcite and hedderich et al. (2020) fs-xlt resembles the intermediate-task transfer (stilt) approach @xcite . in stilt, a pretrained encoder is finetuned on a resource-rich intermedi-ate task, and then finetuned on a (resource-lean) target task. likewise, fs-xlt focuses on transferring knowledge and general linguistic intelligence @xcite , although such transfer is between languages in the same task instead of between different tasks. few-shot learning was first explored in computer vision @xcite @xcite ; the aim there is to learn new concepts with only few images. methods like prototypical networks @xcite and modelagnostic meta-learning (maml; @xcite ) have also been applied to many monolingual (typically english) nlp tasks such as relation classification @xcite , namedentity recognition @xcite , word sense disambiguation @xcite , and text classification @xcite @xcite @xcite . however, recent few-shot learning methods in computer vision consisting of two simple finetuning stages, first on base-class images and then on new-class few shots, have been shown to outperform maml and achieve sota scores @xcite @xcite . inspired by this work, we compare various fewshot finetuning methods from computer vision in the context of fs-xlt. ## experimental setup we consider three types of tasks requiring varying degrees of semantic and syntactic knowledge transfer: sequence classification (cls), namedentity recognition (ner), and part-of-speech tagging (pos) in up to 40 typologically diverse languages (cf., appendix §b). ## conclusion and future work we have presented an extensive study of few-shot crosslingual transfer. the focus of the study has been on an empirically detected performance variance in few-shot scenarios: the models exhibit a high level of sensitivity to the choice of few shots. we analyzed and discussed the major causes of this variance across six diverse tasks for up to 40 languages. our results show that large language models tend to overfit to few shots quickly and mostly rely on shallow lexical features present in the few shots, though they have been trained with abundant data in english. moreover, we have empirically validated that state-of-the-art few-shot learning methods in computer vision do not outperform a conceptually simple alternative: full model finetuning. our study calls for more rigor and accurate reporting of the results of few-shot crosslingual transfer experiments. they should include score distributions over standardized and fixed few shots. to aid this goal, we have created and provided such fixed few shots as a standardized benchmark for six multilingual datasets. few-shot learning is promising for crosslingual transfer, because it mirrors how people acquire new languages, and that the few-shot data annotation is feasible. in future work, we will investigate more sophisticated techniques and extend the work to more nlp tasks.