Nearest Neighbour Few-Shot Learning for Cross-lingual Classification (2021)

Abstract: Even though large pre-trained multilingual models (e.g. mBERT, XLM-R) have led to significant performance gains on a wide range of cross-lingual NLP tasks, success on many downstream tasks still relies on the availability of sufficient annotated data. Traditional fine-tuning of pre-trained models using only a few target samples can cause over-fitting. This can be quite limiting as most languages in the world are under-resourced. In this work, we investigate cross-lingual adaptation using a simple nearest-neighbor few-shot ( <15 samples) inference technique for classification tasks. We experiment using a total of 16 distinct languages across two NLP tasks- XNLI and PAWS-X. Our approach consistently improves traditional fine-tuning using only a handful of labeled samples in target locales. We also demonstrate its generalization capability across tasks.

## introduction the rise of massively pre-trained multilingual language models (lm) 1 @xcite @xcite @xcite ) has significantly improved cross-lingual generalization across many languages @xcite @xcite . recent work on zeroshot cross-lingual adaptation @xcite @xcite , in the absence of labelled target data, has also demonstrated impressive performance gains. despite these successes, however, there still remains a sizeable gap between supervised and zero-shot performances. on the other hand, when limited target language data are available (i.e few-shot setting), traditional fine-tuning of large pre-trained models can cause over-fitting @xcite . one way to deal with the scarcity of annotated data is to augment synthetic data using techniques like paraphrasing @xcite , word translation @xcite @xcite , machine translation @xcite , data-augmentation @xcite @xcite and/or data-diversification @xcite @xcite . few-shot learning, on the other hand, deals with handling out-of-distribution (ood) generalization problems using only a small amount of data @xcite @xcite @xcite . in this setup, the model is evaluated over few-shot tasks, such that the model learns to generalize to new data (query set) using only a hand full of labeled samples (support set). in a cross-lingual few-shot setup, the model learns cross-lingual features to generalize to new languages. recently, @xcite used meta-learning @xcite for few-shot adaptation on several cross-lingual tasks. their fewshot setup used full development datasets of various target languages (xnli development set, for instance, has over 2k samples). in general, they showed the effectiveness of cross-lingual metatraining in the presence of a large quantity of ood data. however, they did not provide any fine-tuning baseline. on the contrary, @xcite explored few-shot learning but did not explore beyond fine-tuning. to the best of our knowledge, there has been no prior work in cross-lingual nlp that uses only a handful of target samples (< 15) and yet surpasses or matches traditional fine-tuning (on the same number of samples). traditional finetuning (parametric) approaches require proper hyperparameter tuning techniques for the learning rate, scheduling, optimizer, batch size, up-sampling few-shot support samples and failing to do so would often led to model over-fitting. it can be expensive to update parameters of large model frequently for few shot adaption, each time there is a fresh batch of support samples. as the model grows bigger, it becomes almost unscalable to update weights frequently for few shot adaptation. it takes significant amount of time to update gradients for a few number of samples and then perform inference. there have been previous successful attempts to inject external knowledge via non-parametric methods @xcite @xcite . in this work, we explore a simple nearest neighbor few-shot inference (nnfs) approach for crosslingual classification tasks. our main objective is to utilize very few samples to perform adaptation on a given target language. to achieve this, we first fine-tune a multilingual lm on a high resource source language (i.e., english), and then apply fewshot inference using few support examples from the target language. unlike other popular metalearning approaches that focus on improving the fine-tuning/training setup to achieve better generalization @xcite , our approach applies to the inference phase. hence, we do not update the weights of the lm using target language samples. this makes our approach complimentary to other regularized finetuning based few-shot meta-learning approaches. our key contributions are as follows: we also opensource our implementation 2 . the objective of few-shot learning is to adapt from a source distribution to a new target distribution using only few samples. the traditional fewshot setup @xcite @xcite involves adapting a model to the distribution of new classes. similarly, in a crosslingual setup, we adapt a pre-trained lm, that has been fine-tuned using a high resource language, to a new target language distribution @xcite . ## conclusion the paper proposes a nearest neighbour based fewshot adaptation algorithm accompanied by a necessary evaluation protocol. our approach does not require updating the lm weights and thus avoids over-fitting to limited samples. we experiment using two classification tasks and results demonstrate consistent improvements over finetuning not only across languages, but also across tasks. imtiaz masud ziko, jose dolz, eric granger, and ismail ben ayed. 2020. laplacian regularized fewshot learning.