”Diversity and Uncertainty in Moderation” are the Key to Data Selection for Multilingual Few-shot Transfer (2022)

Abstract: Few-shot transfer often shows substantial gain over zero-shot transfer (CITATION), which is a practically useful trade-off between fully supervised and unsupervised learning approaches for multilingual pretained model-based systems. This paper explores various strategies for selecting data for annotation that can result in a better few-shot transfer. The proposed approaches rely on multiple measures such as data entropy using n -gram language model, predictive entropy, and gradient embedding. We propose a loss embedding method for sequence labeling tasks, which induces diversity and uncertainty sampling similar to gradient embedding. The proposed data selection strategies are evaluated and compared for POS tagging, NER, and NLI tasks for up to 20 languages. Our experiments show that the gradient and loss embedding-based strategies consistently outperform random data selection baselines, with gains varying with the initial performance of the zero-shot transfer. Furthermore, the proposed method shows similar trends in improvement even when the model is fine-tuned using a lower proportion of the original task-specific labeled training data for zero-shot transfer.

## introduction language resource distribution, for both labeled and unlabeled data, across the world's languages is extremely skewed, with more than 95% of the languages having hardly any task-specific labeled data @xcite . therefore, cross-lingual zero-shot transfer using pretrained deep multilingual language models has received significant attention from the nlp community. during crosslingual zero-shot transfer, first a multilingual model @xcite @xcite @xcite is created using only unlabelled data from a large number of languages (typically in the range of 100) with some selfsupervised learning objectives. these pretrained models are then fine-tuned with task-specific labeled data from one or more languages (we refer to these as the pivot languages) and tested on all the other languages (here referred to as the target languages) for which no annotated data was used during fine-tuning. many recent work @xcite @xcite @xcite have studied the efficacy of zero-shot cross-lingual transfer across languages and factors influencing it. other work have shown that a few-shot transfer, where very little labeled data in the target language is also used during fine-tuning, can result in substantial gains over the zero-shot transfer. for instance, @xcite show that zero-shot transfer does not hold much promise for transfer across typologically different languages or when there is not enough unlabeled data in the target language during model pretraining. in such cases, the gap in the cross-lingual transfer can be effectively reduced by fine-tuning it on a little annotated data in the target language. however, very few languages have readily available annotated resources for different nlp tasks, and collecting annotated data for a large set of target languages can be expensive and time-consuming @xcite @xcite . therefore, it is essential to carefully select and annotate target language data for a few-shot transfer, reducing the transfer gap effectively. training data selection has been investigated for several nlp tasks, especially for domain adaptation @xcite @xcite . the majority of these approaches use different techniques to rank the entire data and use top n data points to train the system @xcite . in addition, active learning @xcite has been widely used to improve annotation efficiently by using model predictions to select informative data. active learning is generally used in an iterative setting, in which a model is learned at each iteration, and samples are selected for labeling to improve performance. however, in this paper, we are trying to select a few samples. hence we are limiting the training to one iteration. in the past, @xcite have used active learning to annotate only uncertain entity spans for dutch and hindi languages. however, to the best of our knowledge, none of these approaches have been studied for a large set of languages in a cross-lingual few-shot transfer setting. the central goal of this work is to propose specific strategies for data selection (and subsequent annotation) for few-shot learning so that the performance in a target language is maximized, given a data budget. the main contributions of this work are: @xcite we propose different data selection strategies based on the notions of cross-entropy, predictive entropy, gradient embedding and loss embedding, and perform various reliability analyses of these strategies. @xcite we conduct experiments on a set of 20 typologically diverse languages including some syntactically divergent from the pivot language -english & chinese. @xcite we propose a loss embedding-based method for sequence labeling tasks which incorporates both diversity and uncertainty sampling. [4] through experiments on three nlp tasks, we show that embedding-based strategies perform consistently better than random data selection baselines, with gains varying with the initial performance of the zero-shot transfer. we also observe several language and data-size dependent trends in the performance across different data selection strategies. [5] finally, we provide a concrete set of recommendations for data selection based on features such as zero-shot performance and the amount of unlabeled data available for a target-language. the rest of the paper is organized as follows. the next section introduces the novel data sampling strategies. section 3, 4, and 5 present the experimental setup, results and related research in the area, respectively. concluding remarks are made in section 6. ## data sampling strategies assuming we have a pre-trained multilingual language model and enough labelled data in a particu-lar language such as english (en) for fine-tuning on a task. we can measure the zero-shot performance on a set of target languages. we observe that zeroshot performances are not uniform and often vary with the typological similarity between the target and pivot language as stated by @xcite . nevertheless, all the target languages show a drop in zero-shot performance compared to the performance achieved in the pivot language. hence, there is a cross-lingual transfer gap for all the target languages. this gap can be attributed to the inherent linguistic property of the target languages however, @xcite have shown that the cross-lingual transfer gap can be reduced by fine-tuning on a little annotated data in a target language. consequently, few-shot performance can reduce the transfer gap for all the languages. given a fixed budget, let say k examples, we want to maximize the few-shot performance in a target language by carefully choosing the effective k examples. to this end, we are proposing several data selection strategies in this section. we compare them with random sampling where k target language examples are randomly selected from task-specific fine-tuning data collection. note that the sampling strategies are oblivious to the actual labels of the data points, as annotation would follow the data selection step in practice. ## experimental setup we conduct various experiments to evaluate effectiveness of our proposed data sampling techniques in a few-shot transfer setting with up to 20 languages from various language families on two different sequential tasks and one classification task. ## results we calculated the difference between the f 1 -scores of few-shot and zero-shot setups, deltas(△), for each language separately, but we observed different sampling strategies to work better depending upon the cross-lingual transfer gap. therefore, we present the experimental results after categorizing languages by the transfer gap as indicated by the zero-shot performance, shown in we report the deltas, for ner and pos tasks in for xnli, the averaged deltas across all languages are reported in we observe low few-shot gains for pe as it does not induce diversity. to measure the impact of pivot size, we trained a zero-shot model with 40k en examples and observe similar trends for both dce and ge (see ## impact of pivot language we conduct few-shot experiments considering zh as the pivot language to validate the effectiveness of our method across different pivots. the delta between the gains using zh as the pivot have been reported in ## discussion and conclusion this work explored various data sampling strategies for few-shot learning for two sequence labeling and a semantic tasks on 20 target languages. our study shows that the embedding-based strategies, le and ge, consistently outperform random sampling baseline across languages and sample sizes. some of the salient observations are as follows. on ner and pos tasks, languages of the group c 2 show significant improvements in few-shot performance, suggesting that the gains from few-shot learning are strongly correlated to the zero-shot transfer gap. le and ge-based data selection methods show consistent gains over the rand strategy for each target language group, but these gains saturate as the sample size, k, increases beyond 500. the saturation occurs due to the relatively smaller target-language corpus size (varies between 5k and 20k for ner and pos, respectively) effectively reducing the diversity in the total sample. le provides better few-shot performance than pe in terms of statistical significance. dce only performs better than rand for thai. as dce does not use any form of information from the fine-tuned model and if the target-language corpus size is small, it fails to select novel target language examples any better than rand. however, in th, for which zero-shot performance is close to zero, dce selects the highly representative and diverse training examples for small values of k. the trends for xnli are different from that of the sequence labeling tasks. ge and dce outperform all other methods, with gains increasing with the value of k, which suggests that the size of the targetlanguage corpus is crucial for data selection. xnli has about 400k examples in each target-language corpus, much larger than that of ner and pos, signifying the importance of diversity sampling. based on our observations, we recommend the le-based sampling strategy for data selection for cross-lingual few-shot transfer for sequence labeling tasks and ge-based sampling for classification tasks. while the optimal parameter setting for the le sampling algorithm varies across tasks, we recommend the vanilla le method without any parameter for most of the tasks. for tasks having higher number of class labels, we recommend using le variant with λ such as 0 or 0.5. in future the work can be extended to other highlevel tasks such as cross-lingual qa and machine translation. we would also like to extend this work in a reinforcement learning @xcite or meta-learning @xcite ) framework, where the parameters can be automatically learnt for various tasks and settings.