Enhancing Few-shot Cross-lingual Transfer with Target Language Peculiar Examples (2023)

Abstract: Few-shot cross-lingual transfer, fine-tuning Multilingual Masked Language Model (MMLM) with source language labeled data and a small amount of target language labeled data, provides excellent performance in the target language. However, if no labeled data in the target language are available, they need to be created through human annotations. In this study, we devise a metric to select annotation candidates from an unlabeled data pool that efficiently enhance accuracy for few-shot cross-lingual transfer. It is known that training a model with hard examples is important to improve the model’s performance. Therefore, we first identify examples that MMLM cannot solve in a zero-shot cross-lingual transfer setting and demonstrate that it is hard to predict peculiar examples in the target language, i.e., the examples distant from the source language examples in cross-lingual semantic space of the MMLM.We then choose high peculiarity examples as annotation candidates and perform few-shot cross-lingual transfer. In comprehensive experiments with 20 languages and 6 tasks, we demonstrate that the high peculiarity examples improve the target language accuracy compared to other candidate selection methods proposed in previous studies.

## introduction sufficient labeled data is essential to train an accurate model. however, few languages have abundant language resources for both labeled and unlabeled data like english @xcite . in addition, constructing a large amount of labeled data through human annotators is costly and time-consuming. the use of multilingual mask language models (mmlms) is one way to overcome this problem * now at hitotsubashi university as they show good zero-shot cross-lingual performance in target languages by fine-tuning with only task-specific labeled data of source language, such as english. while this zero-shot cross-lingual transfer ability is promising for the target languages with no or limited task-specific resources, there is a divergence in accuracy between the source and target languages, meaning the zero-shot cross-lingual transfer ability is imperfect. to analyze the characteristics of hard examples for predictions in zero-shot cross-lingual transfer, we visualize representations of the source and target languages' examples and highlight the target ones with accuracy in the zero-shot setting, as shown in figure 1 . we can observe that the accuracies of target language examples distant from the source language examples are low compared to others. we refer to these as peculiar examples of the target language. we should address peculiar examples to further enhance performance in the target language. few-shot cross-lingual transfer, adapting mmlm with a small number (0.1k-1k) of task-specific labeled examples in the target language, is a promising approach to enhance performance for target languages. specifically, @xcite showed that a small number of examples randomly selected from labeled dataset significantly improves the accuracy for the target languages. however, if there are no labeled examples at all, we should create them by selecting annotation candidates from an unlabeled data pool. from this perspective, @xcite evaluated candidate selection methods proposed in active learning research for selecting the annotation candidates for few-shot cross-lingual transfer. they split the target languages based on their zero-shot cross-lingual transfer performance into "good" and "poor" (or "good", "fair", and "poor") language groups 1 and demonstrated that effective methods vary for each language group. our preliminary analysis (figure 1 ) suggests that adapting mmlm for peculiar examples is crucial to improve performance for the target language and recommends selecting peculiar examples as annotation candidates. therefore, in this study, we first propose a metric to measure peculiarity of the target language examples. note that peculiarity is defined without labels of the downstream tasks. then, we select high peculiarity examples as annotation candidates and conduct few-shot cross-lingual transfer using languages with "good" and "poor" zero-shot crosslingual performance. our experiments show that the proposed metric peculiarity is very simple yet effective in selecting candidates for few-shot crosslingual transfer. our contributions in this study are threefold: target language labeled data d' t m s target language test data d t,test m @xmath0,test x' t,test for hyperparameters and brings consistent performance in few-shot cross-lingual transfer. ## experimental settings we explain experimental settings, which are largely similar to those used in previous works such as @xcite and @xcite , including the dataset and training hyperparameters. ## experimental results in this section, we confirm the effectiveness of peculiarity. we first show that peculiarity isolates the examples based on their accuracy in the zeroshot cross-lingual transfer setting (subsection 6.1), and then demonstrate that the examples of high peculiarity are useful for few-shot cross-lingual transfer (subsection 6.2). ## conclusion in this study, we proposed a simple metric called peculiarity, which measures whether source language examples exist in the neighborhood of target language examples. we showed that high peculiarity examples are not predicted correctly in the zeroshot setting and demonstrated that these examples can enhance accuracy for few-shot cross-lingual transfer regardless of language groups. in addition, we showed that peculiarity-km, the candidate selection method that considers peculiarity and diversity, further boosts few-shot crosslingual transfer performance compared to peculiarity or km alone. however, there are some configurations in which peculiarity-km does not work well, such as the "good" group in marc-5. therefore, we would like to attempt to analyze the causes of this result and develop a new candidate selection method that enhances the few-shot cross-lingual performance across languages and tasks.