Good Meta-tasks Make A Better Cross-lingual Meta-transfer Learning for Low-resource Languages (2023)

Abstract: Model-agnostic meta-learning has garnered attention as a promising technique for enhancing few-shot cross-lingual transfer learning in low-resource scenarios. However, little attention was paid to the impact of data selection strategies on this cross-lingual meta-transfer method, particularly the sampling of cross-lingual meta-training data (i.e. meta-tasks) at the syntactic level to reduce language gaps. In this paper, we propose a Meta-Task Collector-based Cross-lingual Meta-Transfer framework (MeTaCo-XMT) to adapt different data selection strategies to construct meta-tasks for meta-transfer learning. Syntactic differences have an effect on transfer performance, so we consider a syntactic similarity sampling strategy and propose a syntactic distance metric model consisting of a syntactic encoder block based on the pre-trained model and a distance metric block using Word Move’s Distance (WMD). Additionally, we conduct experiments with three different data selection strategies to instantiate our framework and analyze their performance impact. Experimental results on two multilingual NLP datasets, Wikiann and TydiQA, demonstrate the significant superiority of our approach compared to existing strong baselines.

## introduction few-shot cross-lingual transfer surpasses zero-shot transfer @xcite @xcite using multilingual pre-trained language models (plms) @xcite @xcite @xcite . it significantly improves model performance in the target language with minimal annotation costs. recent studies have highlighted the benefits of model-agnostic meta-learning (maml) @xcite for few-shot cross-lingual transfer learning in nlp tasks @xcite @xcite . our focus in this paper is primarily on cross-lingual metatransfer methods for low-resource languages @xcite , utilizing support sets from high-resource languages to establish an effective initialization for training in the lowresource target language. the selection strategies for meta-tasks in crosslingual meta-transfer learning typically involve random sampling @xcite @xcite or semantic similarity @xcite . we compare these two strategies using the x-metra model (m'hamdi et al., 2021) on four languages from the tydiqa dataset @xcite . figure 1 illustrates that models trained with randomly sampled meta-tasks (x-metra seed ) in different seed setting generally perform a big variance of results compared with x-metra imp . different random seed means the different selection of in-stances to construct meta-task, and better selection can generate better results. it emphasize the pivotal role of meta-task construction in enhancing cross-lingual meta-transfer learning. motivated by the above finding, we propose a meta-task collector-based cross-lingual meta-transfer framework (metaco-xmt). as shown in figure 2 (a), the meta-task collector includes a data encoder and data selector with a distance metric block, which can be designed according to different data selection strategies. for instance, multilingual semantic representation and cosine similarity metrics can realize semantic similarity sampling strategy. specifically, the data encoder encodes all data to semantic space, and the data selector selects topv cosine-similar support instances for each query data as candidates. finally, the meta-task can be constructed according to the setting. however, the semantic similarity sampling is limited for different tasks. for example, in the machine reading comprehension (mrc) task, most semantically similar samples are subject-related, ignoring the relationship between question and paragraph. in practice, different questions with a common paragraph may choose the same support instance. for structural tasks like named entity recognition (ner), syntactic similarity sampling is more effective due to language-specific syntactic differences. therefore, we propose a syntactic distance metric model (sdmm) based on multilingual plm and the word mover's distance (wmd) @xcite . as shown in figure 2(b) , the sdmm incorporates a syntactic linear layer for syntactic tree learning and employs a triplet loss to distinguish wmd between close and distant languages from pivot languages (e.g., english). in order to further explore the impact of data selection strategy on cross-lingual meta-transfer performance, we compare three sampling strategies to build meta-tasks including semantic similarity sampling, task-similarity sampling, and syntactic similarity sampling. we conduct experiments on 13 typologically diverse target languages of two cross-lingual tasks: nre and mrc. our main contributions are listed below: ## related work we focus on two threads of related work: (1) metalearning for cross-lingual transfer and (2) training data selection. @xcite and @xcite use meta-learning for crosslingual ner and semantic parsing with a slight enhancement in minimal resources. x-maml @xcite combines the maml and cross-lingual transfer method based on plm and demonstrates improvement in zero-shot and few-shot settings. x-maml samples the support and query data from the same language, which limits the ability of the model for cross-lingual transfer. xla-maml @xcite performs direct cross-lingual adaptation in the meta-learning stage by sampling the meta-tasks from two or more languages. x-metra-ada (m'hamdi et al., 2021) follows the setting of cross-lingual meta-transfer training and adds a meta-adaptation stage for further improvement. while @xcite and m'hamdi et al. (2021) use semantic similarity to select the meta-tasks, we explore more data selection strategies to get better cross-lingual meta-transfer performance. training data selection has been extensively studied for several tasks, such as domain adaptation @xcite and crosslingual transfer @xcite . in the cross-lingual transfer setting, the selection of training data can reduce the performance gaps across languages. @xcite proposed approaches of data selection rely on multiple measures such as data entropy using an n-gram language model, predictive entropy, and gradient embedding. maurya and desarkar (2022) use meta-learning techniques for cross-lingual generation and choose centroid languages to metatraining the model and improve other languages. in this paper, we propose a cross-lingual meta-transfer framework based on a meta-task collector and explore the performance of different data sampling strategies. the optimized-based meta-learning algorithm can perform the model fast adaptation to a new task during the adaptation training phase. cross-lingual meta-transfer learning mainly focuses on the meta-training phase of maml. in the inner step, the model f θ learns a good initialization of parameter θ ′ by repeatedly simulating the learning process on source language support set @xmath0. the meta step is fine-tuning the initialized model in a target language query set @xmath1. these two steps are iterated repeatedly to optimize the parameters θ. we skip the adaptation phase and directly evaluate the target language test dataset. ## methodology figure 2 shows the architecture of our meta-task collector-based cross-lingual meta-transfer framework and the proposed syntactic distance metric model. our framework consists of three essential components: task-specific base plm, cross-lingual meta-transfer learner, and meta-task collector. in this section, we first introduce the training procedure of metaco-xmt ( § 4.1) and a detailed description of the meta-task collector ( § 4.2). then syntactic distance metric model ( § 4.3) is proposed to instantiate the meta-task collector and two other data selection strategies are described in § 4.4. ## analysis our analytical experiments consider the performance of the four cross-lingual meta-transfer models with random, semantic-similar, task-similar, and syntactic-similar samples named 'random', 'sem', 'task', and 'syn' for short, respectively. ## conclusion in this paper, we have presented a novel meta-task collector-based cross-lingual meta-transfer framework, which can adapt different meta-task selection strategies to construct meta-transfer training data, reducing the cross-lingual performance gap between languages due to language differences. to close the syntactic distance between languages, we propose the syntactic distance metric model that encodes text pairs to syntactic space and selects meta-task by wmd for meta-transfer learning. two other data selection strategies are explored: semantic similarity sampling and tasksimilarity sampling. we demonstrate the validity of our framework on both the ner and mrc tasks, especially the syntactic similar samplingbased method reaches a new state-of-the-art for most languages. further analyses suggest that the proposed metaco-xmt with syntactic similar sampling can effectively improve the cross-lingual transfer performance with only a small amount of data, especially for low-resource languages.