M eta XL : Meta Representation Transformation for Low-resource Cross-lingual Learning (2021)

Abstract: The combination of multilingual pre-trained representations and cross-lingual transfer learning is one of the most effective methods for building functional NLP systems for low-resource languages. However, for extremely low-resource languages without large-scale monolingual corpora for pre-training or sufficient annotated data for fine-tuning, transfer learning remains an understudied and challenging task. Moreover, recent work shows that multilingual representations are surprisingly disjoint across languages, bringing additional challenges for transfer onto extremely low-resource languages. In this paper, we propose MetaXL, a meta-learning based framework that learns to transform representations judiciously from auxiliary languages to a target one and brings their representation spaces closer for effective transfer. Extensive experiments on real-world low-resource languages – without access to large-scale monolingual corpora or large amounts of labeled data – for tasks like cross-lingual sentiment analysis and named entity recognition show the effectiveness of our approach. Code for MetaXL is publicly available at github.com/microsoft/MetaXL.

## introduction recent advances in multilingual pre-trained representations have enabled success on a wide range of natural language processing (nlp) tasks for many languages. however, these techniques may not readily transfer onto extremely low-resource languages, where: (1) large-scale monolingual corpora are not available for pre-training and (2) sufficient labeled data is lacking for effective finetuning for downstream tasks. for example, multilingual bert (mbert) @xcite is pre-trained on 104 languages with many articles on wikipedia and xlm-r @xcite ) is pre-trained on 100 languages with commoncrawl corpora. however, these models still leave behind more than 200 languages with few articles available in wikipedia, not to mention the 6, 700 or so languages with no wikipedia text at all @xcite . cross-lingual transfer learning for these extremely low-resource languages is essential for better information access but under-studied in practice @xcite . recent work on cross-lingual transfer learning using pre-trained representations mainly focuses on transferring across languages that are already covered by existing representations @xcite . in contrast, existing work on transferring to languages without significant monolingual resources tends to be more sparse and typically focuses on specific tasks such as language modeling @xcite or entity linking @xcite . building nlp systems in these settings is challenging for several reasons. first, a lack of sufficient annotated data in the target language prevents effective fine-tuning. second, multilingual pre-trained representations are not directly transferable due to language disparities. though recent work on cross-lingual transfer mitigates this challenge, it still requires a sizeable monolingual corpus to train token embeddings @xcite . as noted, these corpora are difficult to obtain for many languages @xcite . additionally, recent work @xcite shows that contextualized representations of different languages do not always reside in the same space but are rather partitioned into clusters in multilingual models. this representation gap between languages suggests that joint training with combined multilingual data may lead to sub-optimal transfer across languages. this problem is further exacerbated by the, often large, lexical and syntactic differences between languages with existing pre-trained representations and the extremely lowresource ones. figure 1 (a) provides a visualization of one such example of the disjoint representations of a resource-rich auxiliary language (english) and resource-scarce target language (telugu). we propose a meta-learning based method, metaxl, to bridge this representation gap and allow for effective cross-lingual transfer to extremely low-resource languages. metaxl learns to transform representations from auxiliary languages in a way that maximally facilitates transfer to the target language. concretely, our meta-learning objective encourages transformations that increase the alignment between the gradients of the source-language set with those of a target-language set. figure 1(b) shows that metaxl successfully brings representations from seemingly distant languages closer for more effective transfer. we evaluate our method on two tasks: named entity recognition (ner) and sentiment analysis (sa). extensive experiments on 8 low-resource languages for ner and 2 low-resource languages for sa show that metaxl significantly improves over strong baselines by an average of 2.1 and 1.3 f1 score with xlm-r as the multilingual encoder. ## conclusions and future work in this paper, we study cross-lingual transfer learning for extremely low-resource languages without large-scale monolingual corpora for pre-training or sufficient annotated data for fine-tuning. to allow for effective transfer from resource-rich source languages and mitigate the representation gap between multilingual pre-trained representations, we propose metaxl to learn to transform representations from source languages that best benefits a given task on the target language. empirical evaluations on cross-lingual sentiment analysis and named entity recognition tasks demonstrate the effectiveness of our approach. further analysis on the learned transformations verify that metaxl indeed brings the representations of both source and target languages closer, thereby, explaining the performance gains. for future work, exploring transfer from multiple source languages to further improve the performance and investigating the placement of multiple representation transformation networks on multiple layers of the pre-trained models are both interesting directions to pursue.