Learn to Cross-lingual Transfer with Meta Graph Learning Across Heterogeneous Languages (2020)

Abstract: Recent emergence of multilingual pre-training language model (mPLM) has enabled breakthroughs on various downstream cross-lingual transfer (CLT) tasks. However, mPLM-based methods usually involve two problems: (1) simply fine-tuning may not adapt general-purpose multilingual representations to be task-aware on low-resource languages; (2) ignore how cross-lingual adaptation happens for downstream tasks. To address the issues, we propose a meta graph learning (MGL) method. Unlike prior works that transfer from scratch, MGL can learn to cross-lingual transfer by extracting meta-knowledge from historical CLT experiences (tasks), making mPLM insensitive to low-resource languages. Besides, for each CLT task, MGL formulates its transfer process as information propagation over a dynamic graph, where the geometric structure can automatically capture intrinsic language relationships to explicitly guide cross-lingual transfer. Empirically, extensive experiments on both public and real-world datasets demonstrate the effectiveness of the MGL method.

## introduction the diversity of human languages is a critical challenge for natural language processing. to alleviate the cost in annotating data for each task in each language, cross-lingual transfer (clt) @xcite , aiming to leverage knowledge from source languages that are sufficiently labeled to improve the learning in a target language with little supervision, has become a promising direction. to bridge the gaps between languages, numerous clt algorithms have emerged, ranging from early translation-based methods @xcite , cross-lingual word representation * the work was done when zheng li was an intern at amazon.com inc. we thank the support of hong @xcite and nsfc 61673202. learning @xcite , to powerful mplm @xcite , from which the versatile multilingual representations derived suffice it to become a mainstream approach for various downstream clt tasks. however, existing mplm-based methods focus on designing costly model pre-training while ignoring equally crucial downstream adaptation. with simply fine-tuning on the downstream labeled data for clt tasks, mplm often underperforms on lowresource target languages, especially for the languages distant from the source ones since the generalization ability of mplm highly relies on lexical overlap across languages @xcite . on the other hand, existing adaptation approaches for mplm behave as a black box without explicitly identifying intrinsic language relations. to address the issues, we propose meta graph learning (mgl), a meta learning framework to learn how to cross-lingual transfer for mplm. specifically, mgl models each clt process as heterogeneous information propagation over a dynamic graph, which captures latent language correlations and makes the downstream clt adaptation more interpretable. however, solely learning the dynamic graph structures may be insufficient since the graph-based metric space usually favors highresource languages over low-resource ones. meta learning, a.k.a learning to learn, @xcite , addresses the few-shot problems, by extracting common metaknowledge from previous tasks (meta-train) that can be rapidly adapted to new tasks with a few examples (meta-test). inspired by it, mgl takes advantage of historical clt experiences to quickly adapt the dynamic graphs to our target clt task. this enables mgl to meta-learn a graph-based cross-lingual metric space that is invariant across languages. for example, suppose we transfer from english (en), french (fr) and german (de) to japanese (ja), i.e., {en, fr, de}→ja. we construct previous clt experiences by leave-one-out among source languages: for each source clt task, we leave one out of source languages as a pseudo-target language in turn and use the remaining ones as the pseudo-source languages. as such, we expect the mgl can borrow knowledge from source clt pairs: {fr, de}→en, {en, de}→fr, {en, fr}→de to improve the transfer effectiveness in the target clt pair {en, fr, de}→ja, which is illustrated in figure 1 . recently, some efforts have been initiated on meta learning for low-resource nlp tasks that straightforwardly views each dataset with its objective as a task @xcite . however, this strategy can only make meta-leaner learn knowledge from each language separately. and meanwhile, most existing meta-learners lack the ability to handle tasks lying in different distributions, especially tasks for heterogeneous languages. on the contrary, mgl resorts to learning how to adapt across languages from each clt task. empirically, extensive experiments on both the public multilingual amazon review dataset @xcite and the real-world industrial multilingual search relevance dataset @xcite demonstrate the effectiveness of the mgl method. overall, our contributions can be summarized as follows: (1) a novel mgl method is proposed to learn to cross-lingual transfer (l2clt) for taskaware adaptation of mplm by leveraging previous clt experiences; (2) the mgl automatically captures intrinsic correlations between languages, which improves the interpretability of the downstream adaptation process; (3) extensive experiments verify the effectiveness of the mgl. 2 related work ## problem definition t high-resource (source) languages {ℓ s i } t @xmath0. each source language ℓ s i has sufficient labeled data d ℓ s @xmath1, y j ℓ s i } |ℓ s i | @xmath2, where |ℓ s i | is the number of labeled data for the i-th source language ℓ s i . besides, only a few labeled data d ℓ @xmath3, y j ℓ t } |ℓ t | @xmath4, i.e., |ℓ t | |ℓ s i |, ∀@xmath5, t ]. all languages share the same label space, i.e., the label set y. our goal aims to leverage knowledge from high-resource languages {ℓ s i } t @xmath6, i.e., {ℓ s i } t @xmath7. ## methodology our framework is a language-agnostic task-aware model for clt. on the one hand, we use the mplm as the base encoder to calculate language-agnostic representations. on the other hand, we propose a meta graph learning (mgl) method to further guide the versatile multilingual representations to be task-aware for downstream clt tasks. ## experiment in this section, we present an extensive set of experiments across two datasets. the first experiment is on a public multilingual amazon review dataset @xcite . in addition, we conduct experiments on a real-world industrial multilingual search relevance dataset @xcite used for e-commerce product search. ## implementation details encoder we use the bert-base-multilingualcased foot_1 model pre-trained on 104 languages as the encoder, which has 12 layers, 768-d hidden size, 12 heads and 110m total parameters. the mbert is jointly optimized with other parameters during both the meta-train and meta-test stages. ## conclusion and future works in this paper, we propose a novel mgl method for task-aware clt adaptation of mplm by leveraging historical clt experiences. extensive evaluations on both the public benchmark and large-scale industrial dataset quantitively and qualitatively demonstrate the effectiveness of the mgl. in the future, the proposed mgl method can potentially applied to more cross-lingual natural language understanding (xlu) tasks @xcite @xcite , and be generalized to learn to learn for domain adaptation @xcite , representation learning @xcite , multi-task learning @xcite