Meta-X _{NLG} : A Meta-Learning Approach Based on Language Clustering for Zero-Shot Cross-Lingual Transfer and Generation (2022)

Abstract: Recently, the NLP community has witnessed a rapid advancement in multilingual and cross-lingual transfer research where the supervision is transferred from high-resource languages (HRLs) to low-resource languages (LRLs). However, the cross-lingual transfer is not uniform across languages, particularly in the zero-shot setting. Towards this goal, one promising research direction is to learn shareable structures across multiple tasks with limited annotated data. The downstream multilingual applications may benefit from such a learning setup as most of the languages across the globe are low-resource and share some structures with other languages. In this paper, we propose a novel meta-learning framework (called Meta-X _{NLG} ) to learn shareable structures from typologically diverse languages based on meta-learning and language clustering. This is a step towards uniform cross-lingual transfer for unseen languages. We first cluster the languages based on language representations and identify the centroid language of each cluster. Then, a meta-learning algorithm is trained with all centroid languages and evaluated on the other languages in the zero-shot setting. We demonstrate the effectiveness of this modeling on two NLG tasks (Abstractive Text Summarization and Question Generation), 5 popular datasets and 30 typologically diverse languages. Consistent improvements over strong baselines demonstrate the efficacy of the proposed framework. The careful design of the model makes this end-to-end NLG setup less vulnerable to the accidental translation problem, which is a prominent concern in zero-shot cross-lingual NLG tasks.

## introduction there are more than 7000 known living languages across the globe. 95% of the world's population does not speak english as their first language and 75% does not speak english at all 1 . most of the lan-1 https://www.ethnologue.com/statistics guages are low-resource languages as they do not have adequate resources for natural language processing research @xcite . on the other hand, a vast majority of studies in nlp research are conducted on english data @xcite . to democratize the nlp research for the benefit of the large global community, it is essential to focus on the non-english languages. however, creating/collecting task-specific annotated data for all the languages is expensive and time-consuming. moreover, human languages are dynamic as new words and domains are added continuously. an alternate solution is to investigate nlp modeling techniques that allow to train the model with highresource languages like english and transfer supervision to low-resource languages (with limited annotated data) or unseen languages for several nlp applications. recently, there has been promising progress on cross-lingual transfer learning research @xcite but supervision transfer is uneven across languages which leads to large performance gaps. such performance gaps are observed because models do not account for cultural and linguistic differences in the modeling @xcite . this paper is a step towards bridging this gap via metalearning and language clustering. meta-learning or learning to learn @xcite is a learning paradigm where the model is trained on diverse tasks and quickly adapts to new tasks given a handful of examples. it has emerged as a promising technique for machine learning @xcite , natural language understanding @xcite and machine translation @xcite tasks. this work -to the best of our knowledge -is the first attempt to study meta-learning techniques for cross-lingual natural language generation (x nlg ). particularly, we focus on zero-shot x nlg for low-resource languages. unlike nlu tasks, we observe that zero-shot nlg is a more challenging setup as text should be generated in unseen languages (which often suffers from accidental translation (at) problem @xcite ) and is expected to be grammatically coherent, semantically correct and fluent. we aim to address the following research problem: does meta-learning algorithm trained on typologically diverse languages (as training task) provide language-agnostic initialization for the zero-shot cross-lingual generation? our main contributions in this work are listed below: ## related work we focus on two threads of related work in this section: (1) cross-lingual generation and (2) metalearning for nlp. traditional approaches for crosslingual generation use machine translation (mt) in the modelling pipeline @xcite @xcite . such approaches have an inherent problem as translations are generally error-prone. the errors are more when at least one of the languages involved in the translation is a low-resource language. recently crosslingual transfer approaches are gaining attention. these methods use parallel data @xcite and small annotated datasets @xcite in zero-shot and few-shot cross-lingual generation respectively. lewis et al. (2020a) 2021 ) modified mbart pretrained model with an unsupervised dataset involving monolingual data in three languages for crosslingual transfer. this model, called zmbart, is tested on a small set of languages -english, hindi and japanese. moreover, it has been observed that such cross-lingual transfers are not uniform across the languages @xcite . we make an attempt to bridge this gap via metalearning. recently, meta-learning has been actively applied for many nlp applications @xcite and also for nlu tasks such as text classification @xcite , ner @xcite , task-oriented dialogue and qa (m'hamdi et al., 2021), etc. @xcite propose joint meta-learning approach on multiple languages and tasks from xtreme benchmark @xcite . close to our work, @xcite propose a meta-learning approach for cross-lingual transfer on nli and qa, both nlu tasks. the authors use one or two randomly selected languages for meta-training. in contrast, we provide a systematic approach based on language clustering to identify the right meta-training languages. moreover, to the best of our knowledge, ours is the first effort that employs meta-learning for natural language generation. ## meta-learning algorithm: maml meta-learning tries to learn structure among multiple tasks such that the new tasks are adapted quickly given few training instances. among several meta-learning algorithms, we focus on optimization-based algorithms, i.e., model agnostic meta-learning (maml) @xcite due to its recent success in multiple nlp and computer vision tasks. maml progresses in two phases: meta-training and adaptation. in the metatraining phase, the model learns a good initialization of parameter values by repeatedly simulating the learning process on training tasks. in the adaptation phase, these learned parameters are quickly adapted to new tasks. the underlying constraint is that all tasks should share some common structure (or come from a task distribution). the world's different languages follow this constraint as they came into existence with a common goal of communication, and share some structure. for meta-learning purposes, we treat them as different tasks. unlike traditional machine learning, metalearning has meta-train and meta-test data splits for meta-training and adaptation respectively. each split consists of tasks that are sampled from a distribution p(d) over task datasets {d 1 , d 2 , . . . , d n } where d i is associated with i th task t i . each d i has support set and query set d @xmath0, q i }. support set and query set are analogous to train and test splits of the traditional machine learning. we use f θ to denote a neural network model parameterized by θ. meta-training has two-levels of optimization: inner-loop optimization and outer-loop optimization. in the inner-loop optimization, for each sampled task t i , the task-specific model parameters θ m i are updated by m iterations of stochastic gradient decent (sgd) with support set s i . the overall model parameters θ are learned to optimize the performance of models f θ (m) i on query sets q i across datasets p(d) in the outer-loop optimization. the maml @xcite objective is: in outer-loop optimization, maml performs metaupdate which a batch as: where α is inner-loop learning rate and β is meta (outer-loop) learning rate. in the adaptation phase, the model is initialized with with learned optimal meta-parameters θ * , which is updated by a few steps of sgd with a support set (aka. few-shot learning) and directly evaluated on the query set of the meta-test dataset. our aim is to perform zeroshot evaluation, so we skip the adaptation phase and directly evaluate the learned model on metatest datasets. ## methodology in the proposed meta-x nlg framework, we first cluster the available languages and identify the centroid languages. then we train a model with maml on centroid languages to obtain an optimal initialization of parameters. finally, the learned model is deployed to generate text in the zero-shot setting. figure -1 provides an overview of proposed framework. we now provide details of each component of the framework. ## experiment setup we investigate meta-x nlg 's performance on two nlg tasks, five datasets and 30 languages. mt5 pre-trained model is used as the base model. the model performance is compared with two strong baselines in zero-shot setting. ## results and analysis automated evaluation results are shown in for the question generation task, meta-x nlg achieves better performance than others except for one experiment -indonesian language for tydiqa. for mlqa, improvements achieved by the proposed model are marginal (see table -6 ). upon close inspection, we notice that mlqa had small number of languages, and the centroid languages are very distinct, i.e. they have higher mean distance to other languages from same cluster as compared to the other datasets (see . this might be a possible reason for such performance. the human evaluation scores for all the three metrics are shown in table-7. the human evaluations (across both annotator sets) correlate with automatic evaluations. similar to the automatic evaluation, meta-x nlg consistently outperformed both baselines for selected languages, tasks and datasets. high fluency and relatedness scores for meta-x nlg indicates that most of generated text are fluent and not hallucinated respectively. the correctness metric considers both semantic and grammatical aspects; good scores on this metric indicate the acceptable performance for the proposed model in zero-shot setting. in qg, generating well-formed interrogative sentences is challenging, particularly in zero-shot setting due to unseen interrogative syntax structure of target language @xcite . the above-average fluency and correctness score for meta-x nlg indicates that the model quickly adapts such syntax and performs better. the consistent improvement in meta-x nlg for most the typologically diverse target languages provides evidence that supervision transfer is more uniform. considering decent automatic and manual evaluation scores in the zero-shot setting, we conclude that our model performs reasonably well except small performance gain with the mlqa dataset. meta-x nlg is a zero-shot framework, and we do not assume any prior training/knowledge for new unseen lrl. the only constraints are: the new language should be part of base pre-trained models (mt5) and adaptive unsupervised pre-training ## conclusion in this work, we propose a novel meta-x nlg framework based on meta-learning and language clustering for effective cross-lingual transfer and generation. this is the first study that uses meta-learning for zero-shot cross-lingual transfer and generation. the evaluations are done on two challenging tasks (ats and qg), five publicly available datasets and 30 languages. consistent improvement for both human and automatic evaluation metrics is observed over baselines. the cross-lingual transfer analysis indicates the model's ability towards uniform cross-lingual transfer across multiple low-resource languages. we will extend this study to more crosslingual tasks and languages in the future.