Meta-Learning for Fast Cross-Lingual Adaptation in Dependency Parsing (2022)

Abstract: Meta-learning, or learning to learn, is a technique that can help to overcome resource scarcity in cross-lingual NLP problems, by enabling fast adaptation to new tasks. We apply model-agnostic meta-learning (MAML) to the task of cross-lingual dependency parsing. We train our model on a diverse set of languages to learn a parameter initialization that can adapt quickly to new languages. We find that meta-learning with pre-training can significantly improve upon the performance of language transfer and standard supervised learning baselines for a variety of unseen, typologically diverse, and low-resource languages, in a few-shot learning setup.

## introduction the field of natural language processing (nlp) has seen substantial performance improvements due to large-scale language model pre-training @xcite . whilst providing an informed starting point for subsequent task-specific fine-tuning, such models still require large annotated training sets for the task at hand @xcite . this limits their applicability to a handful of languages for which such resources are available and leads to an imbalance in nlp technology's quality and availability across linguistic communities. aiming to address this problem, recent research has focused on the development of multilingual sentence encoders, such as multilingual bert (mbert) @xcite and xlm-r @xcite , trained on as many as 93 languages. such pre-trained multilingual encoders enable zero-shot transfer of task-specific models across languages @xcite , offering a possible solution to resource scarcity. zero-shot transfer, how-corresponding author: annalangedijk@gmail.com. ever, is most successful among typologically similar, high-resource languages, and less so for languages distant from the training languages and in resource-lean scenarios @xcite . this stresses the need to develop techniques for fast cross-lingual model adaptation, that can transfer knowledge across a wide range of typologically diverse languages with limited supervision. in this paper, we focus on the task of universal dependency (ud) parsing and present a novel approach for effective and resource-lean cross-lingual parser adaptation via meta-learning, requiring only a small number of training examples per language (which are easy to obtain even for low-resource languages). meta-learning is a learning paradigm that leverages previous experience from a set of tasks to solve a new task efficiently. as our goal is fast cross-lingual model adaptation, we focus on optimization-based meta-learning, where the main objective is to find a set of initial parameters from which rapid adaption to a variety of different tasks becomes possible @xcite . optimization-based meta-learning has been successfully applied to a variety of nlp tasks. notable examples include neural machine translation @xcite , semantic parsing @xcite , pre-training text representations @xcite , word sense disambiguation @xcite and cross-lingual natural language inference and question answering @xcite . to the best of our knowledge, meta-learning has not yet been explored in the context of dependency parsing. we take inspiration from recent research on universal dependency parsing @xcite . we employ an existing ud parsing framework -udify, a multitask learning model @xcite -and extend it to perform few-shot model adap-tation to previously unseen languages via metalearning. we pre-train the dependency parser on a high-resource language prior to applying the modelagnostic meta-learning (maml) algorithm @xcite to a collection of few-shot tasks in a diverse set of languages. we evaluate our model on its ability to perform few-shot adaptation to unseen languages, from as few as 20 examples. our results demonstrate that our methods outperform language transfer and multilingual joint learning baselines, as well as existing (zero-shot) ud parsing approaches, on a range of language families, with the most notable improvements among the low-resource languages. we also investigate the role of a pre-training language as a starting point for cross-lingual adaptation and the effect of typological properties on the learning process. ## dataset we use data from the universal dependencies v2.3 corpus @xcite . we use treebanks from 26 languages that are selected for their typological diversity. we adopt the categorization of highresource and low-resource languages from @xcite and employ their set of training and test languages for comparability. the set covers languages from six language families (indo-european, korean, afro-asiatic, uralic, dravidian, austro-asiatic). their training set (expmix) includes eight languages: english, arabic, czech, hindi, italian, korean, norwegian, and russian. these languages fall into the language families of indo-european, korean and afro-asiatic and have diverse word orders (i.e. vso, svo and sov). joint learning on data from this diverse set yielded state-of-the-art zero-shot transfer performance on low-resource languages in the experiments of @xcite . per training language we use up to 20,000 example trees, predicting dependency arc labels from 132 classes total. we select bulgarian (indo-european) and telugu (dravidian) as validation languages to improve generalization to multiple language families. the 16 test languages cover three new language families that were unseen dur-ing training, i.e. austro-asiatic, dravidian, and uralic. furthermore, three of our test languages (buryat, faroese, and upper sorbian) are not included in the pre-training of mbert. we refer the reader to appendix b for details about the treebank sizes and language families. ## experimental setup we extend the existing udify code foot_0 to be used in a meta-learning setup. all of our code is publicly available. foot_1 ## discussion our experiments confirm that meta-learning, specifically maml, is able to adapt to unseen languages on the task of cross-lingual dependency parsing maml (en)maml (hin) svo ( ) adposition before noun ( ) possessor after noun ( ) negative word after verb ( ) negative word before object ( ) prosubject word ( ) vox ( ) oblique after verb ( ) sov ( ) adposition after noun ( ) object headmark ( ) any redup. ( ) comitative vs instrumental mark ( ) prosubject affix ( ) oxv ( ) oblique before verb ( ) feature in uriel -0.56* 0.67* -0.51* 0.81* -0.65* 0.81* -0.6* 0.75* -0.7* 0.67* -0.61* 0.26 -0.48 0.8* -0.48 0.8* 0.64* -0.51* 0.65* -0.81* 0.57* -0.16 0.48 -0.7* 0.45 -0.86* 0.61* -0.26 0.63* -0.72* 0.48 -0.8* riqrqsurmhfwlyhwuhhv shuirupdqfhlqfuhdvh @xmath0udelf &]hfk (qjolvk )lqqlvk )uhqfk *hupdq +lqgl +xqj ,wdoldq -dsdqhvh .ruhdq 1ru 3huvldq 5xvvldq 6zhglvk 7dplo 8ugx 9lhwq %xojduldq 7hoxjx figure 4: spearman's ρ between the % of nonprojective dependency trees and maml's improvement over the english baseline (ρ = 0.42, @xmath1.03). our ablation experiments on pre-training show that it is beneficial for maml to start from a strong set of parameters, pre-trained on a high resource language. thereby, the pre-training is not dependent on a specific language. maml performs well with a variety of pre-training languages, although improvements for unseen languages vary. when a model is pre-trained on english, there is a large positive correlation for improvements in languages that are syntactically dissimilar to english, such as japanese and tamil. during meta-training, dissimilar training languages such as hindi most contribute to the model's ability to generalize. syntactic features, especially those related to word order, which have already been learned during pretraining, require less adaptation. the same is true, vice versa, for hindi pre-training. this effect is also observed, though only in one direction, when correlating performance increase with non-projectivity. it is beneficial to meta-train on a set of languages that vary in projectivity after pre-training on one which is mostly projective. however, not all variance is explained by the difference in typological features. the fact that maml outperforms maml without pretraining suggests that pre-training also contributes language-agnostic syntactic features, which is indeed the overall goal of multi-lingual ud models. ## conclusion in this paper, we present a meta-learning approach for the task of cross-lingual dependency parsing. our experiments show that meta-learning can improve few-shot universal dependency parsing performance on unseen, unrelated test languages, including low-resource languages and those not covered by mbert. in addition, we see that it is beneficial to pre-train before meta-training, as in the x-maml approach @xcite . in particular, the pre-training language can affect how much adaptation is necessary on languages that are typologically different from it. therefore, an important direction for future research is to investigate a wider range of pre-training/meta-training language combinations, based on specific hypotheses about language relationships and relevant syntactic features. task performance may be further improved by including a larger set of syntax-related tasks, such as pos-tagging, to sample from during metatraining @xcite . all models use the same architecture: an overview of all model parameters can be seen in at pre-training time, we use the default parameters of udify @xcite . we pre-train for 60 epochs. the adam optimizer is used with a 1e-3 learning rate for the decoder and a 5e-5 learning rate for bert layers. weight decay of 0.01 is applied. we employ a gradual unfreezing scheme, freezing the bert layer weights for the first epoch. at meta-training time, we vary the learning rates as shown in we train all models on an nvidia titan rtx. pre-training takes around 3 hours, meta-training takes around 1 hour for 100 episodes per language when the amount of updates k is set to 20. for maml, this amounts to approximately 5 hours, and for the ablated maml-, it amounts to approximately 20 hours, which can be seen as another benefit of pre-training. finally, training in a non-episodic fashion (ne) also takes up less time, namely 2 to 3 hours. all final learning rates can be seen in