Zero-Shot Dependency Parsing with Worst-Case Aware Automated Curriculum Learning (2022)

Abstract: Large multilingual pretrained language models such as mBERT and XLM-RoBERTa have been found to be surprisingly effective for cross-lingual transfer of syntactic parsing models Wu and Dredze (2019), but only between related languages. However, source and training languages are rarely related, when parsing truly low-resource languages. To close this gap, we adopt a method from multi-task learning, which relies on automated curriculum learning, to dynamically optimize for parsing performance on outlier languages. We show that this approach is significantly better than uniform and size-proportional sampling in the zero-shot setting.

## introduction the field of multilingual nlp is booming @xcite . this is due in no small part to large multilingual pretrained language models (plms) such as mbert @xcite and xlm-roberta @xcite , which have been found to have surprising cross-lingual transfer capabilities in spite of receiving no cross-lingual supervision. 1 @xcite , for example, found mbert to perform well in a zero-shot setting when finetuned for five different nlp tasks in different languages. there is, however, a sharp divide between languages that benefit from this transfer and languages that do not, and there is ample evidence that transfer works best between typologically similar languages (pires et al., 2019; lauscher et al., 2020, 1 in the early days, cross-lingual transfer for dependency parsing relied on projection across word alignments @xcite or delexicalized transfer of abstract syntactic features @xcite @xcite . delexicalized transfer was later 're-lexicalized' by word clusters @xcite and word embeddings @xcite , but with the introduction of multilingual contextualized language models, transfer models no longer rely on abstract syntactic features, removing an important bottleneck for transfer approaches to scale to truly low-resource languages. large multilingual plms are typically fine-tuned using training data from a sample of languages that is supposed to be representative of the languages that the models are later applied to. however, this is difficult to achieve in practice, as multilingual datasets are not well balanced for typological diversity and contain a skewed distribution of typological features @xcite . this problem can be mitigated by using methods that sample from skewed distributions in a way that is robust to outliers. zhang et al. ( 2020 ) recently developed such a method. it uses curriculum learning with a worst-case-aware loss for multi-task learning. they trained their model on a subset of the glue benchmark @xcite and tested on outlier tasks. this led to improved zero-shot performance on these outlier tasks. this method can be applied to multilingual nlp where different languages are considered different tasks. this is what we do in this work, for the case of multilingual dependency parsing. multilingual dependency parsing is an ideal test case for this method, as the universal dependency treebanks @xcite are currently the manually annotated dataset that covers the most typological diversity @xcite . our research question can be formulated as such: can worst-case aware automated curriculum learning improve zero-shot cross-lingual dependency parsing? 2 2 worst-case-aware curriculum learning in multi-task learning, the total loss is generally the average of losses of different tasks: the architecture consists of a sampler, a buffer, a trainer and a multilingual dependency parsing model. the two main components are the sampler, which adopts a curriculum sampling strategy to dynamically sample data batches, and the trainer which uses worst-case-aware strategy to train the model. the framework repeats the following steps: ## experiments we base our experimental design on @xcite , a recent paper doing zero-shot dependency parsing with good performance on a large number of languages. they fine-tune mbert for dependency parsing using training data from a sample of 13 typologically diverse languages from universal dependencies (ud; @xcite , listed in we implement all models using machamp (van der goot et al., 2021), a library for multi-task learning based on allennlp @xcite . the library uses transformers from huggingface @xcite . our code is publicly available. 4 our main results are in our results using mbert are more than 1 las point above the corresponding baselines. our best model is significantly better than the best baseline with @xmath0.01 according to a bootstrap test across test treebanks. our best model with mbert comes close to udapter (36.5 las on the same test sets) while being a lot simpler and not using external resources such as typological features, which are not always available for truly low-resource languages. the results with xlm-r are much higher in general foot_3 but the trends are similar: all our models outperform all of our baselines albeit with smaller differences. there is only a 0.4 las difference between our best model and the best baseline, but it is still significant with @xmath1.05 according to a bootstrap test across test treebanks. this highlights the robustness of the xlm-r model itself. our results with xlm-r outperform udapter by close to 7 las points. ## varying the homogeneity of training samples we investigate the interaction between the effectiveness of worst-case learning and the representativeness of the sample of training languages. it is notoriously difficult to construct a sample of treebanks that is representative of the languages in ud @xcite @xcite . we can, however, easily construct samples that are not representative, for example, by taking a sample of related languages. we expect worst-case aware learning to lead to larger improvements in cases where some language types are underrepresented in the sample. we can construct an extreme case of underrepresentation by selecting a sample of training languages that has one or more clear outliers. for example we can construct a sample of related languages, add a single unrelated language in the mix, and then evaluate on other unrelated languages. we also expect that with a typologically diverse set of training languages, worst-case aware learning should lead to larger relative improvements than with a homogeneous sample, but perhaps slightly smaller improvements than with a very skewed sample. we test these hypotheses by constructing seven samples of training languages in addition to the one used so far (13lang). we construct three different homogeneous samples using treebanks from three different genera: germanic, romance and slavic. we construct four skewed samples using the sample of romance languages and a language from a different language family, an outlier language: basque (eu), arabic (ar), turkish (tr) and chinese (zh). since we keep the sample of test sets constant, we do not include training data from languages that are in the test sets. the details of which treebanks are used for each of these samples results are in looking at results of the different models on individual test languages (see figure 1 in appendix c), we find no clear pattern of the settings in which this method works best. we do note that the method always hurts belarusian, which is perhaps unsurprising given that it is the test treebank for which the baseline is highest. worst-case aware learning hurts belarusian the least when using the slavic sample, indicating that, when using the other samples, the languages related to belarusian are likely downsampled in favour of languages unrelated to it. worst-case learning consistently helps breton and swiss german, indicating that the method might work best for languages that are underrepresented within their language family but not necessarily outside of it. for swiss german, worst-case learn-ing helps least when using the germanic sample where it is less of an outlier. ## conclusion in this work, we have adopted a method from multitask learning which relies on automated curriculum learning to the case of multilingual dependency parsing. this method allows to dynamically optimize for parsing performance on outlier languages. we found this method to improve dependency parsing on a sample of 30 test languages in the zeroshot setting, compared to sampling data uniformly across treebanks from different languages, or proportionally to the size of the treebanks. we investigated the impact of varying the homogeneity of the sample of training treebanks on the usefulness of the method and found conflicting evidence with different samples. this leaves open questions about the relationship between the languages used for training and the ones used for testing. thomas wolf, lysandre debut, victor sanh, julien chaumond, clement delangue, anthony moi, pierric cistac, tim rault, remi louf, morgan funtowicz, joe davison, sam shleifer, patrick von platen, clara ma, yacine jernite, julien plu, canwen xu, teven le scao, sylvain gugger, mariama drame, quentin lhoest, and alexander rush. 2020. transformers: state-of-the-art natural language processing. in proceedings of the 2020 conference on empirical methods in natural language processing: system demonstrations, pages 38-45, online. association for computational linguistics. shijie wu and mark dredze. 2019. beto, bentz, becas: the surprising cross-lingual effectiveness of bert. in proceedings of the 2019 conference on empirical methods in natural language processing and the 9th international joint conference on natural language processing (emnlp-ijcnlp), pages 833-844, hong kong, china. association for computational linguistics. weijia xu, batool haider, jason krone, and saab mansour. 2021. soft layer selection with meta-learning for zero-shot cross-lingual transfer. in proceedings of the 1st workshop on meta learning and its applications to natural language processing, pages 11-18, online. association for computational linguistics. daniel zeman and philip resnik. 2008. crosslanguage parser adaptation between related languages. in proceedings of the ijcnlp-08 workshop on nlp for less privileged languages. sheng zhang, xin zhang, weiming zhang, and anders søgaard. 2020. worst-case-aware curriculum learning for zero and few shot transfer. arxiv preprint arxiv:2009.11138.