Genre as Weak Supervision for Cross-lingual Dependency Parsing (2021)

Abstract: Recent work has shown that monolingual masked language models learn to represent data-driven notions of language variation which can be used for domain-targeted training data selection. Dataset genre labels are already frequently available, yet remain largely unexplored in cross-lingual setups. We harness this genre metadata as a weak supervision signal for targeted data selection in zero-shot dependency parsing. Specifically, we project treebank-level genre information to the finer-grained sentence level, with the goal to amplify information implicitly stored in unsupervised contextualized representations. We demonstrate that genre is recoverable from multilingual contextual embeddings and that it provides an effective signal for training data selection in cross-lingual, zero-shot scenarios. For 12 low-resource language treebanks, six of which are test-only, our genre-specific methods significantly outperform competitive baselines as well as recent embedding-based methods for data selection. Moreover, genre-based data selection provides new state-of-the-art results for three of these target languages.

## introduction multilingual masked language models (mlms) trained on immense quantities of heterogeneous texts @xcite @xcite have recently made applications such as highly cross-lingual dependency parsing a reality @xcite . adjacently, it has also been recognized that they capture characteristics relevant for training data selection @xcite and can be efficiently fine-tuned for higher task-specific performance @xcite @xcite . these considerations are especially important in computationally restricted environments and when data from the target distribution are unavailable. ## related work despite advances in zero-shot performance @xcite and increasingly cross-lingual parsers @xcite , fine-tuning has remained a crucial step for achieving state-of-the-art performance. meechan-maddon and @xcite demonstrate that this holds true for low-resource languages in particular, with 200 training instances in the target or related languages producing better results on dependency parsing than a model trained on all available data. @xcite further show that as few as 10 samples in the target language can double parsing performance. @xcite propose udapters, which integrate language and task-specific adaptation modules into the parser to improve cross-lingual, zero-shot performance. considering factors complementary to language is equally important: mlms can for instance be improved for specific domains such as twitter or medical texts by fine-tuning on the same or related sources (dai et al., 2020; gururangan et al., 1 code at https://personads.me/x/emnlp-2021-code . 2020). for dependency parsing, the use of data from matching genres has been explored by @xcite , who find improvements for english and dutch. this is further confirmed for german by @xcite . automatically inferred topics @xcite as well as more abstract selection criteria such as overlapping part-of-speech sequences @xcite have also proven effective at selecting syntactically similar training instances. @xcite further demonstrate that when word embeddings of mutually unintelligible languages align with respect to pos, cross-lingual transfer remains especially effective. with respect to data-driven domain representations, @xcite shows that treebank embeddings can be used to successfully transfer knowledge from in-domain cross-lingual source treebanks when used in conjunction with in-language, out-of-domain data. in this work, we will rely solely on treebank genre labels as weak supervision and forgo the use of in-language training data as well as instance-level annotations thereof (e.g. pos tags). recently, contextualized embeddings have been shown to contain useful information for training data selection. @xcite find that clusters formed by embeddings from untuned, monolingual language models correspond well to the genres of their five-domain corpus. training an english-to-german machine translation model on only the closest embedded sentences to their target 2k-sentence development set outperformed a model trained on the entire dataset. although all aforementioned methods assume some degree of in-language training data, our methods will not have access to any annotated target data and will be trained exclusively on out-of-language instances. building on information stored in pretrained contextual embeddings, we extend genrebased data selection into the massively multilingual, 104-language, 18-genre setting of universal dependencies @xcite . while previous work further assumed sentence-level genre labels @xcite , our methods will only have access to treebank-level metadata. an instance's genre will therefore have to be inferred using weakly supervised approaches. to the best of our knowledge, this constitutes the first application of ud's instance-level genre distribution to the selection of training data for zero-shot, cross-lingual dependency parsing. ## genre in universal dependencies universal dependencies @xcite offer annotations for a broad spectrum of languages, with 104 in version 2.7 (zeman et al., 2020). of the 1.38 million sentences from the 177 treebanks which we consider, 64 are test-set only and many in this latter third constitute the sole treebank of the language they are in. such data sparsity becomes even more critical when both the language and the domain are highly specialized and under-resourced. as more low-resource languages are added in this manner and as the vast majority of the world's languages remain without annotated data, it becomes important to consider new signals for selecting training data in zero-shot scenarios. if no data in the target language are available, we hypothesize that characteristics of most genres are stable enough across languages to offer a useful guiding criterion for data selection in cross-lingual dependency parsing. for 26 of the 177 treebanks, their authors have provided sentence-level genre labels. however, these annotations cover only 6% of ud sentences and are typically incompatible across treebanks (with few exceptions such as pud). at the treebank level, ud fortunately provides 18 approximated genre labels: academic, bible, blog, email, fiction, government, grammar-examples, learner-essays, legal, medical, news, nonfiction, poetry, reviews, social, spoken, web, wiki. genres such as wiki likely have stronger internal consistency due to cross-lingual creation guidelines. others such as fiction or web may have higher variance. while these ud-provided labels are far from perfectly defined @xcite , they nonethe-less allow us to operationalize our hypothesis: if genre is globally consistent, it must have a positive effect on cross-lingual transfer performance. from figure 2 it is evident that these genres are heavily imbalanced. the minimum number of sentences in a genre is inferred from the sum over the number of instances in treebanks containing only that genre. the upper bound is the sum of all treebanks containing the genre among others. as indicated by these distributional bounds, news articles may constitute up to 70% of the whole ud dataset. even assuming uniform genre distributions within each treebank (center marker), over half of all sentences in ud would fall into either the news or the non-fiction category. genres with highly specific lexical and/or structural features such as spoken, social or medical are much more underrepresented. furthermore, they are often only a small part of larger genre mixtures (117 treebanks include multiple genre-labels). these mixtures, with up to 10 genres in one treebank, may contain related genres (e.g. news, nonfiction, web), but also unrelated ones (e.g. medical, poetry, social, web) depending on what data was available to authors during annotation. out-of-the-box, treebank-level genre labels appear to be highly noisy (see also @xcite . additionally, individual treebanks are labeled with multiple genres while lacking such labels at the sentence level. we hypothesize that it is therefore necessary to predict instance-level genre distributions before targeted data selection can be effective. ## targeted data selection in order to measure the effect of genre on the targeted selection of training data, we depart from previous treebank-level selection (section 2) and introduce three new types of instance-level selection strategies in the following section. they are evaluated on the task of zero-shot dependency parsing in sections 5 and 6. all of them build on contextualized embeddings learned by the mbert @xcite masked language model (mlm). while mlms still lack the full breadth of the languages covered in ud (mbert covers 56 of the 104 languages), they have proven robust in zeroshot scenarios @xcite and have also been found to contain a certain amount of genre information -at least monolingually @xcite ; section 2). we evaluate whether ud's definition of genre is also recoverable from these data-driven representations and whether these categories hold crosslingually. ## conclusions in absence of in-language training data, we have explored ud-specified genre as an alternative signal for data selection. while prior work had indicated the presence of genre information in monolingual contextualized embeddings @xcite , an analogous strategy using mbert embeddings proved insufficient in the cross-lingual parsing setting (sen), performing close to the random baseline (rand). relying on manual, treebanklevel genre labels (meta) proved even less performant, producing the lowest scores despite corresponding to a practitioner's typical first choice of selecting the largest number of training instances. in order to enable finer-grained, instance-level data selection, we proposed three methods for combining latent genre information in the unsupervised contextualized representations with the treebank metadata: weakly supervised boot, sentence embedding-based gmm and n-gram-based lda. despite their different approaches to treebank segmentation, each method significantly outperformed the purely embedding-based sent as well as the metadata (meta) and random baselines (rand). their similar performance patterns and selected data distributions further indicate that each method is identifying a shared, data-driven notion of genre. for future work, it will be important to extend our proposed approaches beyond single-genre targets towards genre-mixtures and more treebanks overall. as the data selected by these methods is further limited by the number of treebanks in each respective genre, combining a larger set of selection signals will be equally crucial. dependency parsers every parsing experiment in the main paper uses a biaffine attention parser @xcite implemented in the machamp v0.2 framework (van der goot et al., 2021) using default hyperparameters. the sentence encoder is initialized with standard mbert weights. the training duration is foremost dependent on input data quantity. for the largest corpus (meta for ta-ttb with 524k instances) this corresponds to 55 hours. our proposed methods create smaller, targeted training corpora (around 80k instances on average) such that a better performing parser can be trained in approximately 90 minutes on the same hardware. random initializations each experiment is run thrice using the seeds 41, 42 and 43. this relates to the random subsampling of data as well as to model initialization (both parsers and selection).