Parsing with Multilingual BERT , a Small Corpus, and a Small Treebank (2020)

Abstract: Pretrained multilingual contextual representations have shown great success, but due to the limits of their pretraining data, their benefits do not apply equally to all language varieties. This presents a challenge for language varieties unfamiliar to these models, whose labeled and unlabeled data is too limited to train a monolingual model effectively. We propose the use of additional language-specific pretraining and vocabulary augmentation to adapt multilingual models to low-resource settings. Using dependency parsing of four diverse low-resource language varieties as a case study, we show that these methods significantly improve performance over baselines, especially in the lowest-resource cases, and demonstrate the importance of the relationship between such models’ pretraining data and target language varieties.

## introduction contextual word representations (cwrs) from pretrained language models have improved many nlp systems. such language models include bert @xcite and elmo @xcite , which are conventionally "pretrained" on large unlabeled datasets before their internal representations are "finetuned" during supervised training on downstream tasks like parsing. however, many language varieties foot_0 lack large annotated and even unannotated datasets, raising questions about the broad applicability of such data-hungry methods. one exciting way to compensate for the lack of unlabeled data in low-resource language varieties is to finetune a large, multilingual language model that has been pretrained on the union of many languages' data @xcite . this enables the model to transfer some of what it learns from high-resource languages to low-resource ones, demonstrating benefits over monolingual methods in some cases @xcite , though not always @xcite . specifically, multilingual models face the transfer-dilution tradeoff @xcite : increasing the number of languages during pretraining improves positive crosslingual transfer but decreases the model capacity allocated to each language. furthermore, such models are only pretrained on a finite amount of data and may lack exposure to specialized domains of certain languages or even entire low-resource language varieties. the result is a challenge for these language varieties, which must rely on positive transfer from a sufficient number of similar high-resource languages. indeed, @xcite find that multilingual models often underperform monolingual baselines for such languages and question their off-theshelf viability. we take inspiration from previous work on domain adaptation, where general-purpose monolingual models have been effectively adapted to specialized domains through additional pretraining on domain-specific corpora @xcite . we hypothesize that we can improve the performance of multilingual models on low-resource language varieties analogously, through additional pretraining on language-specific corpora. however, additional pretraining on more data in the target language does not ensure its full representation in the model's vocabulary, which is constructed to maximally represent the model's original pretraining data @xcite . @xcite find that target languages' representation in the vocabulary affects these models' transferability, suggesting that language varieties on the fringes of the vocabulary may not be sufficiently well-modeled. can we incorporate vocabulary from the target language into multilingual models' existing alignment? we introduce the use of additional languagespecific pretraining for multilingual cwrs in a low-resource setting, before use in a downstream task; to better model language-specific tokens, we also augment the existing vocabulary with frequent tokens from the low-resource language ( §2). our experiments consider dependency parsing in four typologically diverse low-resource language varieties with different degrees of relatedness to a multilingual model's pretraining data ( §3). our results show that these methods consistently improve performance on each target variety, especially in the lowest-resource cases ( §4). in doing so, we demonstrate the importance of accounting for the relationship between a multilingual model's pretraining data and the target language variety. because the pretraining-finetuning paradigm is now ubiquitous, many experimental findings for one task can now inform work on other tasks. thus, our findings on dependency parsing-whose annotated datasets cover many more low-resource language varieties than those of other nlp tasks-are expected to interest researchers and practitioners facing low-resource situations for other tasks. to this end, we make our code, data, and hyperparameters publicly available. foot_2 ## overview we are chiefly concerned with the adaptation of pretrained multilingual models to a target language by optimally using available data. as a case study, we use the multilingual cased bert model (mbert) of @xcite , a transformerbased @xcite language model which has produced strong cwrs for many languages @xcite . mbert is pretrained on the 104 languages with the most wikipedia data and encodes input tokens using a fixed wordpiece vocabulary @xcite learned from this data. low-resource languages are slightly oversampled in its pretraining data, but high resource languages are still more prevalent, resulting in a language imbalance. 3 we observe that two types of target language varieties may be disadvantaged by this training scheme: the lowest-resource languages in mbert's pretraining data (which we call type 1); and unseen low-resource languages (type 2). although type 1 languages are oversampled during training, they are still overshadowed by high-resource languages. type 2 languages must rely purely on crosslingual vocabulary overlap. in both cases, the wordpieces that encode the input tokens in these languages may not fully capture the senses in which they are used, or they may be completely unseen. 4 however, other low-resource varieties with more representation in mbert's pretraining data (type 0) may not be as disadvantaged. optimally using mbert in low-resource settings thus requires accounting for limitations with respect to a target language variety. ## experiments we consider two variants of each mbert method: one in which the pretrained cwrs are frozen; and one where they are further finetuned during parser training (ft). following prior work involving these two variants @xcite , ft variants perform biaffine attention directly on the outputs of mbert instead of first passing them through a bil-stm, as in @xcite . we perform additional pretraining for up to 20 epochs, selecting our final models based on average validation las downstream. full training details are given in the appendix. we report average scores and standard errors based on five random initializations. code and data are publicly available (see footnote 2). ## results and discussion tab. 2 shows the performance of each of the method variants on the four universal dependencies datasets, with standard deviations from five different initializations. our experiments demonstrate that additional language-specific pretraining results in more effective representations. lapt 7 our partition of the data is available at https:// github.com/ethch18/parsing-mbert . lapt with our vocabulary augmentation methods yield small but significant improvements over lapt alone, especially for ft configurations and type 1/2 languages. this demonstrates that accurate vocabulary modeling is important for improving representations in the target language, and that va is an effective methods for doing so while maintaining overall alignment. however, tva rarely outperforms va significantly, suggesting that accelerated learning of the new embeddings does not benefit the model overall. the relative error reductions between unadapted mbert and each of our methods correlates with each language variety's relationship to mbert pretraining data. maltese (type 2) improves by up to 39% and irish (type 1) by up to 15%, compared to 11% for singlish and 5% for vietnamese (both type 0). while this trend is by no means comprehensive, it suggests that effective use of mbert requires considering the target language variety. our results thus support our hypotheses and give insight to the limitations of mbert. wordpieces appear in different contexts in different languages, and mbert initially lacks enough exposure to wordpiece usage in type 1/2 target languages to outperform baselines. however, increased exposure through additional language-specific pretrain-ing can ameliorate this issue. likewise, despite mbert's attempt to balance its pretraining data, the existing vocabulary still favors languages that have been seen more. augmenting the vocabulary can produce additional improvement for languages with greater proportions of unseen wordpieces. overall, our findings are promising for lowresource language varieties, demonstrating that large improvements in performance are possible with the help of a little unlabeled data, and that the performance discrepancy of multilingual models for low-resource languages (wu and dredze, 2020) can be overcome. ## further related work our work builds on prior empirical studies on multilingual models, which probe the behavior and components of existing models to explain why they are effective. @xcite , and @xcite note the importance of both vocabulary overlap and the relationship between languages in determining the effectiveness of multilingual models, but they primarily consider high-resource languages. on the other hand, @xcite find vocabulary overlap to be less significant of a factor, instead attributing such models' successes to typological similarity and parameter sharing. @xcite emphasize the importance of sufficiently representing the target language in the vocabulary. unlike these studies, we primarily consider how to improve the performance of multilingual models for a given target language variety. though our experiments do not directly probe the impact of vocabulary overlap, we contribute further evaluation of the importance of improved modeling of the target variety. recent work has also proposed additional pretraining for general-purpose language models, especially with respect to domain @xcite @xcite @xcite @xcite . @xcite perform additional training on parallel data to adapt bilingual translation models to unseen target languages, while @xcite improve a polyglot task-specific model by finetuning on labeled monolingual data in the target variety. to the best of our knowledge, our work is the first to demonstrate the effectiveness of additional pretraining for massively multilingual language models toward a target low-resource language variety, using only unlabeled data in the target variety. ## conclusion we explore additional language-specific pretraining and vocabulary augmentation for multilingual contextual word representations in low-resource settings and find them to be effective for dependency parsing, especially in the lowest-resource cases. our results demonstrate the significance of the relationship between a multilingual model's pretraining data and a target language. we expect that our findings can benefit practitioners in lowresource settings, and our data, code, and models are publicly available to accelerate further study.