W iki BERT Models: Deep Transfer Learning for Many Languages (2021)

Abstract: Deep neural language models such as BERT have enabled substantial recent advances in many natural language processing tasks. However, due to the effort and computational cost involved in their pre-training, such models are typically introduced only for a small number of high-resource languages such as English. While multilingual models covering large numbers of languages are available, recent work suggests monolingual training can produce better models, and our understanding of the tradeoffs between mono- and multilingual training is incomplete. In this paper, we introduce a simple, fully automated pipeline for creating language-specific BERT models from Wikipedia data and introduce 42 new such models, most for languages up to now lacking dedicated deep neural language models. We assess the merits of these models using cloze tests and the state-of-the-art UDify parser on Universal Dependencies data, contrasting performance with results using the multilingual BERT (mBERT) model. We find that the newly introduced WikiBERT models outperform mBERT in cloze tests for nearly all languages, and that UDify using WikiBERT models outperforms the parser using mBERT on average, with the language-specific models showing substantially improved performance for some languages, yet limited improvement or a decrease in performance for others. All of the methods and models introduced in this work are available under open licenses from https://github.com/turkunlp/wikibert .

## introduction transfer learning using language models pretrained on large unannotated corpora has allowed for substantial recent advances at a broad range of natural language processing (nlp) tasks. by contrast to earlier distributional semantics approaches such as random indexing @xcite and context-independent neural approaches such as word2vec @xcite and glove @xcite , models such as ulmfit @xcite , @xcite , gpt @xcite and bert @xcite create contextualized representations of meaning, capable of providing both contextualized word embeddings as well as embed-dings for text segments longer than words. recent pre-trained neural language models have been rapidly advancing the state of the art in a range of natural language understanding and nlp tasks @xcite @xcite . the transformer architecture @xcite and the bert language model of @xcite have been particularly influential, with transformer-based models in general and bert in particular fuelling a broad range of advances and serving as the basis of many recent studies of neural language models (e.g. @xcite @xcite . as is the case for most studies on new deep neural language models, the original study introducing bert addressed only english. the authors later released a chinese model as well as a multilingual model, mbert, trained on text from 104 languages, but opted not to introduce models specifically targeting other languages. while mbert is a powerful multilingual model with remarkable cross-lingual capabilities @xcite , it remains a compromise in that the 104 languages share the model capacity dedicated to one language in monolingual models, and it consequently suffers from degradation of performance in language-specific tasks @xcite . here, we take steps towards closing various parts of the gap between languages with dedicated deep neural models, ones that share capacity with others in a massively multilingual model, and ones that lack any representation at all. we introduce a fully automated pipeline for creating languagespecific bert models from wikipedia data and apply this pipeline to create 42 new such models. ## related work considerable recent effort by various groups has focused on introducing dedicated bert models covering single languages or a small num-ber of (often closely related) languages. dedicated monolingual models include e.g. bertje foot_0 (de vries et al., 2019) for dutch, camembert foot_1 @xcite for french, finbert foot_2 @xcite for finnish, rubert foot_3 (kuratov and arkhipov, 2019) for russian, and romanian bert @xcite ; more focused multilingual models include e.g. the bilingual finnish-english model of @xcite and the trilingual finnish-estonian-english and croatian-slovenian-english models of @xcite . many of these studies have demonstrated the newly introduced models to allow for substantial improvements over mbert in various languagespecific downstream task evaluations, thus supporting the continued value of creating monolingual and focused multilingual models. however, these efforts still cover only a fairly limited number of languages, and do not offer a straightforward way to substantially extend that coverage. the studies further differ considerably in aspects such as data collection, text cleaning and preprocessing, pre-training parameter setting and other details of the pre-training process, making it difficult to meaningfully compare the models to address questions such as which languages benefit most from mono/multilingual pre-training? we are not aware of previous efforts to automate the creation of large numbers of monolingual deep neural language models from comparable, publicly available sources nor efforts to create broadcoverage collections of such models. in a line of study in some senses orthogonal to our work, a number of massively multilingual models improving on mbert in terms of model architecture, training dataset, objectives, and process or other aspects have been introduced (e.g. @xcite . while it is certainly an interesting question to ask what the tradeoffs between monolingual and massively multilingual pre-training are for models other than bert, it is not feasible for us to replicate the training processes for other models, and we have here chosen to focus on bert-based models and wikipedia due to their prominence and status as benchmarks. ## data we next introduce the two primary datasets used in this study: wikipedia, used as the source of unannotated texts for model pre-training, and universal dependencies annotated corpora, used to train preprocessing methods as well as in model evaluation. ## methods we next briefly introduce the primary steps of the preprocessing pipeline for creating pre-training examples from wikipedia source as well as the tools used for text processing, model pre-training, and evaluation. we refer to our published pipeline and its documentation for full processing details. ## results we next present the results of the intrinsic cloze test evaluation and the extrinsic evaluation with syntactic analysis as a downstream task. ## discussion and conclusions in this paper, we have introduced a simple, fully automatic pipeline for creating monolingual bert models from wikipedia data, applied the pipeline to introduce 42 new language-specific models, most covering languages that previously lacked a dedicated deep neural language model. we evaluated the wikibert models intrinsically using cloze evaluation, finding that they outperform the multilingual mbert model for all but one language. an extrinsic evaluation using a dependency parsing task with universal dependencies data and the udify neural parser found a more nuanced picture of the comparative merits of the monolingual and multilingual models: while we found that a wikibert model will provide better performance than mbert on average and in multiple cases provides a more than 10% relative decrease in las error compared to the multilingual model, the wikibert models showed lower performance than mbert for multiple languages. viewing relative change in performance against pre-training data size, we found indications that monolingual models may most benefit languages that have no closely related highresource languages and for which comparatively large pre-training corpora can be assembled. the availability of the wikibert collection of models opens up a broad range of potential avenues for research into the strengths, weaknesses and challenges in both mono-and multilingual language modeling that we hope to pursue in future work. we also hope to encourage both monolingual applications as well as exploration of these questions by others by making the models freely available under open licenses from https:// github.com/turkunlp/wikibert .