M icro BERT : Effective Training of Low-resource Monolingual BERT s through Parameter Reduction and Multitask Learning (2022)

Abstract: BERT-style contextualized word embedding models are critical for good performance in most NLP tasks, but they are data-hungry and therefore difficult to train for low-resource languages. In this work, we investigate whether a combination of greatly reduced model size and two linguistically rich auxiliary pretraining tasks (part-of-speech tagging and dependency parsing) can help produce better BERTs in a low-resource setting. Results from 7 diverse languages indicate that our model, MicroBERT, is able to produce marked improvements in downstream task evaluations, including gains up to 18% for parser LAS and 11% for NER F1 compared to an mBERT baseline, and we achieve these results with less than 1% of the parameter count of a multilingual BERT base–sized model. We conclude that training very small BERTs and leveraging any available labeled data for multitask learning during pretraining can produce models which outperform both their multilingual counterparts and traditional fixed embeddings for low-resource languages.

## introduction pretrained word embeddings are an essential ingredient for high performance on most nlp tasks. transformer language models (tlms) foot_0 such as bert/mbert @xcite roberta/xlm-r, @xcite , and electra @xcite provide state-of-the-art performance, but they expect at least tens of millions of tokens in training data. high-resource languages like english, arabic, and mandarin are able to meet this requirement, but most of the world's languages cannot. two major lines of work have arisen in order to address this gap: the first attempts to use multilingual transfer to pool different languages' data together to meet tlms' data demands, and the second attempts to lower tlms' data demands by changing their architectures and training regimens. in this study, we take up work in the latter direction, asking specifically whether (1) vast reduction of model size and (2) incorporation of explicitly supervised, rather than self-supervised, tasks into model pretraining can produce better monolingual tlms. the former method is motivated by the intuition that normal-sized tlms are so large as to be severely overparameterized for low-resource settings, and the latter method is motivated by an intuition that in the absence of large volumes of unlabeled text, signal from a supervised task with linguistic annotations is less likely be redundant to the model. we find evidence that indicates both methods are helpful: our microbert models produce monolingual embeddings that can outperform comparable multilingual approaches. we summarize our contributions as follows: ## previous work at least since the development of pretrained static word embeddings @xcite @xcite , pretrained word representations have been indispensable resources for nlp models, providing dense numerical representations of tokens' linguistic properties. pretrained contextualized embeddings @xcite @xcite based on the transformer architecture @xcite have since overtaken them in popularity. throughout this period, highresource languages have received the majority of attention, and although interest in low-resource settings has increased in the past few years, there remains a large gap (in terms of linguistic resources, pretrained models, etc.) between low-and highresource languages @xcite . ## motivation as we have seen, monolingual berts trained with standard methods tend to perform poorly when less than 20-40m tokens are available during training, and there is evidence that they do not learn to fully generalize some linguistic patterns without a large (≈1b tokens, @xcite amount of training data. however, most popular methods for pretraining berts are self-supervised, using only unlabeled text. this has turned out well for high-resource languages, where unlabeled text is available in far greater quantities than labeled text, to the point where incorporating labeled text into pretraining does not always provide large gains. however, even in very low-resource settings, it is common for sources of linguistic signal beyond unlabeled text to be available, such as treebanks, interlinearized text, and dictionaries. it is natural to ask whether using them as data for auxiliary supervised tasks during model pretraining could help monolingual models overcome a lack of unlabeled data, and perhaps even interact synergistically with the main pretraining task, such as mlm. it is known, for example, that berts learn to represent words' parts of speech @xcite , and it seems possible that providing direct supervision for predicting parts of speech may help a model acquire good pos representations with less data. this leads us to our first hypothesis h1, that monolingual models should benefit from multitask pretraining with auxiliary tasks incorporating labeled data. previous results also lead us to our second hypothesis h2, that in low-resource settings, monolingual berts are typically severely overparameterized. most berts are overparameterized in the sense that they can have modules removed, disabled, or compressed while showing minimal regressions (or sometimes even improvements) @xcite , but in h2 we mean further that there are so many parameters that the model cannot be effectively learned given the amount of data. as noted in §2, there appears to be a gap in the literature on whether pretraining a vastly scaled down bert model could help monolingual berts perform better in low-resource settings, and we take up the question in this work. ## approach we propose an architecture and training regime for monolingual berts which we call microbert. we keep the basic architecture of bert, but we reduce encoder layer count to 3, hidden representation size to 100, and number of attention heads to 5. (compare this to bert base's 12, 768, and 12, respectively.) excluding prediction heads, this reduces parameter count from 108m foot_1 to 1.29m, or just 1.19% of a normal bert model's size. after the encoder stack, one dedicated head is used for each task, where each head is provided with the last encoder layer's hidden states. to prepare the sequences of batches @xmath0, . . . , b |b| for a given epoch, construct each batch b i using only instances from exactly one dataset d t , and sample batches so that each dataset d t is represented at least ⌊λ t |b|⌋ times in b. each batch is sent not only to its dataset's corresponding prediction head, but also to any other prediction heads which are compatible with it. for example, a batch containing dependency syntax labels would be sent to the parsing prediction head, and it would also be sent to the mlm head, since the mlm head only requires unlabeled text. 3 if a dataset is exhausted in the course of this procedure, new instances are sampled anew from the beginning of the dataset. this is a simple means for addressing the fact that some datasets will be much larger than others, which without intervention could have led to one task's parameter updates drowning out others. we consider three tasks in this work. the first is mlm implemented as whole-word, dynamic masking, as in roberta @xcite . the second is pos tagging, for which our prediction head is a simple linear projection. the third is dependency parsing, for which we use a modified form of the biaffine dependency parser of @xcite which has had the encoder lstm stack removed. cross-entropy loss is used for all tasks and summed together: each head produces an associated loss ℓ i , which is summed into a single loss ℓ which is used to begin backpropagation. we note that it would be straightforward to add other tasks, though we choose pos tagging and parsing for this work since pos tagged and dependency parsed datasets are relatively common for low-resource languages. this multitask setup is not novel-in fact, @xcite use the the same three tasks for a similar purpose, though instead of pretraining a bert from scratch, they use the multitask setup to perform adaptive finetuning on a pretrained multilingual model, and find a negative result. ## experimental methods to evaluate our approach, we train microbert models on several languages and compare them to a variety of baselines. all our experiments are implemented using allennlp @xcite , transformers @xcite , and pytorch @xcite . all code and models are available at https://github.com/lgessler/ microbert . ## results results for the parser evaluation are given in ## discussion main findings we take our most important result to be our demonstration that it is possible to train a monolingual bert from scratch that can compete with and even outperform multilingual models by up to 18% las and 11% ner f1 using as little as 500,000 tokens and a ud treebank of 44,000 tokens and less than 1% of the parameters. ## conclusion we have shown that it is possible to train monolingual tlms that are competitive with multilingual models using as little as 500k tokens and a 40k token treebank with greatly reduced model size and multitask learning on pos tagging and dependency syntax parsing. while multilingual models did have some advantages over our approach, we observe that our microbert approach has unique strengths for work on low-resource tlms, including its lack of reliance on successful cross-lingual transfer and radically reduced computational demands for pretraining and downstream use. we take this result to call into question whether multilingual representation learning can scale down effectively to truly "low-resource" languages that have less than a few million tokens in training data. sometimes languages like these can be well served by transfer from related languages, even if all languages are low-resource @xcite , but not all languages may be so lucky: language isolates by definition lack related languages, and small language families are likely less able to benefit from transfer, since transfer tends to be enabled by phylogenetic @xcite or areal @xcite ) relatedness between languages. while multilingual methods hold much promise, it is important to examine other approaches to low-resource representation learning which, if not strictly better, may at least be complementary. fore take the additional step of rendering the dump into html using https://github.com/ lgessler/wiki-thresher , which can then be used to obtain useful information about guaranteed sentence splits, e.g. between html elements @xmath1. we perform rule-based sentence splitting and tokenization on this html to obtain our final tokenized texts. for all 7 languages, we reserve around 10% of documents for validation and use the rest for training. a test split is unnecessary because our models are not being evaluated on unlabeled data. ud treebanks a summary of the treebanks we use and their versions is given in wikiann datasets new train/dev/test splits were created in an 8:1:1 ratio for the wikiann dataset, which only divides sentences by language. it was not possible to split at the document level because no document metadata is available in the wikiann dataset. tags are converted from the native iob1 scheme into the bioul scheme. some manual edits, logged in our version control history, were made to sentence boundaries in order to keep wordpiece sequence lengths below 512.