Scaling Laws for BERT in Low-Resource Settings (2023)

Abstract: Large language models are very resource intensive, both financially and environmentally, and require an amount of training data which is simply unobtainable for the majority of NLP practitioners. Previous work has researched the scaling laws of such models, but optimal ratios of model parameters, dataset size, and computation costs focused on the large scale. In contrast, we analyze the effect those variables have on the performance of language models in constrained settings, by building three lightweight BERT models (16M/51M/124M parameters) trained over a set of small corpora (5M/25M/125M words).We experiment on four languages of different linguistic characteristics (Basque, Spanish, Swahili and Finnish), and evaluate the models on MLM and several NLU tasks. We conclude that the power laws for parameters, data and compute for low-resource settings differ from the optimal scaling laws previously inferred, and data requirements should be higher. Our insights are consistent across all the languages we study, as well as across the MLM and downstream tasks. Furthermore, we experimentally establish when the cost of using a Transformer-based approach is worth taking, instead of favouring other computationally lighter solutions.

## introduction pre-trained neural language models based on the transformer architecture have shown impressive results on many nlp tasks to the point that their use has become standard practice. the capabilities of these models improve as the complexity (in terms of parameters) of their architecture @xcite and the size of the corpora on which the pre-training is performed increase @xcite . for this reason, there is now a tendency to build ever-larger models trained on ever-growing corpora. this trend has resulted in a never-ending increase of the computational requirements to perform model pre-training, but also for the subsequent fine-tuning and inference processes at production time. moreover, building very large models require huge training corpora, which is only available for a handful of rich-resource languages. @xcite and @xcite propose power-law formulas that relate model size, corpora size and computation power, and help find the optimal settings in advance given a fixed budget. however, their analysis is focused on autoregressive models of relatively big sizes, that require large corpora to train. in this paper, we analyze whether the conclusions drawn in these works also apply to discriminative (encoder-only) models in low-resource settings, where both the data size and budget are constrained. we analyze the performance of several combinations of model and data sizes using a simulated low-resource scenario in four linguistically diverse languages from different families (basque, spanish, swahili and finnish). our study reveals that the data size and model size power law values provided by @xcite and @xcite are not optimal in these scenarios. instead, our experiments show that data size should be relatively bigger than what those scaling laws estimated when training small models (16-124m parameters) for optimal performance. furthermore, given a fixed computational budget, it is better to train big models instead of computing more model updates in smaller models. additionally, we establish the minimally required combinations of compute, model and dataset sizes of transformer-based approaches that outperform other lighter neural baselines, taking co 2 emissions into consideration. ## related work since the emergence of the attention-based transformer @xcite architecture and the masking pre-training strategies introduced since bert @xcite , different pre-training strategies have been published. but aside from the improvements to the architecture or training procedures, the qualitative improvement in results is mainly the result of increasing the model size, alongside the amount of text corpora used to train them: chinchilla (70b parameters) @xcite , lamda (137b) @xcite , gpt-3 (175b) @xcite , gopher (280b) @xcite and palm (540b) @xcite . this fast-growing increase in model sizes and data is proven to surface new abilities in larger models, but not present in smaller models @xcite . the relationship between the size of the pretraining corpus and the performance of the language model in nlu tasks has been addressed in the literature before. the performance improves when the amount of data is increased @xcite @xcite , although, at a certain point, the increase in performance slows down when the model size is kept fixed @xcite @xcite @xcite . furthermore, it is more convenient to improve the diversity of training datasets, than to add more text from the same domains @xcite @xcite . the correlation between model size and model performance on nlu tasks has also been analyzed. the performance of the model improves when scaling the model size (and flops) @xcite @xcite . however, all these works used very large pre-training datasets. they do not analyze if the increase in performance slows down bottlenecked by the pre-training dataset size and thus, the conclusion of scaling being always beneficial cannot be extended to low-data scenarios. the works of @xcite and @xcite , whose aim is aligned with this work, empirically study the optimal ratios of the training tokens, model parameters, and computation to train dense language models and infer scaling laws. @xcite train models of a size ranging from 768 to 1.5 billion parameters with datasets ranging from 22 million to 23 billion tokens and conclude that lm performance improves smoothly as we increase the model size, dataset size, and amount of computation. they show that all three factors must be scaled up in tandem, to avoid bottleneck issues. furthermore, they note that larger models are more sample-efficient, and that convergence is inefficient, suggesting that it's better to under-train a bigger model than converge a smaller one on the same computing budget. @xcite find the optimal model size and the number of training tokens for given a fixed flops budget. for this purpose, they draw their own scaling laws, based on the losses of over 400 models, ranging from 70m to 16b parameters, and trained on 5b to 400b tokens. they state that model size and the number of training tokens should scale equally, based on three alternative approaches, while @xcite extrapolates that every time the model size is increased by 8, the data only needs to be increased by 5. thus, after concluding that the performance of most of the current large language models is bottlenecked by the undersized corpora, they train chinchilla. however, the scaling laws of @xcite are not useful for the low-resource settings we want to focus on. according to @xcite we need very small training corpora (e.g. 744k tokens for a bert base , which is clearly not enough or optimal). @xcite infers significantly bigger training corpora: e.g. 3m tokens for a bert m in i or 86m tokens for a bert base . current models for low-resource languages are trained with corpora around that range @xcite : 161m for irish @xcite , 130m for luxembourgish @xcite , 45m for galician @xcite , 16m for swahili @xcite and 4.4m for quechua @xcite . however, increasing pre-training data several orders of magnitude has been proved beneficial for base size models @xcite . finally, those optimal scaling laws have been deduced from models trained over one epoch, while in low/medium-resource settings models are often trained over several epochs @xcite @xcite . nonetheless, for certain nlu tasks (e.g. neqa and quote repetition) scaling language models is detrimental @xcite , creating inverse scaling laws. however, @xcite implies u-shaped scaling laws where even larger models might be able to solve those tasks that comprise a true and a distractor task. ## experimental setup we aim to find the optimal combination of modelsize, dataset-size and computing in low-resource environments and assess whether they follow the scaling laws established in the literature. in addition, we seek to find the minimum requirements to overcome computationally lighter neural baselines. therefore, we carry out experiments for 3 corpus sizes and 3 model sizes, in 4 languages, training a total of 36 different models. ## evaluation settings we evaluate all models intrinsically and extrinsically. for the intrinsic evaluation, we tested the models on masked language modeling; for the extrinsic evaluation, we selected four nlu downstream tasks with available datasets in all the selected languages: name entity recognition and classification (nerc), topic classification (topic), sentiment analysis (sa) and questionanswering nli (qnli). ## conclusions we present a study of the performance of language models in constrained settings, to analyze if the same scaling laws studied for large-language mod- from our experiments, we conclude that it is preferable to train big models on as much data as possible rather than using the computational power to further train smaller models. we see a clear trend where bigger models tend to quickly overfit when pre-trained for many epochs with small corpora. still, even when they overfit in the pretraining stage, bigger models consistently outperform smaller models in downstream applications which require fine-tuning. the experimental results are consistent among languages. additionally, we empirically establish when the computational cost of using a transformer-based approach is worth taking. all the pre-training corpora, models and datasets created in this work are publicly available 16 .