The Diminishing Returns of Masked Language Models to Science (2023)

Abstract: Transformer-based masked language models such as BERT, trained on general corpora, have shown impressive performance on downstream tasks. It has also been demonstrated that the downstream task performance of such models can be improved by pretraining larger models for longer on more data. In this work, we empirically evaluate the extent to which these results extend to tasks in science. We use 14 domain-specific transformer-based models (including ScholarBERT, a new 770Mparameter science-focused masked language model pretrained on up to 225B tokens) to evaluate the impact of training data, model size, pretraining and finetuning time on 12 downstream scientific tasks. Interestingly, we find that increasing model size, training data, or compute time does not always lead to significant improvements (i.e., >1% F1), if any, in scientific information extraction tasks. We offer possible explanations for this surprising result.

## introduction massive growth in the number of scientific publications places considerable cognitive burden on researchers @xcite . language models can potentially alleviate this burden by automating the scientific knowledge extraction process. bert @xcite was pretrained on a general corpus (bookscorpus and wikipedia) which differs from scientific literature in terms of the context, terminology, and writing style @xcite . other masked language models have since been pretrained on domain-specific scientific corpora @xcite @xcite with the goal of improving downstream task performance. (we use the term domain to indicate a specific scientific discipline, such as biomedical science or computer science.) other studies @xcite explored the impact of varying model size, training corpus size, and compute time on downstream task performance. however, no previous work has investigated how these parameters affect science-focused models. in this study, we train a set of scientific language models of different sizes, collectively called scholarbert, on a large, multidisciplinary scientific corpus of 225b tokens to understand the effects of model size, data size, and compute time (specifically, pretraining and finetuning epochs) on downstream task performance. we find that for information extraction tasks, an important application for scientific language models, the performance gains by training a larger model for longer with more data are not robust-they are highly taskdependent. we make the scholarbert models and a sample of the training corpus publicly available to encourage further studies. ## related work prior research has explored the effects of varying model size, dataset size, and amount of compute on language model performance. @xcite demonstrated that crossentropy training loss scales as a power law with model size, dataset size, and compute time for unidirectional decoder-only architectures. @xcite showed that language model few-shot learning abilities can be improved by using larger models. however, both studies explored only the generative pretrained transformer (gpt), an autoregressive generative model @xcite . comparing bert-base (110m parameters) and bert-large (340m parameters), @xcite showed that masked language models can also benefit from more parameters. likewise, @xcite demonstrate how bert models can benefit from training for longer periods, with bigger batches, and with more data. models such as bert and roberta were pretrained on general corpora. to boost performance on scientific downstream tasks, scibert @xcite , pubmedbert @xcite , biobert @xcite , and matbert (trewartha et al., 2022) were trained on domainspecific text with the goal of enhancing performance on tasks requiring domain knowledge. yet there is no work on how that task performance varies with pretraining parameters. ## data and methodology we outline the pretraining dataset, related models to which we compare performance, and the architecture and pretraining process used for creating the scholarbert models. ## experimental results we first perform sensitivity analysis across scholar-bert pretraining dimensions to determine the trade-off between time spent in pretraining versus finetuning. we also compare the downstream task performance of scholarbert to that achieved with other bert models. details of each evaluation task are in appendix c. ## conclusions we have reported experiments that compare and evaluate the impact of various parameters (model size, pretraining dataset size and breadth, and pretraining and finetuning lengths) on the performance of different language models pretrained on scientific literature. our results encompass 14 existing and newly-developed bert-based language models across 12 scientific downstream tasks. we find that model performance on downstream scientific information extraction tasks is not improved significantly or consistently by increasing any of the four parameters considered (model size, amount of pretraining data, pretraining time, finetuning time). we attribute these results to both the power of finetuning and limitations in the evaluation datasets, as well as (for the scholarbert models) small model sizes relative to the large pretraining corpus. we make the scholarbert models available on huggingface ( https://huggingface.co/ globuslabs ). while we cannot share the full public resource dataset, we have provided a sample of open-access articles from the dataset ( https://github.com/tuhz/ publicresourcedatasetsample ) in both the original pdf and extracted txt formats to illustrate the quality of the pdf-to-text preprocessing.