Choose Your Words Wisely: Domain-adaptive Masking Makes Language Models Learn Faster (2025)

Abstract: Foundational Language Models perform significantly better on downstream tasks in specialised domains (such as law, computer science, and medical science) upon being further pre-trained on extensive domain-specific corpora, but this continual pre-training incurs heavy computational costs. Indeed, some of the most performant specialised language models such as BioBERT incur even higher computing costs during domain-specific training than the pre-training cost of the foundational models they are initialised from. In this paper, we argue that much of the extended pre-training is redundant, with models seemingly wasting valuable resources re-learning lexical and semantic patterns already well-represented in their foundational models such as BERT, T5 and GPT. Focusing on Masked Language Models, we introduce a novel domain-specific masking strategy that is designed to facilitate continual learning while minimizing the training cost. Using this approach, we train and present a BERT-based model trained on a biomedical corpus that matches or surpasses traditionally trained biomedical language models in performance across several downstream classification tasks while incurring up to 11 times lower training costs.

## introduction rapid advancements in large language models (llms) @xcite have resulted in an increased focus on their capabilities in specialised domains like biology, law and computer science @xcite . the significance of foundational models like bert, t5, gpt and llama for application in such fields is particularly evident from the multitude of models based on them -such as legalbert @xcite , scifive @xcite , biogpt @xcite and pmc-llama @xcite -delivering state-of-the-art results on benchmarks in their respective domains. given the vast amount of training data available for continual pre-training across various fields, existing literature shows that further pre-training foundational language models on a domain-specific corpus yields better model performance across downstream tasks @xcite . however, pre-training language models can be resource-prohibitive, both in terms of monetary cost and time spent on training. this is especially true for large, parameter-dense language models like llama @xcite , which are not only significantly more expensive to train further, but also exhibit much smaller improvements in downstream performance per unit of compute spent @xcite . it is therefore worthwhile to attempt to leverage the capabilities of contemporary foundational language models for tasks across such domains without expending exorbitant computing resources on diminishing returns. in this paper, we present a novel strategy for continual/mixed-domain pretraining that emphasises selecting relevant (as opposed to random) training samples to maximise compute efficiency. we test our strategy in the biomedical domain by further pre-training bert on a corpus of pubmed abstracts, mirroring the selection of architecture and pre-training corpora of biobert @xcite , one of the most popular biomedical language models. in our testing over 8 named entity recognition (ner) tasks, the resultant model significantly outperforms biobert-v1.0 at two-thirds of the compute cost, and performs similarly to biobert-v1.1 at about oneeleventh of its compute cost. ## methodology we begin by creating a biomedical corpus consisting of pubmed abstracts publicly available at https://pubmed.ncbi.nlm.nih.gov/download/ , amounting to about 9.4gb of text. leveraging the linguistic difficulty criterion and subsequent curriculum generation approach introduced by @xcite , who claim that frequently occuring words that have many connections in a large knowledge graph are easier to learn, we build a set s of "basic" concepts -i.e. the n concepts with the most connections in a large-scale knowledge graph that occur in the corpus above a threshold frequency f . iterating through all the elements s i of s, we add s i and every concept in conceptnet within k hops of s i to a new set c, which acts as a "curriculum" consisting of relevant concepts. following manual assessment of the curriculum generated, we settled on using @xmath0, 000, @xmath1, 000 and @xmath2. for our purposes, despite the availability of biomedical knowledge graphs like bikg @xcite and bios @xcite , we chose to use the general-domain conceptnet @xcite as the knowledge graph. we did not assess the overlap between these specific knowledge graphs and our corpus, and could not be certain that their usage would not be counterproductive given that bert's vocabulary itself is not tailored towards biomedical terms. moreover, this makes our approach easier to generalise for other domains without pre-existing large knowledge graphs. nonetheless, we recognise that the use of domainspecific knowledge graphs for concept extraction, wherever available, is worth investigating in future studies. since this curriculum includes general-domain concepts already represented well in bert, we iterate once over bert's corpus (wikipedia + bookscorpus), identify concepts occuring more than f /3 times and remove them from c. note that this cutoff frequency has been scaled down with respect to the threshold frequency used for the pubmed corpus above to account for the difference in sizes between the corpora. we then iterate through c and remove any concepts that do not occur at all in the pubmed corpus, ensuring that the concepts now contained in c are relevant to the biomedical domain. we then initialize our model from the publicly available bert-base checkpoint, and train it for masked language modeling (mlm) over our corpus. past studies indicate that the difference between using cased and uncased models to warmstart biomedical language models is minimal with no clear advantage for either @xcite it is beyond the scope of our current experiment to test and compare the two. for our purposes, we use the uncased version of the model. differing from the likes of bert and biobert that randomly mask 15% of the tokens in each batch, we mask only the tokens that form a concept within the previously curated curriculum c while ensuring that no more than 20% of the tokens in any batch are masked. as concepts can span multiple tokens, we follow @xcite 's whole concept masking (wcm) strategy such that all the tokens comprising a single concept are simultaneously masked. as is the standard, we replace 80% of the masked concepts with a mask token, replace another 10% with a random token and do not replace the remaining 10%. since existing literature shows minimal gains from calculating the next sentence prediction (nsp) loss @xcite , we chose to omit it; mlm was our sole pre-training objective. ## results the results obtained by our model relative to bert, biobert-v1.0, biobert-v1.1 and the current sota foot_1 are shown in