Inexpensive Domain Adaptation of Pretrained Language Models: Case Studies on Biomedical NER and Covid-19 QA (2020)

Abstract: Domain adaptation of Pretrained Language Models (PTLMs) is typically achieved by unsupervised pretraining on target-domain text. While successful, this approach is expensive in terms of hardware, runtime and CO 2 emissions. Here, we propose a cheaper alternative: We train Word2Vec on target-domain text and align the resulting word vectors with the wordpiece vectors of a general-domain PTLM. We evaluate on eight English biomedical Named Entity Recognition (NER) tasks and compare against the recently proposed BioBERT model. We cover over 60% of the BioBERT - BERT F1 delta, at 5% of BioBERT’s CO 2 footprint and 2% of its cloud compute cost. We also show how to quickly adapt an existing general-domain Question Answering (QA) model to an emerging domain: the Covid-19 pandemic.

## introduction pretrained language models (ptlms) such as bert @xcite have spearheaded advances on many nlp tasks. usually, ptlms are pretrained on unlabeled general-domain and/or mixed-domain text, such as wikipedia, digital books or the common crawl corpus. when applying ptlms to specific domains, it can be useful to domain-adapt them. domain adaptation of ptlms has typically been achieved by pretraining on target-domain text. one such model is biobert @xcite , which was initialized from general-domain bert and then pretrained on biomedical scientific publications. the domain adaptation is shown to be helpful for target-domain tasks such as biomedical named entity recognition (ner) or question answering (qa). on the downside, the computational cost of pretraining can be considerable: biobertv1.0 was adapted for ten 1 www.github.com/npoe/covid-qa days on eight large gpus (see we therefore propose a fast, cpu-only domainadaptation method for ptlms: we train word2vec @xcite on target-domain text and align the resulting word vectors with the wordpiece vectors of an existing general-domain ptlm. the ptlm thus gains domain-specific lexical knowledge in the form of additional word vectors, but its deeper layers remain unchanged. since word2vec and the vector space alignment are efficient models, the process requires a fraction of the resources associated with pretraining the ptlm itself, and it can be done on cpu. in section 4, we use the proposed method to domain-adapt bert on pubmed+pmc (the data used for biobertv1.0) and/or . we improve over general-domain bert on eight out of eight biomedical ner tasks, using a fraction of the compute cost associated with biobert. in section 5, we show how to quickly adapt an existing question answering model to text about the covid-19 pandemic, without any target-domain language model pretraining or finetuning. in the case of bert, f lm is a transformer @xcite , followed by a final feed-forward net. during pretraining, the feed-forward net predicts the identity of masked wordpieces. when finetuning on a supervised task, it is usually replaced with a randomly initialized layer. ## conclusion as a reaction to the trend towards high-resource models, we have proposed an inexpensive, cpuonly method for domain-adapting pretrained language models: we train word2vec vectors on target-domain data and align them with the wordpiece vector space of a general-domain ptlm. on eight biomedical ner tasks, we cover over 60% of the biobert -bert f1 delta, at 5% of biobert's domain adaptation co 2 footprint and 2% of its cloud compute cost. we have also shown how to rapidly adapt an existing bert qa model to an emerging domain -the covid-19 pandemic -without the need for target-domain language model pretraining or finetuning. we hope that our approach will benefit practitioners with limited time or resources, and that it will encourage environmentally friendlier nlp.