ex BERT : Extending Pre-trained Models with Domain-specific Vocabulary Under Constrained Training Resources (2020)

Abstract: We introduce exBERT, a training method to extend BERT pre-trained models from a general domain to a new pre-trained model for a specific domain with a new additive vocabulary under constrained training resources (i.e., constrained computation and data). exBERT uses a small extension module to learn to adapt an augmenting embedding for the new domain in the context of the original BERT’s embedding of a general vocabulary. The exBERT training method is novel in learning the new vocabulary and the extension module while keeping the weights of the original BERT model fixed, resulting in a substantial reduction in required training resources. We pre-train exBERT with biomedical articles from ClinicalKey and PubMed Central, and study its performance on biomedical downstream benchmark tasks using the MTL-Bioinformatics-2016 datasets. We demonstrate that exBERT consistently outperforms prior approaches when using limited corpus and pre-training computation resources.

## introduction pre-trained language representation models have led to breakthrough performance improvements in downstream natural language processing (nlp) tasks including named entity recognition @xcite , question answering @xcite , and sentence classification @xcite . however, pre-trained language models face two challenges as their applications expand: 1) large training resources: training requires substantial computation and data, see, e.g., bertlarge @xcite , roberta @xcite . 2) embedding of domain-specific vocabulary: a specialized domain, such as the biomedical domain on which this work focuses, has its own vocabulary, and sentences in the domain may have words from both the original language model's vocabulary and new domain-specific vocabulary. being able to operate on this mixture of vocabulary is essential in achieving high performance on downstream tasks in the new domain @xcite . these challenges are particularly pronounced in biomedical domains, where there are many domainspecific words. prior approaches have addressed these issues by either constructing the pre-trained model from scratch with a new vocabulary (e.g., scibert @xcite ) or adapting the existing pre-trained model by using it as the initial model in learning vocabulary embeddings for the new domain (e.g., biobert @xcite ). however, constructing the model with a new vocabulary from scratch requires substantial computational resources and training data. adapting the existing pre-trained model leads to sub-optimal performance on downstream tasks because the original vocabulary may not be proper for biomedical domains @xcite . we propose exbert, a novel approach that addresses these challenges by explicitly incorporating the new domain's vocabulary, while being able to reuse the original pre-trained model's weights as is to reduce required computation and training data. specifically, exbert extends bert by augmenting its embeddings for the original vocabulary with new embeddings for the domain-specific vocabulary via a learned small "extension" module. the output of the original and extension modules are combined via a trainable weighted sum operation. exbert after pre-training achieves higher performance than the biobert adaption method under constrained training resources when evaluated on two biomedical downstream benchmark nlp tasks: name entity recognition (ner) @xcite and relation extraction (re) @xcite . the primary contribution of this paper is a pretraining method allowing low-cost embedding of domain-specific vocabulary in the context of an existing large pre-trained model such as bert. the source code is available at https://github.com/ cgmhaicenter/exbert . ## related work learning representations of natural languages is useful for a variety of nlp tasks @xcite . it has been demonstrated that larger model size and corpus size benefit performance @xcite . a widely used pre-training model, bert @xcite , is a transformer-based model @xcite pre-trained with a masked language model and next sentence prediction task. the vocabulary used by bert contains words and subwords extracted from a general language corpus (english wikipedia and bookscorpus) by wordpiece @xcite . to get a biomedical domain-specific pre-training language model, biobert @xcite continues training the original bert model with a biomedical corpus without changing the bert's architecture or the vocabulary, and achieves improved performance in several biomedical downstream tasks. however, the use of original bert's general vocabulary often splits a domain-specific word into several sub-words, making the training much more challenging. scibert @xcite compares the vocabulary extracted from general and scientific articles, and finds 58% of the scientific vocabulary is not included in the original bert's vocabulary. to address this problem, scibert uses a new vocabulary, including high-frequency words and subwords in scientific articles. results show that the new vocabulary helps the performance of downstream tasks. however, the new vocabulary is not recognized by the pre-trained model; therefore, the model needs to be trained from scratch, requiring substantial computing resources and training data. in a recent study, pubmedbert @xcite ) pre-trained the model from scratch with pubmed articles and a customized vocabulary (constructed from the pubmed articles). this study indicates that a proper vocabulary helps the performance of downstream tasks in specific domains. however, training the model from scratch is extremely expensive in terms of data and computation. in multilingual language modeling, the out of vocabulary (oov) problem harms the performance due to the limited vocabulary that cannot cover all the words in each language. the mixture mapping method of @xcite represents each oov word as a mixture of english subwords where the english subwords are already in the original vocabulary. however, our preliminary experiments have shown that directly initializing the embedding of the domain-specific words with the mixture of the subword embeddings dose not benefit the performance. transfer learning with extra adaptors @xcite applied to the pre-trained model shows competitive performance compared with fine-tuning the pre-trained model. training only a relatively small adaptor module is parameter efficient and the origin model is kept the same. similar to this concept but not in a fine-tuning paradigm, we pre-train only the size-free extension module and the embedding layer of the extension vocabulary. ## exbert approach for exbert, we augment the original bert's embedding layer with an extension embedding layer and corresponding domain-specific extension vocabulary, and add an extension module to each transformer layer. ## conclusion with exbert we can reuse pre-trained language models for new domains under limited training resources. the approach could be particularly attractive to ad-hoc and special-purpose domains with unique vocabularies, such as some fields in law, medicine, and engineering, which have very limited training data for model pre-training and demand fast turnaround training.