Efficient Domain Adaptation of Language Models via Adaptive Tokenization (2021)

Abstract: Contextual embedding-based language models trained on large data sets, such as BERT and RoBERTa, provide strong performance across a wide range of tasks and are ubiquitous in modern NLP. It has been observed that fine-tuning these models on tasks involving data from domains different from that on which they were pretrained can lead to suboptimal performance. Recent work has explored approaches to adapt pretrained language models to new domains by incorporating additional pretraining on domain-specific corpora and task data. We propose an alternative approach for transferring pretrained language models to new domains by adapting their tokenizers. We show that domain-specific subword sequences can be determined efficiently directly from divergences in the conditional token distributions of the base and domain-specific corpora. In datasets from four disparate domains, we find adaptive tokenization on a pretrained RoBERTa model provides greater than 85% of the performance benefits of domain specific pretraining. Our approach produces smaller models and less training and inference time than other approaches using tokenizer augmentation. Although using adaptive tokenization incurs a 6% increase in model parameters (due to the introduction of 10k new domain-specific tokens), our approach, using 64 CPUs, is >72x faster than further pretraining the language model on domain-specific corpora on 8 TPUs.

## introduction pretrained language models (plms) trained on large "base" corpora, @xmath0; @xcite , are used in many nlp tasks. these models first learn contextual representations in an unsupervised manner by minimizing a masked language modeling objective over a base corpus. this stage of unsupervised language model training is referred to as "pretraining". subsequently, for supervised classification tasks, the output head of this pretrained model is swapped for a lightweight classifier and trained further on a classification objective over labeled data, referred to as "fine-tuning". recent work has examined the transferability of plms @xcite and their contextual representations to domains differing from their base corpora. on text classification tasks from four different domains, it was shown that continuing to pretrain roberta's contextual embeddings on additional domain (dapt) and/or task-specific data (tapt) resulted in performance gains over only fine-tuning a baseline roberta model. these performance gains, however, were inferior to each task's start-of-the-art metrics which were largely based on training versions of roberta, or other lms, from scratch on a large sample of in-domain data. these performance gains come at substantial financial, time, and environmental costs in the form of increased computation, with pretraining an lm from scratch being the most expensive, using additional pretraining in the middle, and only fineturning an off-the-shelf model the most economical. one observed advantage @xcite that pretraining from scratch on in-domain data has over continual pretraining is that the tokenizer's vocabulary captures domain-specific terms. this allows semantics of those terms to be directly learned in their fixed embeddings, and relieves the language model from having to encode these semantics through the contextual embeddings of these domain-specific term's subwords. recent work @xcite ; @xcite has shown adding whole words common to the target domain but absent from a plm's tokenizer improves performance on single tasks. in this work, we show that augmenting an plm with statistically derived subword tokens selected for domain association with simple embedding initializations and no further pretraining provide an effective means of adapting a plm across tasks and domains. in contrast, both @xcite and @xcite add inefficiencies by respectively requiring further masked language model (mlm) pretraining and doubling the resources needed for inference. in this paper, we efficiently adapt a plm by simply augmenting its vocabulary with domainspecific token sequences. we find that this adaptation, which requires no further pretraining, rivals the accuracy of domain and task-adapted pretraining approaches proposed in @xcite but requires only a small fraction of the compute cost. zhang et al. (2020) augment roberta's vocabulary with in-domain oov whole words. the most frequently occurring whole words are added until the oov rate drops to 5% on the task corpus. they randomly initialize weights and pretrain a model. this improves performance on techqa and askubuntu. @xcite also augmented bert with tokens selected by frequency (12k oov wordpieces were used) and pretrained a modified version of bert which allowed for only new token's embeddings to be modified while the original embeddings remained fixed. they found that using more than 12k augmented tokens didn't improve their biomed ner and relation extraction performance, and that, once augmented, performance improved with more pretraining (4-24 hours were studied.) poerner et al. (2020) augment bert's vocabulary with all in-domain oov whole words, adding 31k tokens to bert-base-cased's 29k wordpieces. they trained a word2vec model on an in-domain corpus and fit a linear transformation to project the word embeddings into the model's input embedding space. no further pretraining is done, but during finetuning, the original tokenizer and the adapted tokenizer are both used. for inference, the finetuned model is run with both the original tokenizer and the adapted tokenizer and the outputs are averaged. their f1 score outperforms bert on all eight biomedical ner tasks studied. the approach has the disadvantage of increasing the parameter size of bert-base-cased by 2.2x due to the embeddings of added tokens and doubles the resources needed for inference. @xcite demonstrates how wordpiece tokenization does not capture the semantics of derivationally complex words as well as an approach using a modified version of wordpiece designed to produce subword segmentations consisting of linguistic prefixes, suffixes and affixes @xcite . this subword tokenizer outperformed wordpiece in determining words' polarity or their source domains. experiments were conducted on novel embedding tokens in bert via approaches including a projection-based method and mean pooling (both similar to §3.3). training language models from scratch in the domain of interest has been shown to provide improved in-domain performance when compared to out-of-domain plms @xcite . in addition to @xcite , prior work has shown the effectiveness of continued pretraining for domain adaptation of plms @xcite ; @xcite ; @xcite . for the task of aspect-target sentiment classification, @xcite uses both dapt and task-specific fine-tuning in order to adapt language models representations. identifying domaincharacteristic words is a well-studied problem, and many metrics have been proposed for this task through comparing the distributions of tokens in contrasting corpora @xcite ; @xcite ; @xcite . @xcite used the pointwise kl-divergence to distinguish informativeness of key phrase candidates in a domain corpus relative to a background. ## adaptive tokenization of contextual embeddings we define adaptive tokenization (at) as the process of augmenting a plm's tokenizer and fixed subword embeddings with new entries taken from a novel corpus. at consists of two goals which must be achieved for domain adaptation. first, selection of domain-specific tokens, with which to augment a pretrained tokenizer, from an in-domain corpus must be determined. second, an appropriate initialization in the input space of the contextual embedding models needs to be determined for additions to the tokenizer vocabulary. in this section, we detail approaches for each of these linked tasks. ## experimentation in this section, we perform evaluation of our adaptation approach on six natural language processing tasks in four domains, biomedical, computer science, news, and reviews, following the evaluations in @xcite . due to resource constraints, we perform experimentation on all datasets in @xcite excluding the helpfulness dataset from the reviews domain and the hyperpartisan dataset in the news domain. each of the excluded datasets contain greater than 100k training examples, resulting in greater than 12 hours of time required for finetuning on 8 tesla v100 gpus for a single seed. approaches roberta-base, a commonly used plm with high performance, is used as a baseline on which supervised finetuning is performed separately for each dataset. additionally, we compare at to the dapt method from @xcite . as we do not make use of task specific data (i.e., the training data used in fine-tuning), at is comparable to dapt in terms of the data utilized. we focus on using large, in-domain data sets which are commonly used in further pretraining (rather than variably sized task-data) since their size both allows for reliable extraction of characteristic subtoken sequences to use in tokenizer augmentation. adaptive tokenization for task-specific data is future work. ## conclusion in this paper, we introduced adaptive tokenization (at) a method for efficiently adapting pretrained language models utilizing subword tokenization to new domains. at augments a plm's tokenization vocabulary to include domain-specific token sequences. we provide two approaches for initializing augmented tokens: mean subword and projections from static subword embeddings. at requires no further language model pretraining on domain-specific corpora, resulting in a 38x speedup over pretraining on the corpora without specialized hardware. across four domains, at @xmath1. this initial work suggests that adapting the subword tokenization scheme of plms is an effective means of transferring models to new domains. future work entails hybrid approaches using both at and small amounts of lm pretraining, alternative metrics for augmented token selection, improved initialization of augmented token representations, and the use of task data.