Towards Simple and Efficient Task-Adaptive Pre-training for Text Classification (2022)

Abstract: Language models are pre-trained using large corpora of generic data like book corpus, com- mon crawl and Wikipedia, which is essential for the model to understand the linguistic characteristics of the language. New studies suggest using Domain Adaptive Pre-training (DAPT) and Task-Adaptive Pre-training (TAPT) as an intermediate step before the final finetuning task. This step helps cover the target domain vocabulary and improves the model performance on the downstream task. In this work, we study the impact of training only the embedding layer on the model’s performance during TAPT and task-specific finetuning. Based on our study, we propose a simple approach to make the in- termediate step of TAPT for BERT-based mod- els more efficient by performing selective pre-training of BERT layers. We show that training only the BERT embedding layer during TAPT is sufficient to adapt to the vocabulary of the target domain and achieve comparable performance. Our approach is computationally efficient, with 78% fewer parameters trained during TAPT. The proposed embedding layer finetuning approach can also be an efficient domain adaptation technique.

## introduction large-scale pre-trained language models (plms) are extensively trained on massive heterogeneous datasets, known as pre-training datasets. these models are "pre-trained" @xcite @xcite , where they learn contextual representations by unsupervised learning methods like masked language modeling and next sentence prediction. pre-training is followed by "finetuning", * authors contributed equally which uses supervised learning for tasks such as text-classification @xcite . the task for which finetuning is performed is called the downstream task. previous works have shown that the representations learned from pre-training datasets help the model achieve a strong performance across multiple downstream tasks @xcite . contextual representations are typically generated from generic large-scale corpora-based pre-training, while downstream tasks are usually domain-specific. recent work shows that additional pre-training performed on plms using in-domain and downstream task data improves the model's performance. domain adaptive pretraining @xcite ) is a method used to achieve the same by continued pre-training of the plm on a large corpus of unlabelled in-domain data. to expand on this approach, @xcite ; konlea and jannidisa (2020) continued pre-training of language models on smaller unlabeled data drawn from the given task (task-adaptive pre-training or tapt) mirrors the gains associated with domain adaptation and can also be used as an additional pre-training step after dapt to improve performance further. tapt (fig. 1 ) can also be used as an alternative to dapt when sufficient in-domain data is unavailable in the worst-case scenario. while the initial pre-training is essential for the model to understand and learn the linguistic characteristics of the english language through the encoder layer, dapt and tapt are essential for the model to get familiar with the domainspecific vocabulary without disturbing the domainindependent linguistic features learned by the encoder layer @xcite @xcite . with the motivation of adapting the pre-trained model to the target domain, we study the impact of training of only the embedding layer on the performance of the bert-based model during tapt. we specifically target the embedding layer as it encodes the token information. although additional pre-training on in-domain or task data has shown to improve model performance on downstream tasks, it adds to the computational overhead and an indirect financial and environmental impact @xcite . this work proposes a new method for efficient domain adaptation for bert-based models. after the initial pretraining of the language model, during tapt, we freeze the encoder layer and update only the embedding layer along with the task-specific dense layers. we show that this simple yet effective approach significantly reduces the number of trainable parameters and the time taken to train the model without impacting the model accuracy. the advantages of this restricted intermediate pre-training are twofold. it adapts the token embeddings to the target domain without forgetting the language characteristics learned by the upper bert encoder layers during initial large-scale pre-training, thus preventing catastrophic forgetting or overfitting @xcite due to smaller task-specific datasets. the main contributions of this work are as follows: ## related work recently, task-adaptive pre-training (tapt) has become a popular topic for research, introduced by @xcite . it is essentially the adaptation of a language model (lm) to a target task leading to the improvement in model performance. work done by @xcite expands and discusses the effectiveness of tapt and its fusion with self-training. @xcite investigated the benefits of tailoring a pertained model like roberta to the domain of a target task. their work analyses four domains, namely biomedical and computer science publications, news and reviews, on eight classification tasks. this investigation is further extended into the transferability of adapted language models across all the tasks and domains. finally, a study of the significance of pre-training on humancurated data is carried out. the study proposed in konlea and jannidisa (2020) discusses various strategies to adapt bert and distilbert to historical domains and tasks exemplary for computational humanities. the results encourage the integration of continued pertaining into machine learning tasks for improved performance stability. a combination of domain adaptation and task adaptation shows positive effects. when applied individually, task adaptation can be performed in every setup, unlike domain adaptation, where sufficient in-domain data is necessary. several approaches have been undertaken to make tapt more efficient, especially with methods involving word embeddings. a study in @xcite focuses on the static word embeddings of the pre-trained language models for domain adaptation. the researchers propose a process called task-adaptive pre-training with word embedding regularization (tapter) to teach pre-trained language models the domainspecific meanings of words. word embeddings in the target domain are obtained by training a fast-text model on the downstream task training data. these word embeddings are made close to the static word embeddings during tapt. tapter performs better than the standard finetuning and tapt when in-domain data is deficient in the initial pre-training dataset. another method is proposed by el boukkouri (2021) for specializing in general-domain embeddings in a low-resource context. more specifically, they have considered a worst-case scenario where only the target task corpus is available. given the availability of general-domain models which can be used to initialize training for any specialized do-main, it is concluded that re-training from a general model is less expensive and leads to comparable, although slightly lower performance. researchers propose an alternative approach in @xcite for transferring pretrained language models to new domains by adapting their tokenizers. it is shown that adaptive tokenization on a pre-trained roberta model provides more than 97% of the performance benefits of domain-specific pre-training. however, this approach incurs a 6% increase in model parameters due to the introduction of 10,000 new domainspecific tokens. methods to make tapt and dapt more efficient, as discussed above, involve methods such as using static pre-trained in-domain embeddings, adapting the tokenizer and training fast-text or word2vec models on in-domain data. these approaches introduce multiple extra steps to the pretraining of the plms, whereas our approach leverages bert's static embeddings matrix. given a specialized target domain, we aim to improve the quality of general-domain word representations using in-domain corpora. ## experimentation setup this section discusses the experimental setup followed to study the effect of restricting training to the embedding layer during tapt and finetuning. this setup is also used to evaluate the proposed efficient domain or task adaptation method. ## results as observed in we also study the impact of training the embedding layer during final finetuning. the results of the six configurations are shown in