Domain Knowledge Transferring for Pre-trained Language Model via Calibrated Activation Boundary Distillation (2022)

Abstract: Since the development and wide use of pretrained language models (PLMs), several approaches have been applied to boost their performance on downstream tasks in specific domains, such as biomedical or scientific domains. Additional pre-training with in-domain texts is the most common approach for providing domain-specific knowledge to PLMs. However, these pre-training methods require considerable in-domain data and training resources and a longer training time. Moreover, the training must be re-performed whenever a new PLM emerges. In this study, we propose a domain knowledge transferring (DoKTra) framework for PLMs without additional in-domain pretraining. Specifically, we extract the domain knowledge from an existing in-domain pretrained language model and transfer it to other PLMs by applying knowledge distillation. In particular, we employ activation boundary distillation, which focuses on the activation of hidden neurons. We also apply an entropy regularization term in both teacher training and distillation to encourage the model to generate reliable output probabilities, and thus aid the distillation. By applying the proposed DoKTra framework to downstream tasks in the biomedical, clinical, and financial domains, our student models can retain a high percentage of teacher performance and even outperform the teachers in certain tasks. Our code is available at https://github.com/DMCB-GIST/DoKTra .

## introduction recently, transformer @xcite -based language models have been successfully applied in the field of natural language processing (nlp). in particular, the two-stage approach of "pre-training and fine-tuning," such as bert @xcite , has become the standard for nlp applications. generally, a transformer-based model is pretrained with a large amount of text data in an unsu- * hyunju lee is the corresponding author. pervised manner, and then fine-tuned with a small dataset for several downstream tasks. further, advanced pre-trained language models (plms) with improved architectures or training methods continue to emerge, including albert @xcite or roberta @xcite . however, these models must be further improved for tasks requiring domain knowledge, such as those in the biomedical or financial domains, as the pre-training data usually consist of general domain text (e.g., wikipedia). additional pre-training with in-domain text has been proposed to provide the plms with domain-specific knowledge. for example, in the biomedical domain, several domainspecific plms trained with large biomedical texts, such as biobert @xcite , pubmedbert @xcite and bluebert @xcite , have been successfully used as strong baselines for several downstream tasks. nevertheless, additional pre-training has several limitations, such as the need for sufficient training data and resources, and a longer training time. furthermore, whenever a new plm emerges, it must be re-trained to create more advanced domain-specific models. to address this issue, we propose an efficient domain-knowledge transferring framework that does not require additional pre-training steps. specifically, we focus on the applicability of knowledge distillation @xcite as a domainknowledge transfer method, not only for model compression. knowledge distillation is a wellknown knowledge transfer method that is primarily used for model compression. the knowledge from a larger and more effective teacher model is distilled to a smaller student model by encouraging it to mimic the teacher characteristics, such as soft probabilities @xcite or hidden representations @xcite . in this study, we propose a domain knowledge transfer (doktra) framework for an advanced plm via calibrated activation boundary distillation. in contrast to the existing in-domain pretraining methods, we transfer domain knowledge to a new language model using only an existing in-domain pre-trained model, and without a timeconsuming pre-training on the new model. for instance, biobert was pre-trained for 23 days on 8 nvidia v100 gpus @xcite . we can estimate that if a new, larger language model is pre-trained with a large number of biomedical texts, its training duration would be longer than that of biobert. however, our framework can be executed in a few hours on a single 24 gb gpu. the comparison between our framework and a conventional approach is visualized in figure 1 . specifically, we apply the calibration method to generate a reliable and well-supervising teacher model. then, we apply activation boundary distillation @xcite to distill the domain knowledge to the student, which is more efficient with a small amount of training data. moreover, by selecting language models more advanced than the teacher as students, we allow the student models to acquire additional domain knowledge while preserving its superiority. we apply our framework to the biomedical domain and verify its effectiveness by conducting experiments on several biomedical and clinical downstream tasks. consequent to applying our framework to albert and roberta student models, we were able to obtain models that retained most of the teacher model's performance with fewer model parameters (albert), and models with a higher performance than both students and teachers (roberta). we also investigate the general applicability of our framework by applying it to a financial domain plm and downstream tasks. the contributions of this study can be summarized as follows: ## doktra framework in this section, we introduce the doktra framework, which is the main approach to transfer domain-specific knowledge. ## conclusion in this study, we proposed the doktra framework as a domain knowledge transfer method for plms. the experimental results from the biomedical, clinical, and financial domain downstream tasks demonstrated that our proposed framework could transfer domain-specific knowledge into a plm, while preserving its own expressive advantages without any further pre-training with additional in-domain data. we employed advanced models as the student model and verified the future applicability of our framework to emerging language models by achieving even higher performances than the teacher model. however, the limitations of our approach are that it is task-specific and was evaluated only in classification tasks. our future studies would focus on developing the proposed framework as a task-agnostic method and evaluating it on various tasks. same way as the experiments for the biomedical domain. the actual values of the hyperparameters for the calibrated teacher training and activation boundary distillation with albert and roberta are summarized in tables a4 , a5 , and a6 . dataset ctt e β 1 chemprot 5 0.3 gad 3 0.7 ddi 5 0.3 i2b2 5 0.3 hoc 10 0 table a1: the hyperparameters for calibrated teacher training dataset abd e 1 lr 1 table a2: the hyperparameters for activation boundary distillation of the albert model dataset abd e 1 lr 1 e 2 lr 2 β 2 γ chemprot 5 table a3: the hyperparameters for activation boundary distillation of the roberta model. dataset ctt e β 1 fpb 5 0.7 fts 5 0. 3 table a4 : the hyperparameters for calibrated teacher training for the financial domain. table a5 : the hyperparameters for activation boundary distillation of the albert model for the financial domain. table a6 : the hyperparameters for activation boundary distillation of the roberta model for the financial domain.