Adapting a Language Model While Preserving its General Knowledge (2022)

Abstract: Domain-adaptive pre-training (or DA-training for short), also known as post-training, aimsto train a pre-trained general-purpose language model (LM) using an unlabeled corpus of aparticular domain to adapt the LM so that end-tasks in the domain can give improved performances. However, existing DA-training methods are in some sense blind as they do not explicitly identify what knowledge in the LM should be preserved and what should be changed by the domain corpus. This paper shows that the existing methods are suboptimal and proposes a novel method to perform a more informed adaptation of the knowledge in the LM by (1) soft-masking the attention heads based on their importance to best preserve the general knowledge in the LM and (2) contrasting the representations of the general and the full (both general and domain knowledge) to learn an integrated representation with both general and domain-specific knowledge. Experimental results will demonstrate the effectiveness of the proposed approach.

## introduction pre-trained general-purpose language models (lms) like bert @xcite , roberta @xcite , and gpt-3 @xcite have become a standard component in almost all nlp applications. researchers have also found that domain-adaptive pre-training (or da-training for short) using an unlabeled corpus in a specific domain to adapt an lm can further improve the end-task performance in the domain @xcite @xcite . note that domain-adaptive pre-training is also called post-training @xcite . existing da-training methods simply apply the same pre-training objective, i.e., the mask language model (mlm) loss, to further train an lm using a domain corpus. these methods are sub-optimal because they do not explicitly identify what should be preserved and what should be updated in the lm by the domain corpus. this paper argues that a good da-training method has two needs. on the one hand, the general language knowledge learned in the lm should be preserved as much as possible because the target domain data is typically not large enough to be sufficient to learn the general knowledge well. for example, some words and their contexts may appear infrequently in a particular domain. the knowledge about them cannot be learned accurately based on the domain data alone. when these words and contexts appear in an end-task, the system will have difficulties. thus, we need to rely on the knowledge about them in the lm. since existing da-training updates the lm with little guidance, such useful general knowledge may be corrupted. on the other hand, due to polysemy (same word with different meanings in different domains) and the fact that different domains also have their special word usages and contexts, the lm should be specialized or adapted to the target domain. a good da-training should balance these two needs to adapt the lm to the target domain with minimal corruption to the good general knowledge in the lm. this paper proposes a novel technique to enable a more informed adaptation to (1) preserve the general knowledge in the lm as much as possible, and (2) update the lm to incorporate the domainspecific knowledge of the target domain as needed. the focus of the existing da-training research has been on (2). as we argued above, (1) is also important as focusing only on (2) may destroy some useful general knowledge and produce sub-optimal results for end-tasks. to achieve (1), the system should constrain the gradient update of each attention head 2 based on its importance to the general knowledge so that the general knowledge in lm can be preserved as much as possible. with (1), (2) will be able to change the part of the general knowledge that needs to be updated to adapt the lm to suit the target domain. 3 in this paper, we propose a novel model called dga (da-training -general knowledge preservation and lm adaptation) for the purpose. the key idea of the proposed method is to preserve the general language knowledge in the lm while adapting the lm to a specific domain. however, it is not obvious how this can be done, i.e., how to find those parameters that are important for the general knowledge and how to protect them. this paper proposes a novel proxy-based method to achieve the objectives. it works as follows. dga first estimates the importance of each attention head in the lm via the newly proposed proxy kl-divergence loss (sec. 3.1). this importance score reflects how important each attention head is to the general knowledge. based on the importance scores, it performs two key functions: the first function uses the scores to soft-mask (rather than binary-mask or completely block) the gradient update to prevent important general knowledge in lm from being unnecessarily corrupted. this is related to pruning of unimportant attention heads @xcite . however, pruning is not directly applicable to da-training as we will show in sec. 2. the proposed soft-masking constrains only the backward gradient flow in training. it is not necessary to soft-mask the forward pass in either training or inference. this is important because using the knowledge in the full network encourages maximal integration of pre-trained general knowledge and the target domain-specific knowledge. the second function contrasts the representation for the general knowledge in the lm and the full (including both the general and the domain-specific) knowledge to learn an integrated representation (sec. 3.2). 4 in summary, this paper makes two key contributions. to the best of our knowledge, none of these has been reported in the literature before. extensive experiments have been conducted in 6 different domains and on 10 baselines to demonstrate the effectiveness of the proposed dga. researchers have applied da-training to many domains, e.g., reviews @xcite , biomedical text @xcite , news and papers @xcite , and social media @xcite . however, they all use the same mask language model (mlm) loss. we argue that it is sub-optimal and it is also important to preserve the general knowledge in the lm as much as possible and integrate it with the target domain knowledge. network pruning as importance computation. it is known that many parameters in a neural network are redundant and can be pruned @xcite . this has also been shown for pre-trained transformer @xcite @xcite @xcite . a popular pruning method is to discard the parameters with small absolute values @xcite . other methods prune the network at a higher level. in a transformer-based model, these include pruning the attention head @xcite @xcite and pruning sub-layers in a standard transformer layer @xcite . however, the above methods are not directly applicable to us as we need to compute the head importance for the lm using unlabeled domain data, while the above approaches are all for supervised end-tasks. we propose to use a proxy kl-divergence loss for our purpose. note that it is possible to prune other sublayers in the transformer. however, as shown in sec. 4.3, estimating the importance for other layers does not improve the performance. contrastive learning. contrastive learning @xcite can learn good representations by maximizing the similarity of positive pairs and minimizes that of negative pairs: we mainly use contrasive loss to contrast the representations of the important general knowledge in the original lm and the full knowledge (both the general and domain-specific knowledge) to achieve a good integration of the general knowledge and the domain specific knowledge. ## proposed dga system as discussed earlier, dga goes beyond the mlm loss to perform two more functions: (1) preserving the important general knowledge in the lm by soft-masking the attention heads based on their importance. this helps avoid potential corruption of the general knowledge in the lm in da-training (sec. 3.1). however, the challenge is how to identify the general knowledge in the lm and how to protect it. we will propose a method to do that. ## experiments we follow the experiment setup in @xcite . roberta @xcite 7 is used as the lm. in each experiment, we first da-train the lm and then fine-tune it on the end-task. the final evaluation is based on the end-task results. ## conclusion this paper argued that an effective da-training method should effectively integrate the target domain knowledge to the general knowledge in the lm. existing approaches do not explicitly do this. this paper proposed a novel method dga to achieve it (1) by estimating the attention heads importance in lm and using the importance scores to soft-mask the attention heads in da-training to preserve the important knowledge in lm as much as possible, and (2) by contrasting the general and the full knowledge. extensive experiment results demonstrated the effectiveness of the proposed approach dga.