G - MAP : General Memory-Augmented Pre-trained Language Model for Domain Tasks (2022)

Abstract: General pre-trained language models (PLMs), such as BERT, have achieved remarkable performance on various NLP tasks. Recently, domain-specific PLMs have been proposed to boost the task performance of specific domains (e.g., biomedical and computer science) by continuing to pre-train general PLMs with domain-specific corpora. However, this domain-adaptive pre-training (DAPT (CITATION)) tends to forget the previous general knowledge acquired by general PLMs, which leads to a catastrophic forgetting phenomenon and sub-optimal performance. To alleviate this problem, we propose a new framework of M emory- A ugmented P re-trained Language Model ( MAP ), which augments the domain-specific PLM by a memory built from the frozen general PLM without losing the general knowledge. Specifically, we propose a new memory-augmented layer, and based on it, different augmentation strategies are explored to build memory and fusion memory into domain-specific PLM. We demonstrate the effectiveness of MAP on different domains (biomedical and computer science publications, news, and reviews) and different kinds (text classification, QA, NER) of tasks, and the extensive results show that the proposed MAP can achieve SOTA results on these tasks.

## introduction pre-trained language models (plms), such as bert @xcite and roberta @xcite , have achieved promising performance on nlp tasks. typically, these general models are firstly pre-trained on large unlabeled corpus and then directly fine-tuned on downstream tasks. however, there is an inherent gap in text distribution between unlabeled pre-training corpus plms on the samples of biomedical (bm) and computer science (cs). figure c means the loss of these models on the samples from the pre-training (pt) corpus of roberta. we report the results of @xcite and lower mlm loss is better. and labeled task corpus, which leads to the distribution shift problem @xcite and makes plms perform poorly on some domain tasks @xcite . to address this shift problem, the domain-adaptive pretraining (dapt) is proposed @xcite @xcite to further pretrain general plms on large-scale domain corpora, achieving better performance than general plms. although dapt can effectively learn the domain distribution of the target task, its continual pretraining process updates the parameters of general plms, which inevitably leads to partial general knowledge being forgotten. this catastrophic forgetting @xcite @xcite phenomenon is verified in figure 1 , where we observe that the domainspecific plms show better results than general plms on domain corpus, but perform worse on the general corpus. we argue that this forgotten knowledge is beneficial for domain-specific plms and should be used to improve their generalization ability on domain tasks. to alleviate the catastrophic forgetting, we propose a simple yet effective memory-augmented framework named general memory-augmented pre-trained model (g-map). in addition to the backbone domain-specific plm, g-map introduces a new memory-augmented layer. it explicitly incorporates the representation built from a frozen general plm as the memory to make the backbone model access the complete general knowledge. then, a new proposed memory-attention within the memory-augmented layer enables the domain-specific plm adaptively combine the memory representation and the domain-specific representation. using the memory built from the frozen general plm has two advantages: (1) frozen plm never suffers from forgetfulness since the parameters remain unchanged @xcite ; (2) it doesn't require additional training for the general plm during fine-tuning. however, building and fusing memory into a backbone model is essentially a many-to-many scenario, where we need to choose which layer output of the general plm as the memory representation, and which layer in the domain-specific plm should be fused. thus, we propose several memory-augmented strategies for better building and then combining the memory representation into domain-specific plm. we evaluate our g-map on text classification, question answering (qa) and name entity recognition (ner) tasks covering four domains: biomedical science, computer science, news, and reviews. experimental results demonstrate that g-map outperforms existing baselines on all tasks. we compare different memory-augmented strategies, and the results show that the proposed chunk-based gated memory transfer strategy achieves the best results. in addition, for the memory representation building, we empirically find that the freezing way is better than the unfreezing one, which also has better training efficiency. furthermore, we apply the proposed framework to a small-scale domain pre-training setting and find that g-map is also practical in achieving lower mlm loss. our contributions are summarized below: plm-g memory-augmented strategy memory cache m ... more layers ... transformer layer mem-augmented layer transformer layer embedding layer ...... classifcation layer plm-d ... conditional probability scores for pseudodisambiguation... frozen unfrozen m f mem-augmented layer ...... ... ... augmented layer. in this section, we first overview the g-map framework. then we detail a new memory-augmented layer that fuses general knowledge into domainspecific plms. finally, we propose different memory-augmented strategies, including singlelayer memory transfer, multiple-layer memory transfer, gated memory transfer, and chunk-based gated memory transfer. ## experiments in this section, we first introduce the evaluation tasks and metrics. then we illustrate the baseline methods, and implementation settings. finally, we conduct the experimental analysis of g-map. ## further discussion in the following sections, we conduct some detailed analysis of g-map to demonstrate the effectiveness of the frozen general plm and memoryattention. moreover, we apply the proposed framework in the pre-training stage and study the effect of layer selection on performance. ## conclusion in this work, we propose g-map, a novel framework that utilizes the memory-augmented layer to fuse the memory representation built from the frozen general plm to mitigate catastrophic forgetting of general knowledge caused by domain-adaptive pre-training. we explore dif-ferent memory-augmented strategies to construct the memory representation and empirically find that chunk-based gate memory transfer achieves the most optimal performance. we validate g-map on various domains of classification, qa, and ner tasks. the results show that our method consistently outperforms existing baselines on all datasets, implying that explicitly leveraging forgotten general knowledge is beneficial for domainspecific downstream tasks.