An Empirical Investigation Towards Efficient Multi-Domain Language Model Pre-training (2020)

Abstract: Pre-training large language models has become a standard in the natural language processing community. Such models are pre-trained on generic data (e.g. BookCorpus and English Wikipedia) and often fine-tuned on tasks in the same domain. However, in order to achieve state-of-the-art performance on out of domain tasks such as clinical named entity recognition and relation extraction, additional in domain pre-training is required. In practice, staged multi-domain pre-training presents performance deterioration in the form of catastrophic forgetting (CF) when evaluated on a generic benchmark such as GLUE. In this paper we conduct an empirical investigation into known methods to mitigate CF. We find that elastic weight consolidation provides best overall scores yielding only a 0.33% drop in performance across seven generic tasks while remaining competitive in bio-medical tasks. Furthermore, we explore gradient and latent clustering based data selection techniques to improve coverage when using elastic weight consolidation and experience replay methods.

## introduction transformer @xcite based language modeling has taken over many previous pretraining and initialization approaches @xcite @xcite . fine-tuning using these architectures yields state-of-the-art results in the order of a few hours. the caveat to these models is that the initial training can be on the scale of many days if not weeks, distributed across multiple gpus @xcite , a costly endeavour. pre-trained language models are adapted to perform strongly in more specific domains as well. for example, while the original bert models @xcite were trained on english wikipedia articles and bookscorpus @xcite , the figure 1: traditional approaches (top) train independent domain specific language models (newswire, bio-medical, and clinical) which share no cross domain knowledge. they are further fine-tuned on their respective in-domain tasks. our approach (bottom) shows how several domains are introduced in sequence, with knowledge retention using mitigation techniques across all domains. here the final model has the capability to properly fine-tune on any domain specific task. unfortunately, continual learning (ring, 1997) suffers from catastrophic forgetting @xcite when incorporating domain data in a sequential manner. parameters shift towards capturing the current task (or domain) and if previous data is no longer available the model will lose representation of it. for many tasks the straightforward solution is to combine datasets during training and approach this as a multi-task learning (mtl) @xcite problem. mixing data has the desired effect of constraining parameters to find a space where both tasks reach close to optimal performance. we argue that these expensive pre-trained models are an example where mtl is not feasible in practice for several reasons. time and hardware accessibility are the largest constraints for developing such systems. access to large scale training data is generally not possible @xcite , and exact training configurations are equally difficult to gather with results being arduous to reproduce. resource usage has recently been criticized from another perspective as well. @xcite show that as deep neural architectures in the natural language community grow we increasingly trade results for carbon emissions. our work conducts an empirical investigation into suitable methods for multi-domain pre-training in a continual learning setting. we focus our efforts towards three methods: (i) elastic weight consolidation (ewc), (ii) learning rate control (lrc), and (iii) experience replay (er). ewc @xcite is a parameter constraining method, an upgrade to vanilla regularization (e.g. l 2 ). lrc is borrowed from stage two of ulmfit @xcite pre-training as a data independent method. finally, as a scaled back version of mtl we investigate experience replay (er), reintroducing data at a fixed scale from previous domains during pre-training. furthermore we explore data selection approaches to improve efficiency for both er, and ewc. our goal is to understand the trade-offs across these models in terms of resources and setup. to this end we conduct experiments across multiple domain shifts while pre-training. to evaluate the efficacy of the methods we use downstream finetuning tasks in the domains we study. to better understand how knowledge across domains is transferred, we perform layer-wise analysis and observe that outer layer are the most transferable. our contributions are as follows foot_0 : ## continual learning we empirically study three forms of mitigation for catastrophic forgetting. constraint based training in the form of ewc and learning rate control, and experience replay. ## catastrophic forgetting in language modeling we motivate our own experiments by first exploring off-the-shelf models to get a sense of the problem. to this end we fine tuned a bert base architecture on all nine glue @xcite tasks. these were compared directly against biobert, which has been further trained on full pubmed articles. as reported in ## experimental details we first cover the data domains, fine-tuning tasks, and general modeling setup used in both our heuristic search as well as our main experiments in section 6.2.2. ## data selection methods data selection is an important component of both supervised, and unsupervised training. in our case, there is an abundance of data to build both the fisher matrix, as well as the replay buffer. to do this efficiently for ewc and er we need to severely restrict the number of datapoints we utilize. for example a mere 1.0% of generic pre-training data makes up over 400k segments. we require this subset to be comprehensively representative of the domain. therefore, rather than randomly sampling data, we can use model generated features to induce better coverage of previous domains. ## mitigation of catastrophic forgetting we provide results for one and two stage domain shifts as given by fine-tuning tasks. again, we apply mitigation only to pre-training and express our model performance by using them to fine-tune downstream tasks. ## analysis to further understand learning and forgetting across different mitigation strategies, we conduct analyses to investigate how different layers of the model adapt to in-domain pre-training, whether the adaptation helps in transferring knowledge to downstream tasks, and how knowledge learned from in & out of domain data cooperates together. on bio-medical data. we compare roberta base (denoted as generic) against the pmc model (row 2 in we investigate if constraining the weights to a shared space is enough to produce a good overall model. we observed that without the fisher matrix, weight similarity between ewc and roberta base is lower than 0.2, which is confirmed by the low f 1 scores noted in section 6.2.2. this indicates that the fisher diagonal plays an important role in fluctuating variance. ## related work current work in catastrophic forgetting mitigation in nlp has been limited. howard and ruder (2018) introduced a multi stage training scheme for fine tuning lstm based universal language models (ulmfit). the authors proposed that current methods, rather than data, are ineffective and focused on learning rate control across layers, as well as modifying learning rate scheduling. a larger category of work deals with constraining model parameters to a latent space where they continue to capture previous tasks. initial work focused on model regularization and varying activations @xcite . @xcite provided a more sophisticated solution constraining weights individually termed elastic weight consolidation (ewc). we make use of both ewc and ulmfit and provide further technical detail in this paper. the final approach is focused on experience replay. using small samples of data from previous tasks coupled with local adaptation de masson d' @xcite demonstrate improvement in a lifelong learning training scheme. chaudhry et al. (2019) also explore lifelong learning by experimenting with updating the memory bank for experience replay. our work focuses on both of these techniques with the major difference being problem scale. many existing works apply these solutions on small networks whereas we experiment on architectures having several orders of magnitude more parameters. there has been a recent focus on more effective pre-training which focuses on narrowing the pretraining domain as we move closer towards finetuning. stilts @xcite and tanda @xcite use intermediate tasks (in a data rich domain) training to lower variance during target task fine-tuning. this intuition was also covered text model label conf. (i): entire social infrastructures in the icy far north where norilsk ground truth s-org is based depend on the company, and government has said that roberta base s-misc 0.609 expenditure could far outstrip norilsk 's debts. [norilsk] pmc s-misc 0.983 officials declined to comment. pmc+er s-org 1.000 (ii): president arafat's position is clear that such a meeting should ground truth s-loc come after successful negotiations so that the meeting would have roberta base s-per 0.998 positive results. especially since the [hebron] issue has not been pmc o 1.000 agreed yet and the crucial disputed issues have not been resolved. pmc+er s-loc 0.994 (iii): the committee said the italian club had violated regulations by ground truth s-org failing to inform feyenoord, with whom the player was under roberta base s-loc 0.815 contract. blinker was fined 75,000 swiss francs ($57,600) for pmc s-loc 1.000 failing to inform the english club of his previous commitment pmc+er s-org 1.000 to [udinese]. in the visio-linguistic domain by @xcite . finally @xcite work on mlm pretraining and provide conclusive evidence at scale of the works listed above. this last body of work, although dealing with pre-training is different from our work in that we study mitigation of domain forgetting, rather than reducing variance by adding intermediate domains or tasks to pre-training. ## conclusion in this work, we empirically investigated the existence of catastrophic forgetting in large language model pre-training. we further explored constraint and replay based mitigation techniques to close the performance gap between general and domain specific natural language tasks. we find that training a single model across multiple domains is possible. due to practical considerations, we would suggest using latent representation for data selection when working with a data dependent model such as er or ewc. when no previous data is available lrc provides a simple yet powerful solution for retaining prior domain knowledge. in the future work wish to explore more data independent methods such as lrc, for both speed and lack of data dependency, as well as manipulation of the decay w.r.t. what we have discovered from our layer-wise analysis.