ELLE : Efficient Lifelong Pre-training for Emerging Data (2022)

Abstract: Current pre-trained language models (PLM) are typically trained with static data, ignoring that in real-world scenarios, streaming data of various sources may continuously grow. This requires PLMs to integrate the information from all the sources in a lifelong manner. Although this goal could be achieved by exhaustive pre-training on all the existing data, such a process is known to be computationally expensive. To this end, we propose ELLE, aiming at efficient lifelong pre-training for emerging data. Specifically, ELLE consists of (1) function preserved model expansion, which flexibly expands an existing PLM’s width and depth to improve the efficiency of knowledge acquisition; and (2) pre-trained domain prompts, which disentangle the versatile knowledge learned during pre-training and stimulate the proper knowledge for downstream tasks. We experiment ELLE with streaming data from 5 domains on BERT and GPT. The results show the superiority of ELLE over various lifelong learning baselines in both pre-training efficiency and downstream performances. The codes are publicly available at https://github.com/thunlp/ELLE .

## introduction pre-trained language models (plm) have broken the glass ceiling for various natural language processing (nlp) tasks @xcite @xcite . however, most of the existing plms are typically trained with a static snapshot of the web information, ignoring that in real-world scenarios, streaming data from various sources may continuously grow, e.g., the gatherings of literary works @xcite , news articles @xcite and science papers @xcite . in addition, the distribution of incoming data may also vary over time. this requires plms to continually integrate the information from all the sources to grasp the versatile structural and semantic knowledge comprehensively, so that plms could utilize the proper knowledge to boost the performance in various downstream tasks. a simple yet effective way to integrate all the information is to pre-train plms on all the existing data exhaustively. however, such a process is computationally expensive @xcite , especially under the information explosion era when tremendous data is continually collected. this leaves us an important question: with limited computational resources, how can we efficiently adapt plms in a lifelong manner? we formulate it as the efficient lifelong pre-training problem. similar to conventional lifelong learning, plms are expected to continually abosrb knowledge from emerging data, and in the meantime, mitigate the catastrophic forgetting (mccloskey and cohen, 20p) on previously learned knowledge. in addition, efficient lifelong pre-training poses two new challenges: (1) efficient knowledge growth. when the overall data scale accumulates to a certain magnitude, packing more knowledge into a fixed-sized plm becomes increasingly hard, which significantly impacts the efficiency of plm's knowledge growth. this is because larger plms show superior sample efficiency and training efficiency over their smaller counterparts @xcite due to overparameterization @xcite . that is, larger plms learn knowledge in a more efficient way. therefore, timely model expansions are essential for efficient knowledge growth; (2) proper knowledge stimulation. during pre-training, various knowledge from all domains is packed into plms hastily. however, a certain downstream task may largely require the knowledge from a specific domain. thus it is essential for plms to disentangle different kinds of knowledge and properly stimulate the needed knowledge for each task. in this paper, we propose elle, targeting at efficient lifelong pre-training for emerging data. specifically, (1) to facilitate the efficiency of knowledge growth, we propose the function preserved model expansion to flexibly expand an existing plm's width and depth. in this way, we increase plm's model size and thus improve its training efficiency. before being adapted to a new domain, the expanded plm performs a function recovering warmup to regain the functionality of the original plm; (2) for proper knowledge stimulation, we pre-implant domain prompts during pre-training to prime the plm which kind of knowledge it is learning. therefore, versatile knowledge from multiple sources can be disentangled. during downstream fine-tuning, we could further utilize these implanted prompts and manipulate the plm to stimulate the proper knowledge for a specific task. to demonstrate the effectiveness of elle, we simulate the scenario where streaming data from 5 domains sequentially comes. we pre-train two typical plms (bert and gpt) and expand their model sizes each time when the new data is available. we experiment when the number of parameters is sequentially grown from both 30m to 125m and 125m to 355m. the experimental results show the superiority of elle over multiple lifelong learning baselines in both pre-training efficiency and downstream task performances. in addition, we conduct sufficient experiments to verify the effectiveness of each component of elle. in general, we provide a promising research direction and hope this work could inspire more future attempts towards efficient lifelong pre-training. efficient pre-training in nlp. many attempts have been made towards improving the efficiency of pre-training, such as designing novel pretraining tasks @xcite , model architectures @xcite , optimization algorithms @xcite and parallel architectures @xcite . until recently, researchers propose to "back distill" the knowledge from existing plms to accelerate large plms' pre-training @xcite . another line of work proposes progressive training to dynamically expand an existing plm's size through parameter recycling @xcite @xcite . however, these methods typically focus on training plms on one static corpus, and thus cannot be directly applied to our lifelong pre-training setting. ## analysis in this section, we conduct analyses to investigate the effect of elle's components. we follow the setting in § 4 by choosing bert l6_d384 as the initial model and continually growing it to bert l12_d768 . specifically, we investigate the effect of (1) width expansion (we), (2) depth expansion (de), (3) function recovering warmup (frw), (4) the random noises added into the newly constructed parameters during model expansion (δ n ) and ( 5 ) the pre-trained domain prompts (pt). we test elle under different combinations of the above components and compare the results. the experimental results of pre-training and downstream tasks are summarized in ## conclusion in this paper, we present the efficient lifelong pretraining problem, which requires plms to continually integrate the information from emerging data efficiently. to achieve our goal, we propose elle and progressively expand plms to acquire knowledge efficiently and mitigate the knowledge forgetting. we also pre-implant domain prompts during pre-training and use them to stimulate the needed knowledge for downstream tasks. the experimental results show the superiority of elle over various lifelong learning baselines in both pre-training efficiency and downstream performances.