Recyclable Tuning for Continual Pre-training (2023)

Abstract: Continual pre-training is the paradigm where pre-trained language models (PLMs) continually acquire fresh knowledge from growing data and gradually get upgraded. Before an upgraded PLM is released, we may have tuned the original PLM for various tasks and stored the adapted weights. However, when tuning the upgraded PLM, these outdated adapted weights will typically be ignored and discarded, causing a potential waste of resources. We bring this issue to the forefront and contend that proper algorithms for recycling outdated adapted weights should be developed. To this end, we formulate the task of recyclable tuning for continual pre-training. In pilot studies, we find that after continual pre-training, the upgraded PLM remains compatible with the outdated adapted weights to some extent. Motivated by this finding, we analyze the connection between continually pre-trained PLMs from two novel aspects, i.e., mode connectivity, and functional similarity. Based on the corresponding findings, we propose both an initialization-based method and a distillation-based method for our task. We demonstrate their feasibility in improving the convergence and performance for tuning the upgraded PLM. We also show that both methods can be combined to achieve better performance.

## introduction the emergence of pre-trained language models (plms) has revolutionized the entire field of natural language processing (nlp) @xcite . through downstream adaptation, plms effectively stimulate the knowledge acquired during pre-training and achieve remarkable success in various downstream tasks @xcite @xcite . such adaptation can be achieved by either full-parameter fine-tuning or parameter-efficient tuning @xcite , and the latter enables learning lightweight adapted modules for downstream tasks. currently, a de facto paradigm for handling nlp tasks has been formed, dividing practitioners into two groups: (1) upstream suppliers, who pre-train plms on taskagnostic data and release them on public platforms, e.g., huggingface @xcite , and (2) downstream consumers, who download the plm and conduct personalized adaptation using taskspecific data. the corresponding adapted weights might then be shared with third parties via platforms such as adapterhub @xcite . in real-world scenarios, plms may constantly get upgraded and released by the supplier. correspondingly, the customer-side compatible update of adapted weights becomes necessary. continual pre-training @xcite is a typical scenario where plms continually acquire fresh knowledge from growing data and gradually get upgraded. before an upgraded plm is released, consumers may have tuned the original plm for various tasks and stored the adapted weights. however, when tuning the upgraded plm, these outdated adapted weights will typically be ignored and discarded. this can lead to a loss of knowledge about downstream tasks encapsulated in the outdated weights, as well as a potential waste of computational resources. in this paper, we bring this issue to the forefront and argue that proper algorithms for recycling outdated adapted weights should be developed. to this end, we formulate the task of recyclable tuning for continual pre-training, which is illustrated in figure 1 . due to the parameter change during continual pre-training, one potential concern for recycling outdated adapted weights is their mismatch with the upgraded plm. however, our pilot studies reveal that directly applying the outdated weights to the upgraded plm yields substantial performance improvements as compared to zero-shot inference of the plm. this shows that the upgraded plm remains compatible with the outdated weights to some extent, indicating a close connection between continually pre-trained plms. intuitively, such a connection provides a strong basis for our assertion that outdated weights are recyclable and useful. to uncover hints for solving our task, we further investigate such a connection from two aspects: (1) linear mode connectivity @xcite . we demonstrate that after adapting both the upgraded plm and the original plm to the same task, linearly interpolating the parameters of both adapted models could produce a series of checkpoints with high task performance (low loss). such a property indicates a close parametric connection of both plms in the loss landscape; (2) functional similarity. after adapting both plms to the same task, we observe that their corresponding attention heads exhibit similar patterns given the same input. such representational proximity implies that both plms own similar functionalities during text processing. both analyses above demonstrate the close connections between continually pre-trained plms. based on the corresponding findings, we propose two methods for recyclable tuning: in a nutshell, these results highlight the practical benefits of recyclable tuning and point to an important future direction in sustainable nlp. ## related work continual pre-training. conventionally, plms are trained on static data, ignoring that streaming data from various sources could continually grow. continual pre-training requires plms to accumulate new knowledge in a continual manner @xcite , meanwhile alleviating the catastrophic forgetting problem. prior works in this field focus on building benchmarks and analyses @xcite . later works explored the applicability of traditional continual learning algorithms under this setting @xcite . recent efforts were also spent on continual pre-training in a computationally efficient way @xcite . previous works focus on improving the capabilities of plms during pre-training from the standpoint of upstream suppliers. instead, we shift the focus to downstream adaptation from the perspective of customers. we highlight a previously overlooked issue of the incompatibility between upgraded plms and the existing adapted weights. for the first time, we examine the connections between continually pre-trained models and demonstrate the potential benefits of recycling outdated weights. knowledge transfer for plms. transfer learning for plms has gained increasing attention recently. some works study task-level transferability for an individual plm and find that fine-tuning on certain source tasks conduces to the performance on similar target tasks @xcite @xcite . differently, we also study cross-task knowledge transfer for two different plms under the continual pre-training scenario ( § 5.1). besides, researchers also investigate cross-model knowledge transfer. they try to recycle lightweight adapted weights of the same task between two independently pre-trained plms, e.g., plms with distinct data @xcite . as we would show later, unlike independently trained plms, continually pre-trained plms are guaranteed close connections. this distinction determines our setting is unique to previous works and may require different solutions. ## problem formulation continual pre-training. following qin et al. (2022c), we simulate the scenario where new data from 4 domains is gathered sequentially, i.e., biomedical papers (bio, d 1 ) @xcite , amazon reviews (rev, d 2 ) @xcite , computer science papers (cs, d 3 ) @xcite , and news articles (ns, d 4 ) @xcite . starting from the official roberta base @xcite (denoted as m 0 ), we continually pre-train m 0 on 4 domains. for each domain, we set the pre-training steps to 12.5k and the batch size to 2048. denote m i as the plm that finishes training on d i , and m i (t) as the plm that starts from m i-1 and is trained on d i for t steps. we assume the suppliers only release the plm that finishes training on each domain, i.e., {m 1 , • • • , m 4 } are developed and released. the pre-training details are described in appendix d.1. ## empirical analysis we first investigate the compatibility of outdated weights and the upgraded plm ( § 4.1), then we explore the (1) parametric connections and (2) representational connections of continually pre-trained plms from two aspects: (1) linear mode connectivity ( § 4.2) and (2) functional similarity ( § 4.3). the implementation details are left in appendix d.2. ## methods and experiments based on the findings in § 4, we propose two ways to explore the practical benefits of recyclable tuning: initialization-based method ( § 5.1) and distillation-based method ( § 5.2). the training details of this section are discussed in appendix d.3. ## discussion training-free weight recycling. both methods proposed in § 5 necessitate tuning the upgraded plm. such a process often relies on abundant computational costs and may be infeasible practically. given the close connections among continually pretrained plms, we contend that weight recycling can be realized without training. as a preliminary exploration, we show in appendix b that it is possible to learn a cross-task generalizable projection to directly upgrade the outdated weights and make them compatible with the new plm. upgrading outdated weights using such a projection requires far fewer computations (< 0.002‰) and still achieves satisfactory performance. ## conclusion in this paper, we formulate the task of recyclable tuning for continual pre-training. we conduct empirical analyses for this task through the lens of model compatibility, linear mode connectivity, and functional similarity. inspired by the corresponding findings, we explore the practical benefits of recyclable tuning through parameter initialization and knowledge distillation. we also envision our setup to serve as the testbed for other topics, e.g., crossmodel knowledge transfer and continual learning.