Towards Anytime Fine-tuning: Continually Pre-trained Language Models with Hypernetwork Prompts (2023)

Abstract: Continual pre-training has been urgent for adapting a pre-trained model to a multitude of domains and tasks in the fast-evolving world. In practice, a continually pre-trained model is expected to demonstrate not only greater capacity when fine-tuned on pre-trained domains but also a non-decreasing performance on unseen ones. In this work, we first investigate such anytime fine-tuning effectiveness of existing continual pre-training approaches, concluding with unanimously decreased performance on unseen domains. To this end, we propose a prompt-guided continual pre-training method, where we train a hypernetwork to generate domain-specific prompts by both agreement and disagreement losses. The agreement loss maximally preserves the generalization of a pre-trained model to new domains, and the disagreement one guards the exclusiveness of the generated hidden states for each domain. Remarkably, prompts by the hypernetwork alleviate the domain identity when fine-tuning and promote knowledge transfer across domains. Our method achieved improvements of 3.57% and 3.4% on two real-world datasets (including domain shift and temporal shift), respectively, demonstrating its efficacy.

## introduction pre-trained language models (lms), such as gpt-3 @xcite and bert @xcite , have revolutionized a wide spectrum of downstream natural language processing (nlp) tasks. being initially pre-trained on a vast unlabeled corpus (e.g., c 0 in fig. 1 ), unfortunately, they struggle to keep up to date with language evolution (e.g., emerging internet slang, expanded meaning of "omicron") and domain shift (e.g., electronic health records for medical diagnosis). continual pre-training methods @xcite have recently emerged to address it by continually adapting an lm to a sequence of domains (e.g., t domains in fig. 1 ). two major lines of existing approaches, including knowledge distillation @xcite and parameter isolation @xcite , make strides toward (1) maximizing the adaptability, i.e., the performance of an lm (e.g., b 2 in fig. 1 ) when fine-tuning it onto the domain where it is pre-trained (e.g., d 2 in fig. 1 ), and (2) avoiding catastrophic forgetting (cf), which is measured by the fine-tuned performance of an lm (e.g., b 2 in fig. 1 ) on the already pre-trained domains (e.g., d 1 in fig. 1 ). beyond the above two criteria, in practice, a continually pre-trained lm is also anticipated to offer non-decreasing generalization capability on unseen domains. as illustrated in fig. 1 , it is likely that the unlabeled corpus for the domain of interest (e.g., electronic health records as d t ) remains inaccessible to an lm (e.g., b 2 ) beforehand, while this lm should be superior or at least on par with its preceding models (e.g., b 1 ) on the t -th domain. on f )-(h) are by the naively continual pre-training method (ncl), das @xcite , and ours. detailed settings are available in sec. 5.2. in this paper, we first conduct an empirical investigation of existing pre-training approaches under anytime fine-tuning (see fig. 2 ) and identify the following two prominent unresolved research questions. (1) parameter-efficient pre-training, such as training adapters @xcite and prompts @xcite only for each individual domain, does not even contribute greater adaptability than that before pre-training (i.e., evidenced in negative diagonal values of to address the above issues, we propose a hypernetwork prompt guided continual pre-training method (namely hprompt-cpt foot_0 ) that strikes a balance between forgetting, adaptability, and generalization. first, inspired by recent success of prompt engineering paired with full finetuning in domain adaptation @xcite , we introduce the hnet-prompt module consisting of a hypernetwork to automatically generate domain-specific prompts without handcrafted engineering. different from parameterefficient pre-training that train prompts only, we optimize both the hypernetwork and the full lm so as to fully adapt to the current domain. an added benefit of hypernetwork prompts is that they eliminate the reliance on the domain identity to pinpoint prompts when fine-tuning. second, we maximally preserve the generalization while mitigating cf of a continually pre-trained lm via the agreement and disagreement losses. we prompt the previous and current lm with a random prompt that simulates generic or learned domains and introduce the agreement loss to enforce consistency between their predictions to avoid forgetting while preserving model plasticity on other prompts. on the other hand, the disagreement loss promotes the exclusiveness of generated hidden states for the current domain, thus minimizing interference to the established knowledge and encouraging generalization during fine-tuning through diverse domain knowledge. noteworthy, the hypernetwork also favors knowledge generalization, compared to disparate prompts of different domains. ## related work continual learning (cl) focuses on the problem of sequential learning from a stream of data that comes in different distributions. it has achieve a great success in computer vision @xcite , natural language pro-cessing @xcite , and data mining @xcite . in this paper, we focus on one of the important aspects, continual pre-training and present recent progresses below. more related works are given in appendix a. ## preliminaries our language model b is constructed using the roberta architecture @xcite , which is based on a bi-directional transformer structure. lm takes a text sentence x 1:@xmath0, x 2 , ..., x t ] as input and encodes it into a contextual embedding @xmath1, h 2 , ..., h t ] = b(x 1:t ). ## method a successful algorithm of continual pre-training for anytime fine-tuning should meet the following requirements: (1) effective adaptation to the current domain and capturing more domain knowledge, (2) strong generalization to tasks in unseen domains, and (3) minimal catastrophic forgetting of previ-ously learned knowledge. to achieve this, we propose a framework, dubbed hprompt-cpt, which consists of two components: the hnet-prompt module and agreement and disagreement losses. the overview is presented in fig. 3 . ## experiment in this section, we conduct experiments on two benchmarks to investigate the adaptability, generalization, and degree of forgetting of hprompt-cpt. ## conclusion this paper introduces hprompt-cpt, a novel prompt-guided continual pre-training method towards anytime fine-tuning, which enables better performance when fine-tuned on seen and unseen domains. by training a hypernetwork to generate domain-specific prompts with agreement and disagreement losses, it results in (i) greater capacity on pre-trained domains by learning domain knowledge with generated prompts while preserving previous knowledge with random prompts, (ii) improved performance on unseen domains by retaining model plasticity with agreement loss and the ability of knowledge transfer with hypernetwork, and (iii) no need for domain-id during fine-tuning. we set a new sota on both well-established benchmark and a temporal shift benchmark.