Prompts Can Play Lottery Tickets Well: Achieving Lifelong Information Extraction via Lottery Prompt Tuning (2023)

Abstract: Thanks to the recent success of Pre-trained Language Models (PLMs), it has become a promising research direction to develop a universal model (UIE) that can solve all typical information extraction tasks within one generative framework. Nonetheless, in real-world scenarios of UIE applications, new data of different IE tasks and domains usually come in a stream over time. A desirable UIE system should be capable of continually learning new tasks without forgetting old ones, thereby allowing knowledge and functionalities expansion without re-training the whole system. In this paper, we study the UIE system under a more challenging yet practical scenario, i.e., “lifelong learning” settings, to evaluate its abilities in three aspects, including knowledge sharing and expansion, catastrophic forgetting prevention, and rapid generalization on few-shot and unseen tasks. To achieve these three goals, we present a novel parameter- and deployment-efficient prompt tuning method namely Lottery Prompt Tuning (LPT).LPT freezes the PLM’s parameters and sequentially learns compact pruned prompt vectors for each task leveraging a binary prompt mask, while keeping the prompt parameters selected by the previous tasks insusceptible. Furthermore, we use a simple yet effective method to perform mask selection and show the powerful transferability of Lottery Prompts to novel tasks. Extensive experiments demonstrate that LPT consistently sets state-of-the-art performance on multiple lifelong learning settings of UIE, including task-incremental setting on seen tasks, few-shot adaptation, and zero-shot generalization on novel tasks.

## introduction information extraction (ie) is one of the fundamental tasks in natural language processing (nlp), which aims to extract the desired structural information from unstructured texts (andersen et al., 1992; @xcite @xcite . previous ie research mostly focuses on one specific ie task @xcite @xcite and designs different model architectures @xcite @xcite to tackle different tasks. to facilitate knowledge sharing between different tasks, various efforts have been paid for unifying all ie tasks with one model structure @xcite @xcite . most recently, @xcite unify general ie tasks in a generative way with a text-to-structure framework (uie), which proves that universally modeling various ie tasks can better learn general knowledge from varying data sources. nonetheless, current work usually assumes the accessibility of training data for every task. in many real-world scenarios, as shown in figure 1 , the training data are often streamed, and the ie systems are required to identify new mention spans or semantic relations to support new domains and functionalities, which can be formulated as the paradigm of lifelong learning. the ability to accumulate knowledge continually is crucial for the quick deployment of uie systems based on plms, which allows the system to add new domains and functionalities over time without incurring the high cost of re-training the whole system each time. in addition, considering that humans can acquire new knowledge from a few examples @xcite , it is expected for the models to generalize well on novel tasks with few-shot data or even no data. motivated by this, our work aims to address these more challenging yet practical issues by proposing a lifelong learning setup for uie. in this setup, the system sequentially learns over multiple ie tasks (potentially of different task types and varying domains) one by one. then it will be evaluated to preserve its performance on solving previously seen tasks, and generalize well to novel tasks with few examples or even no examples. we cover two conventional properties of lifelong learning @xcite , i.e., catastrophic forgetting prevention (cf) and knowledge transfer (kt), while in our setup, the evaluation of kt extends to the novel tasks. in nlp community, large pre-trained language models (plms) have been widely applied in many downstream tasks. in order to lower computation and storage costs, recent popular lifelong learning techniques @xcite @xcite try to solve the cf and kt leveraging parameter-efficient fine-tuning (peft) methods @xcite . in this work, we inherit this wisdom and also focus on parameter-efficient methods for lifelong learning. inspired by the lottery ticket hypothesis and the efficiency of prompt tuning, we propose a novel framework for lifelong uie, named lottery prompt tuning (lpt). specifically, we adopt an encoder-decoder model architecture @xcite and re-frame all types of ie tasks into a text-to-structure format @xcite . first, we prepend a sequence of continuous prompt vectors to the input, which is shared across tasks. to continually learn a new ie task, we simultaneously learn the prompt vectors together with a task-aware binary prompt mask. the task-aware mask is devoted to pruning the shared prompt vectors and producing an optimal task-specific pruned prompt, i.e., lottery prompt. to provide a pruning criterion for finding the lottery prompt online, we introduce a separate set of learnable parameters serving as the importance scores, which have the same shapes as the soft prompts. hence, the lottery prompt can be easily found by selecting the parameters with the top-k% importance scores online, without iterative retraining and pruning procedure. to facilitate the forward knowledge transfer when learning a new task, the lottery prompt is permitted to selectively reuse the learned prompt parameters for the former tasks. besides, the proposed lpt eliminates catastrophic forgetting and negative transfer by freezing the prompt parameters for the previous tasks during back-propagation. in the whole learning process, the plm is kept frozen to maintain general knowledge. during inference, the same model can handle different tasks by inputting different lottery prompts, which is friendly for deployment. we show that our proposed framework effectively outperforms state-of-the-art baselines on lifelong learning for uie in terms of catastrophic forgetting prevention and knowledge transfer. moreover, lpt closes the gap between continual learning and multi-task learning. the efficacy of the proposed modules is thoroughly studied both empirically and analytically. in summary, this work makes three key contributions: ## conclusions in this paper, we study a lifelong learning paradigm for uie systems, which we regard as an important step towards general ie intelligence. we propose a novel parameter-efficient framework, i.e., lottery prompt tuning (lpt), to achieve positive knowledge transfer, catastrophic forgetting prevention, and rapid generalization. experimental results validate the capability of our method on three settings.