Incremental Prompting: Episodic Memory Prompt for Lifelong Event Detection (2022)

Abstract: Lifelong event detection aims to incrementally update a model with new event types and data while retaining the capability on previously learned old types. One critical challenge is that the model would catastrophically forget old types when continually trained on new data. In this paper, we introduce Episodic Memory Prompts (EMP) to explicitly retain the learned task-specific knowledge. Our method adopts continuous prompt for each task and they are optimized to instruct the model prediction and learn event-specific representation. The EMPs learned in previous tasks are carried along with the model in subsequent tasks, and can serve as a memory module that keeps the old knowledge and transferring to new tasks. Experiment results demonstrate the effectiveness of our method. Furthermore, we also conduct a comprehensive analysis of the new and old event types in lifelong learning.

## introduction class-incremental event detection @xcite is a challenging setting in lifelong learning, where the model is incrementally updated on a continual stream of data for new event types while retaining the event detection capability for all the previously learned types. the main challenge of class-incremental event detection lies in the catastrophic forgetting problem, where the model's performance on previously learned types significantly drops after it is trained on new data. recent studies @xcite have revealed that replaying stored samples of old classes can effectively alleviate the catastrophic forgetting issue. however, simply finetuning the entire model on the limited stored samples may result in overfitting, especially when the model has a huge set of parameters. how to ef-fectively leverage the limited stored examples still remains an important question. prompt learning, which is to simply tune a template-based or continuous prompt appended to the input text while keeping all the other parameters freezed, has recently shown comparable or even better performance than fine-tuning the entire model in many nlp tasks @xcite @xcite . it is especially flavored by lifelong learning since it only tunes a small amount of parameters. however, it is still non-trivial to equip prompts with the capability of retaining acquired knowledge and transferring to new tasks in the class-incremental setting. in this work, we propose an incremental prompting framework that introduces epsodic memory prompts (emp) to store and transfer the learned type-specific knowledge. at each training stage, we adopt a learnable prompt for each new event type added from the current task. the prompts are initialized with event type names and fine-tuned with the annotations from each task. to encourage the prompts to always carry and reflect typespecific information, we entangle the feature representation of each event mention with the typespecific prompts by optimizing its type distribution over them. after each training stage, we keep the learned prompts in the model and incorporate new prompts for next task. in this way, the acquired task-specific knowledge can be carried into subsequent tasks. therefore, our emp can be considered as a soft episodic memory that preserves the old knowledge and transfers it to new tasks. our method does not require task identifiers at test time, which enables it to handle the challenging class-incremental setting. our contributions can be summarized as follows: ## problem formulation given an input text x 1:l and a set of target spans {(x i , x j )} from it, an event detection model needs to assign each target span with an event type in the ontology or label it as other if the span is not an event trigger. for class-incremental event detection, we aim to train a single model f θ on a sequence of t tasks {d 1 , ..., d t } that consist of non-overlapping event type sets {c 1 , ..., c t } foot_1 . in each t-th task, the model needs to classify each mention to any of the types that have seen so far o @xmath0... c t . the training instances in each task d t consist of tuples of an input text x t 1:l , a target span xt , and its corresponding label y t where y @xmath1. for convenience, the notations are for the t-th training stage by default unless denoted explicitly in the following parts of the paper. ## experiments and discussions experiment settings we conduct experiments on two benchmark datasets: ace05-en @xcite and maven @xcite , and construct the class-incremental datasets following the oracle negative setting in @xcite . we divided the ontology into 5 subsets with distinct event types, and then use them to constitute a sequence of 5 tasks denoted as d 1:5 . we use the same partition and task order permutations in @xcite . during the learning process from d 1 to d 5 , we constantly test the model on the entire test set (which contains the whole ontology) and take the mentions of unseen event types as negative instances. more implementation details, including parameters, initialization of prompts as well as baselines are shown in appendix a. baselines we consider the following baselines for comparison: (1) bert-ed: simply trains the bert based event detection model on new tasks without prompts, experience replay or knowledge distillation. it's the same as the span-based event detection baseline in section 3.1. (2) kcn @xcite : use a prototype-based example sampling strategy and hierarchical distillation. as the original approach studied a different setting, we adapt their prediction-level and feature-level distillation as the baseline. (3) kt @xcite : transfer knowledge between old types and new types in two directions. (4) icarl* (rebuffi et al., 2017): use nearest-mean-of-exemplars rules to perform classification combined with knowledge distillation. icarl adopts different strategies for classification, experience replay, and distillation. we thus directly report the result in @xcite for reference. (5) eeil @xcite : use an additional finetuning stage on the balanced dataset. (6) bic @xcite : use a bias correction layer after the classification layer. (7) upperbound: trains the same model on all types in the datasets jointly. for icarl, eeil, and bic, we use the same implementation in @xcite . for fair comparison, our approach and all baselines (except for the upperbound baseline) are built upon kcn and use the same experience replay and knowledge distillation strategies described in section 3.2. we set the exemplar buffer size as 20, and allow one exemplar instance to be used in each training batch instead of the whole memory set. note that this replay setting is different from the one in @xcite , where we allow much less frequent exemplar replay, and thus our setting is more efficient, challenging, and realistic. ## conclusion we propose a novel episodic memory prompting (emp) framework for class-incremental event detection. during each training stage, emp learns type-specific knowledge via a continuous prompt for each event type. the emps trained in previous tasks are kept in the model, such that the acquired task-specific knowledge can be transferred into the following new tasks. experimental results validate the effectiveness of our method comparing with competitive baselines. our extensive analysis shows that by employing emps, both event detection itself and the incremental learning capability of our approach are significantly improved. ji, and shih-fu chang. 2017. improving event extraction via multimodal integration. in proceedings of the 25th acm international conference on multimedia, pages 270-278. kang zhao, hua xu, jiangong yang, and kai gao. 2022. consistent representation learning for continual relation extraction. corr, abs/2203.02721. qi zhu, bing li, fei mi, xiaoyan zhu, and minlie huang. 2022. continual prompt tuning for dialog state tracking. corr, abs/2203.06654.