Lifelong Event Detection with Embedding Space Separation and Compaction (2024)

Abstract: To mitigate forgetting, existing lifelong event detection methods typically maintain a memory module and replay the stored memory data during the learning of a new task. However, the simple combination of memory data and new-task samples can still result in substantial forgetting of previously acquired knowledge, which may occur due to the potential overlap between the feature distribution of new data and the previously learned embedding space. Moreover, the model suffers from overfitting on the few memory samples rather than effectively remembering learned patterns. To address the challenges of forgetting and overfitting, we propose a novel method based on embedding space separation and compaction. Our method alleviates forgetting of previously learned tasks by forcing the feature distribution of new data away from the previous embedding space. It also mitigates overfitting by a memory calibration mechanism that encourages memory data to be close to its prototype to enhance intra-class compactness. In addition, the learnable parameters of the new task are initialized by drawing upon acquired knowledge from the previously learned task to facilitate forward knowledge transfer. With extensive experiments, we demonstrate that our method can significantly outperform previous state-of-the-art approaches.

## introduction event detection (ed) aims to detect the event type of trigger words in a given sentence, e.g., extracting the event type injure from the trigger word scalded in text "he was scalded by hot water". traditional ed methods typically consider a fixed pre-defined set of event types @xcite @xcite . however, as the environment and data distributions change in real scenarios, the model might face challenges in handling rapidly emerging new types @xcite . a more practical setting is lifelong event detection or led @xcite , where the model learns event knowledge from a sequence of tasks with different sets of event types. in led, the model is expected to retain and accumulate knowledge when learning new tasks, which is challenging due to catastrophic forgetting @xcite of previously acquired knowledge. existing methods @xcite for mitigating forgetting in led typically maintain a memory that saves a few key samples of previous tasks, which are then combined with new data for training. recently, @xcite introduce episodic memory prompts (emp) that leverages soft prompts to remember learned event types, achieving state-ofthe-art performance on led. despite its effectiveness, emp has two key limitations. first, simply combining new data and memory samples for training can still result in forgetting as the feature distribution of new data might overlap with the previously learned embedding space (see @xcite . second, it may overfit on a few memory samples after frequent replays rather than effectively retaining learned patterns. to address the above limitations of emp, in this paper, we introduce a novel method based on embedding space separation and compaction (esco) for led. in particular, we propose a margin-based loss that forces the feature distribution of new event types away from the learned embedding space to alleviate forgetting. inspired by @xcite , we introduce a memory calibration mechanism to encourage memory data to be close to its prototype to avoid overfitting on the few memory samples. in addition, the learnable parameters of the new task are initialized using those of the previously learned task to facilitate forward knowledge transfer, which is as important for lifelong learning as preventing forgetting @xcite . the empirical results show that our method significantly outperforms previous state-of-the-art approaches. in summary, our main contributions are: ## problem formulation led involves learning from a stream of event detection tasks @xmath0, . . . , t 𝑛 ), where each task t 𝑘 has its own training set d 𝑘 train , validation set d 𝑘 valid , and test set d 𝑘 test . for every input text 𝑥 𝑖 in d 𝑘 , it contains a set of target spans {𝑥 𝑖 𝑡 } and their corresponding labels 𝑦 𝑖 𝑡 which belong to the event type set c 𝑘 of task t 𝑘 . note that the event type sets of different tasks are non-overlapping. after the training on d 𝑘 train , the model is expected to perform well on all the 𝑘 tasks that it has learned and will be evaluated on the combined test set d𝑘 @xmath1. during the learning, a memory module m which stores a few key samples of previous tasks is maintained to overcome the forgetting problem. ## embedding space separation and compaction when learning a new task t 𝑘 , following @xcite , we first initialize a set of soft prompts p 𝑘 = {𝑝 𝑘 1 , ..., 𝑝 𝑘 | c 𝑘 | } where c 𝑘 is the event type set of t 𝑘 . the accumulated prompts q 𝑘 = [p 1 , ..., p 𝑘 ] until time step 𝑘 are then combined with the input text 𝑥 𝑖 to obtain the contextual representations using a frozen bert @xcite : to predict the event type of the span 𝑥 𝑖 𝑡 , we concatenate the representations corresponding to the start and end token and obtain the logits over all learned types through a feed-forward network (ffn) as well as a linear layer: after learning the previous task t 𝑘-1 , we select the top-𝑙 most informative training examples for each event type in c 𝑘-1 using the herding algorithm @xcite , which are then saved in the memory module m for replay to mitigate forgetting. similar as eq. 3, the training objective for memory replay when learning t 𝑘 is: however, the simple combination of l new and l mem can still result in substantial forgetting of acquired knowledge due to the potential overlap between the feature distribution of new event types and the previously learned embedding space (see appendix a.1). to ensure that the new feature distribution is away from the learned embedding space, we design a margin-based loss, which decreases the similarity scores between new samples and prototypes (see eq. 8 for the calculation of prototypes) of learned event types: as the size of memory m is typically small, the model is prone to overfit on the few memory samples after frequent replays, making learned distributions distorted. to effectively recover from distorted learned distributions, we introduce a memory calibration mechanism inspired by @xcite . specifically, for each memory sample in m, we encourage it to be close to its corresponding prototype to improve the intra-class compactness of learned distributions. more formally, after learning t 𝑘 and selecting memory data for t 𝑘 , we use the memory m to calculate prototypes of all learned event types in c 𝑘 . specifically, for each event type 𝑒 𝑗 in c 𝑘 , we obtain its prototype e 𝑗 by averaging the span representations of all samples labeled as 𝑒 𝑗 in m as follows: ## conclusion in this work, we have introduced embedding space separation and compaction (esco) for lifelong event detection (led). esco imposes novel feature constraints in the embedding space to alleviate forgetting and overfitting problems. it initializes the learnable parameters for the new task by inheriting those from the previously learned task to facilitate forward knowledge transfer. with extensive experiments and analysis, we have demonstrated that esco significantly outperforms previous methods. for future work, we are interested in exploring esco in a meta-learning paradigm for led.