S harp S eq: Empowering Continual Event Detection through Sharpness-Aware Sequential-task Learning (2024)

Abstract: Continual event detection is a cornerstone in uncovering valuable patterns in many dynamic practical applications, where novel events emerge daily. Existing state-of-the-art approaches with replay buffers still suffer from catastrophic forgetting, partially due to overly simplistic objective aggregation. This oversight disregards complex trade-offs and leads to sub-optimal gradient updates, resulting in performance deterioration across objectives. While there are successful, widely cited multi-objective optimization frameworks for multi-task learning, they lack mechanisms to address data imbalance and evaluate whether a Pareto-optimal solution can effectively mitigate catastrophic forgetting, rendering them unsuitable for direct application to continual learning. To address these challenges, we propose **SharpSeq**, a novel continual learning paradigm leveraging sharpness-aware minimization combined with a generative model to balance training data distribution. Through extensive experiments on multiple real-world datasets, we demonstrate the superior performance of SharpSeq in continual event detection, proving the importance of our approach in mitigating catastrophic forgetting in continual event detection.

## introduction event detection (ed) is one of the fundamental topics in nlp @xcite and aims to classify an event trigger into a predefined event type in an established ontology. continual event detection (ced), however, introduces a distinct challenge as it deals with a continuously evolving ontology that accommodates previously unseen event types as new data emerges. an effective ced system requires robust mechanisms that not only enable the identification of novel events but also mitigate the phenomenon of catastrophic forgetting, where the detection performance on previously encountered events deteriorates when new events are introduced. most state-of-the-art methodologies for effective continual event detection @xcite @xcite are built upon memory-based techniques from continual learning @xcite @xcite . these techniques store a fraction of previously learned data in a replay buffer, allowing the model to reinforce its performance on past tasks while acquiring new knowledge. however, when employing memory-based techniques in continual learning, the management of multiple objectives associated with previous and current tasks becomes crucial. naively aggregating these objectives by simple summation overlooks the inherent, complicated trade-offs involved. thus, a more sophisticated strategy is imperative to address the challenge of multiple-objective optimization (moo) for memory-based mechanisms. in this context, gradient-based frameworks for moo, aiming to find a solution on the pareto front @xcite , have emerged as some of the most successful approaches. despite their achievements, the effective application of such methods to continual nlp remains largely unexplored. our empirical observations have shown that directly applying these frameworks for continual event detection leads to degraded performance, as evident in the first conundrum of utilizing moo for continual event detection is the highly imbalanced distribution of training data in later tasks, where current-task classes are significantly more prevalent. failing to adequately address this problem can make the model susceptible to poor generalization (johnson and @xcite . second, the existing moo methods lack a clear criterion to determine whether a solution on the pareto front would be ideal for mitigating catastrophic forgetting, as well as a systematic approach to reach such a solution. in the context of continual learning, where tasks are not simultaneously presented at the beginning, an efficient solution must surpass the pareto-optimality criteria; it should also exhibit remarkable robustness in learning new tasks and minimize performance decline in previously encountered tasks. regarding the above criterion, several studies (mirzadeh et al., 2020) have proposed a relationship between solutions found at flat minima in the loss landscape and their effectiveness in alleviating catastrophic forgetting. "flat minima" refers to regions where the loss function demonstrates a relatively wide and flat basin, fostering more robust models with reduced overfitting risks. recently, a study @xcite has proposed the use of sharpness-aware minimization (sam) @xcite for finding pareto-optimal solutions in flatter regions of the loss landscape to improve moo. however, in the context of continual learning, the utilization of sharpness-aware minimization necessitates a nuanced approach to fully harness its potential. specifically, it is crucial to overcome the inherent instability associated with sam's adversarial nature and prevent it from negatively impacting the performance of the model. contribution. to address the above challenges, in this paper, (i) we propose sharpseq, a novel approach that enables effective utilizations of multitask learning frameworks with sharpness-aware minimization (sam) for continual event detection. it is a meticulously devised adaptation of sam, tailored for the context of sequential task emergence in continual learning. our method entails strategically applying sam exclusively to the objectives affiliated with the current task, while excluding its application to the objectives of previously encountered tasks. this selective integration introduces a layer of objective filter, effectively addressing the unique challenges posed by continual learning. to the best of our knowledge, our work is the first to propose a strategic adaptation of sam and moo for enhancing continual learning. (ii) to address data imbalance, we propose using a generative model to learn the underlying distribution of event triggers representations from each event type, and thereby synthesize data to alleviate the imbalance between past-task and current-task data during replay. our extensive experiments on multiple datasets show that our method outperforms state-of-the-art baselines by significant margins. ## proposed method to improve the common continual event detection workflow, which we describe in subsections 2.2 and 2.3, we propose the following method. in our proposed method, we use bert, which is frozen during training, as the pretrained language model. the overview workflow of our proposed method is illustrated in appendix a.1. ## experiments in this section, we present empirical results and analysis to demonstrate the effectiveness of our contributions in improving continual event detection, in comparison against several state-of-the-art baselines. ## conclusion in this work, we introduce sharpseq, a novel framework that enables the seamless integration of state-of-the-art gradient-based multi-objective optimization methods into continual event detection systems. by addressing the challenges of imbalanced training data and the unique nature of continual learning, our method significantly enhances the performance of continual event detection. through rigorous empirical benchmarks, we demonstrate the effectiveness and versatility of our contributions, extending beyond the realm of continual event detection and showcasing the potential for leveraging multi-objective optimization in solving various continual learning problems across various domains. this work sets a solid foundation and paves the way for future research in this exciting and rapidly evolving field.