Lifelong Event Detection via Optimal Transport (2024)

Abstract: Continual Event Detection (CED) poses a formidable challenge due to the catastrophic forgetting phenomenon, where learning new tasks (with new coming event types) hampers performance on previous ones. In this paper, we introduce a novel approach, Lifelong Event Detection via Optimal Transport (**LEDOT**), that leverages optimal transport principles to align the optimization of our classification module with the intrinsic nature of each class, as defined by their pre-trained language modeling. Our method integrates replay sets, prototype latent representations, and an innovative Optimal Transport component. Extensive experiments on MAVEN and ACE datasets demonstrate LEDOT’s superior performance, consistently outperforming state-of-the-art baselines. The results underscore LEDOT as a pioneering solution in continual event detection, offering a more effective and nuanced approach to addressing catastrophic forgetting in evolving environments.

## introduction event detection (ed) @xcite presents a pivotal challenge in the domain of information extraction, tasked with identifying event triggers and their associated types from natural language text. however, the conventional ed training paradigm, characterized by its static nature, falls short in capturing the dynamic nature of real-world data. as highlighted by @xcite , the ontology of events in ed research has been exhibiting a constant shift since its introduction, prompting the exploration of continual event detection (ced), where data arrives continuously as a sequence of non-overlapping tasks. although large language models (llms) have recently emerged, showcasing the ability to tackle numerous problems using only prompts without the need for fine-tuning, they * equal contribution. ced presents many issues, most notably the catastrophic forgetting @xcite phenomenon, where the training signal from new task hampers performance on past tasks. to provide a solution for this issue, numerous methods have been proposed, which usually fall into one of the three eminent approaches: regularization-based @xcite @xcite @xcite ; architecture-based @xcite ; and memory-based (belouadah and popescu, 2019; @xcite . out of these three, memory-based methods have demonstrated superiority, leveraging access to the replay buffer, a memory of limited size containing a portion of data from previously learned tasks for the model to rehearse during the training of new tasks. despite the promise of memory-based methods, challenges abound. first, the finite capacity of the replay buffer results in the eviction of valuable information, leading to incomplete representations of past tasks and hence, inadequate generality. furthermore, the process of sampling and replaying data might not be optimally curated, potentially hindering the model's ability to generalize across tasks effectively. this setback arises because the conventional practice of discarding the original head of pretrained language models (plms) during fine-tuning on downstream tasks overlooks valuable linguistic information encoded within it. in training the classifier module, state-of-the-art approaches @xcite @xcite often do so in isolation, devoid of any priors or foundations. discarding the lan-guage modeling head in plms is highly wasteful. the language modeling head contains essential information about vocabulary distribution based on contextual representations. losing this head sacrifices crucial linguistic nuances, making it harder to align the classifier module and ensure efficient fine-tuning. aligning our classifier module to this information is an essential but also formidable challenge. this alignment is crucial for ensuring a more efficient fine-tuning process, as it provides a foundational standard of learning that mitigates unnecessary overplasticity and prevents catastrophic forgetting. to address the limitations discussed, this paper introduces a method to enhance memory-based ced by integrating optimal transport (ot) principles, which provide a robust framework for measuring the distance between probability distributions. by incorporating ot into the fine-tuning process, we aim to retain essential linguistic information from the plm head, ensuring the model remains invariant to specific tasks. this integration involves defining an appropriate cost matrix, a key challenge that we address by proposing a novel construction tailored to our method. our approach ensures effective alignment between the plm head and the classifier's output, leveraging ot to enhance the model's performance and robustness across various tasks while preserving the plm's inherent linguistic knowledge. ## lifelong event detection via optimal transport we incorporate optimal transport (ot) as a foundational element of our methodology. ot is a mathematical framework designed to compute the distance between two probability distributions with different supports. in our methodology, ot is applied to align the probability distribution output of the classifier head with the distributional characteristics inherent in the vocabulary of the pre-trained language model (plm) head. the softmax class probabilities from the classifier head are transported to closely match the pre-trained distribution, facilitating a seamless integration of task-specific knowledge while minimizing the divergence from the model's preexisting linguistic understanding. we forward each event trigger through a pretrained language model and its original language modeling head, and obtain a distribution over a dictionary of v words: each event trigger is associated with a distribution over c classes: @xmath0, where each entry indicates the probability that the event trigger belongs to a class in the ontology. an encoder is employed to generate p from x, defined as @xmath1, where θ represents the parameters of the neural network as described in section 2.1. given that x and p are distributions with different supports for the same event trigger, we aim to train the model by minimizing the following optimal transport (ot) distance to push p towards x: directly optimizing eq. ( 7 ) poses a timeconsuming challenge. to address this, an entropicconstrained regularized optimal transport (ot) distance is introduced, known as the sinkhorn distance: the cost matrix m is a trainable variable in our model. to overcome the challenge of learning the cost function, we propose a specific construction for m: to maintain the consistency of class representations across tasks, an additional regularization term enforces the proximity of class representations in the current task to those in the most recent task: finally, we can write our final objective function: avoiding catastrophic forgetting similar to many ced baselines, our method incorporates a replay process. however, our approach to constructing the memory buffer is distinct. for each class in the training data, we retain the prototype mean µ and diagonal covariance σ of its trigger representations encountered by the model, rather than storing explicit data samples. during replay, ## conclusion harnessing the inherent linguistic knowledge from pre-trained language modeling heads in encoderbased language models play a pivotal role in enhancing performance in downstream tasks. with the introduction of ledot, we present a novel approach utilizing optimal transport to align the learn-ing of each task with a common reference-the pretrained distribution of the vocabulary. this alignment mitigates overfitting to the current task and effectively addresses the challenge of catastrophic forgetting. our method, demonstrating superior performance across various benchmarks, stands as a testament to the effectiveness of leveraging pre-trained language modeling heads for continual event detection, offering a promising avenue for future research in this domain. in the future, we plan to extend our method to solve continual learning challenges for other information extraction tasks, such as event-event relation extraction @xcite .