History repeats: Overcoming catastrophic forgetting for event-centric temporal knowledge graph completion (2023)

Abstract: Temporal knowledge graph (TKG) completion models typically rely on having access to the entire graph during training. However, in real-world scenarios, TKG data is often received incrementally as events unfold, leading to a dynamic non-stationary data distribution over time. While one could incorporate fine-tuning to existing methods to allow them to adapt to evolving TKG data, this can lead to forgetting previously learned patterns. Alternatively, retraining the model with the entire updated TKG can mitigate forgetting but is computationally burdensome. To address these challenges, we propose a general continual training framework that is applicable to any TKG completion method, and leverages two key ideas: (i) a temporal regularization that encourages repurposing of less important model parameters for learning new knowledge, and (ii) a clustering-based experience replay that reinforces the past knowledge by selectively preserving only a small portion of the past data. Our experimental results on widely used event-centric TKG datasets demonstrate the effectiveness of our proposed continual training framework in adapting to new events while reducing catastrophic forgetting. Further, we perform ablation studies to show the effectiveness of each component of our proposed framework. Finally, we investigate the relation between the memory dedicated to experience replay and the benefit gained from our clustering-based sampling strategy.

## introduction knowledge graphs (kgs) provide a powerful tool for studying the underlying structure of multirelational data in the real world @xcite . they present factual information in the form of triples, each consisting of a subject entity, a relation, and an object entity. despite the development of advanced extraction techniques, knowledge graphs often suffer from incompleteness, which can lead to errors in downstream applications. as a result, the task of predicting missing facts in knowledge graphs, also known as knowledge graph completion, has become crucial. @xcite @xcite kgs are commonly extracted from real-world data streams, such as newspaper texts that change and update over time, making them inherently dynamic. the stream of data that emerges every day may contain new entities, relations, or facts. as a result, facts in a knowledge graph are usually accompanied by time information. a fact in a semantic knowledge graph, such as yago @xcite , may be associated with a time interval, indicating when it appeared and remained in the kg. for example, consider @xcite in a semantic kg. a link between obama and united states appears in the graph after 2009, and it exists until 2017. on the other hand, a fact in a temporal event-centric knowledge graph (tkgs), such as icews @xcite , is associated with a single timestamp, indicating the exact time of the interaction between the subject and object entities. for example, in an event-centric tkg, (obama, meet, merkel) creates a link between obama and merkel several times within 2009 to 2017 since the temporal links only show the time when an event has occurred. therefore, eventcentric tkgs exhibit a high degree of dynamism and non-stationarity in contrast to semantic kgs. to effectively capture the temporal dependencies within entities and relations in tkgs, as well as new patterns that may emerge with new data streams, it is necessary to develop models specifically designed for tkg completion. a significant amount of research has been dedicated to developing evolving models @xcite @xcite for tkg completion. these models typically assume evolving vector representations for entities or relations. these representations change depending on the timestep, and they can capture temporal dependencies between entities. however, these models often assume that the entire dataset is available during training. they do not provide a systematic method for updating model parameters when new data is added. one potential solution is to retrain the model with new data. however, this approach can be resource-intensive and impractical for large-scale knowledge graphs. an alternative approach is to fine-tune the model with new data, which is more time and memory efficient. however, this approach has been shown to be susceptible to overfitting to the new data, resulting in the model forgetting previously learned knowledge, a phenomenon known as catastrophic forgetting (fig. 1 ). a limited number of studies @xcite @xcite have addressed this problem for semantic knowledge graphs using continual learning approaches, with tie @xcite being the most closely related work to current research. nevertheless, the development of efficient and effective methods for updating models with new data remains a significant challenge in event-centric temporal knowledge graphs. we propose a framework for incrementally training a tkg completion model that consolidates the previously learned knowledge while capturing new patterns in the data. our incremental learning framework employs regularization and experience replay to alleviate catastrophic forgetting. we propose a temporal regularization method based on elastic weight consolidation @xcite . by estimating an importance weight for every model parameter at each timestep, the regu-larization term in the objective function 'freezes' the more important parameters from past timesteps, encouraging the use of less important parameters for learning the current task. additionally, an exponentially decaying hyperparameter in the objective function further emphasizes the importance of the most recent tasks over older ones. our selective experience replay method uses clustering over the representation of the data points to first capture the underlying structure of the data. the points closest to the clusters' centroid are selected for experience replay. we show that the temporal regularization combined with clustering-based experience replay outperforms all the baselines in alleviating catastrophic forgetting. our main contributions include: our work is related to tkg completion, continual learning methods, and recent developments of continual learning for knowledge graphs. ## problem definition this section presents the formal definition of continual temporal knowledge graph completion. ## methodology our proposed framework is a training approach that can be applied to any tkg completion model. it enables the incremental updating of model parameters with new data while addressing the issues of catastrophic forgetting associated with fine-tuning. to achieve this, we utilize experience replay and regularization techniques -methodologies commonly employed in image processing and reinforcement learning to mitigate forgetting. additionally, we introduce a novel experience replay approach that employs clustering to identify and select data points that best capture the underlying structure of the data. furthermore, we adopt the regularization method of ewc, as proposed in @xcite , which incorporates a decay parameter that assigns higher priority to more recent tasks. our results demonstrate that the incorporation of a decay parameter into the ewc loss and prioritizing more recent tasks leads to improved performance. ## experiments in this section, we explain the evaluation protocol to quantitatively measuring the model catastrophic forgetting. from know tkg datasets, we create two benchmarks for tkg continual learning. we evaluate our proposed training method using the benchmark, compare them with various baselines and show the effectiveness of our approach in alleviating catastrophic forgetting. finally, we conduct ablation studies on different components of our training method to validate our model. ## conclusion we propose a framework for incrementally training a tkg completion model that consolidates the previously learned knowledge while capturing new patterns in the data. our incremental learning framework employs regularization and experience replay techniques to alleviate the forgetting problem. our regularization method is based on temporal elastic weight consolidation that assigns higher importance to the parameters of the more recent tasks. our selective experience replay method uses clustering over the representation of the data points and selects the data points that best represent the underlying data structure. our experimental results demonstrate the effectiveness of our proposed approach in alleviating the catastrophic forgetting for the event-centric temporal knowledge graphs. this work is the first step towards incremental learning for event-centric knowledge graphs. potential future work might involve exploring, and taking into consideration the effect of time on task similarities which might differ for various applications.