T emp C aps: A Capsule Network-based Embedding Model for Temporal Knowledge Graph Completion (2022)

Abstract: Temporal knowledge graphs store the dynamics of entities and relations during a time period. However, typical temporal knowledge graphs often suffer from incomplete dynamics with missing facts in real-world scenarios. Hence, modeling temporal knowledge graphs to complete the missing facts is important. In this paper, we tackle the temporal knowledge graph completion task by proposing TempCaps , which is a Caps ule network-based embedding model for Temp oral knowledge graph completion. TempCaps models temporal knowledge graphs by introducing a novel dynamic routing aggregator inspired by Capsule Networks. Specifically, TempCaps builds entity embeddings by dynamically routing retrieved temporal relation and neighbor information. Experimental results demonstrate that TempCaps reaches state-of-the-art performance for temporal knowledge graph completion. Additional analysis also shows that TempCaps is efficient.

## introduction knowledge graphs (kgs) organize and integrate information in a structured manner, which is humanreadable and suitable for computer processing. this advantage of knowledge graphs is helping to bridge the gap between humans and computers. numerous real-world applications have benefited from kgs. in particular, recent advances in artificial intelligence have motivated researchers to use knowledge graphs to boost performance in downstream applications, including natural language processing @xcite and computer vision @xcite . despite the usefulness of knowledge graphs, existing knowledge graphs are often incomplete, which means important facts might be missing. to tackle this problem, researchers have developed various methods for the task of knowledge graph completion @xcite , aiming to recover missing facts for existing knowledge graphs. in particular, @xcite explored the capsule network (cap-snet) @xcite for modeling knowledge graphs. @xcite demonstrate that each dimension of the entity, as well as relation, embeddings also have diverse variations in different contexts. thus, they used capsules to encode many characteristics in the embedding triple and represent the entries at the corresponding dimension, showing superior performance to other kg models. existing studies, including capse, focus on completing static knowledge graphs. in reality, however, multi-relational data is often time-dependent. moreover, static knowledge graphs fail to adequately describe the changing essence of the world, indicating that knowledge or facts being true in the past might not always stay true. for instance, social networks constantly change. static knowledge graphs fail to model these changes. to this end, temporal knowledge graphs (tkgs) are introduced to grasp these dynamic changes. specifically, temporal facts are represented as a quadruple by extending the static triplet with a timestamp describing when these facts occurred, i.e. (barack obama, inaugurated, president of the us, 2009). similar to static kg, tkgs also suffer from the problem of incompleteness, making the task of temporal knowledge graph completion eminent @xcite . in this paper, we take advantage of the capsule network paradigm and generalize it for modeling tkgs. we introduce tempcaps, which is a capsule network-based embedding model for temporal knowledge graph completion. as shown in figure 1 , tempcaps consists of a neighbor selector, an entity embedding layer, a dynamic routing aggregator and a multi-layer perceptron (mlp) decoder. unlike capse, we incorporate the temporal information of tkgs into our model: first, we pose temporal constraints on neighbor selection by introducing a time window. at a given time step, we only take the neighbors that interact with the source entity within the time window into account for capturing the entity features. second, we propose a time-dependent dynamic routing mechanism that incorporates time information into routing weight matrix. third, we exploit the temporal weighting vectors generated during the dynamic routing to calculate the output probability, which reflects how tightly lower capsules connect with higher capsules. our contributions are in the following: (i) we propose tempcaps, which leverages capsule networks by dynamically routing retrieved temporal relations and neighboring entities. an advantage of our model is that different capsules can capture different aspects of the same entity. such advantage is important for modeling temporal knowledge graphs, which are dynamic, and often involve one entity in multiple timestamps. (ii) our tempcaps improves the performance of temporal knowledge graph completion. experimental results show that our model achieves state-of-the-art performance on the gdelt and icews datasets. furthermore, our model is light-weighted and efficient compared to previous methods for modeling tkgs. (iii) as far as we know, we are the first to use capsule networks for tkgs. our experiments show that by leveraging dynamic routing, tempcaps is suitable for both discrete and continuous timestamps and can be easily generalized to unseen timestamps. (iv) we conduct additional ablation studies to understand how each part of tempcaps contributes to the model performance. we also show that tempcaps is efficient by analyzing time and space complexity. ## conclusion in this paper, we propose tempcaps, which is a light-weighted capsule network-based embedding model for temporal knowledge graph completion. tempcaps consists of a neighbor selector, a dynamic routing aggregator, and an mlp decoder. experimental results show that tempcaps reaches state-of-the-art performance on the gdelt and icews05-15 dataset. we conduct additional ablation studies to understand how each part of temp-caps and hyperparameter choices contribute to the model performance. our analysis also shows that tempcaps is efficient both in time and space. in the future, we plan to extend tempcaps to forecasting in temporal knowledge graphs.