S i MF y: A Simple Yet Effective Approach for Temporal Knowledge Graph Reasoning (2023)

Abstract: Temporal Knowledge Graph (TKG) reasoning, which focuses on leveraging temporal information to infer future facts in knowledge graphs, plays a vital role in knowledge graph completion. Typically, existing works for this task design graph neural networks and recurrent neural networks to respectively capture the structural and temporal information in KGs. Despite their effectiveness, in our practice, we find that they tend to suffer the issues of low training efficiency and insufficient generalization ability, which can be attributed to the over design of model architectures. To this end, this paper aims to figure out whether the current complex model architectures are necessary for temporal knowledge graph reasoning. As a result, we put forward a simple yet effective approach (termed SiMFy), which simply utilizes multilayer perceptron (MLP) to model the structural dependencies of events and adopts a fixed-frequency strategy to incorporate historical frequency during inference. Extensive experiments on real-world datasets demonstrate that our SiMFy can reach state-of-the-art performance with the following strengths: 1) faster convergence speed and better generalization ability; 2) a much smaller time consumption in the training process; and 3) better ability to capture the structural dependencies of events in KGs. These results provide evidence that the substitution of complex models with simpler counterparts is a feasible strategy.

## introduction knowledge graphs (kgs), which represent events as triples (s, r, o), facilitate a wide range of natural language processing tasks, including semantic search @xcite , product recommendations @xcite , and question-answering systems @xcite ). 1 the source code and datasets are available at https:// github.com/cgcl-codes/simfy . however, in real-world scenarios, tkgs are often incomplete, highlighting the vital importance of tkg reasoning, which aims to predict future facts by utilizing the temporal information. as illustrated in figure 1 , given the tkgs associated with entity a at the timestamps of time i, time ii, and time iii, our objective is to predict a quadruple containing an unknown entity, i.e., (a, negotiate, ?, time iv). to achieve this goal, we can observe a significant structural dependency of facts along the timeline. that is, interactions between entity a and entity b at time i could influence the interaction patterns at time ii, which, in turn, might provide clues for time iii. we also notice that historical events may recur. for instance, entity a and entity b had conflicts at both time i and time ii timestamps. these observations provide us valuable insights into tkg reasoning. in the tkg reasoning task, events to be predicted can typically be classified into two main types: historical events and unseen events @xcite . historical events, also known as repetitive pattern, refer to facts that have already occurred in the historical kg sequence. unseen events refer to events that have not occurred in the historical kg sequence. many methods @xcite can predict historical events effectively by modeling the historical kg sequence auto-regressively. however, for unseen events, it is necessary to consider both the information of structural dependency and temporality of entities and relations. typically, existing works @xcite apply graph neural networks (gnns) and recurrent neural networks (rnns) as encoders to capture the structural and temporal information in the historical kg sequence. then, a translational model, such as conv-transe @xcite , is used as a decoder to obtain predicted entities. despite their effectiveness, these approaches are conceptually and technically complex due to their advanced model architectures. moreover, these complex methods tend to suffer the issues of low training efficiency and insufficient generalization ability. in this paper, we aim to answer the following research question: are these complex model architectures indispensable for temporal knowledge graph reasoning? as a solution, we design a simple mlp-frequency-based model (simfy) to evaluate it against other complex baselines. specifically, we use a one-layer mlp to jointly model entities and relations, capturing structural and temporal information in tkgs. this allows us to obtain embedding vectors for entities and relations, which are then used to calculate the similarity between queries and candidate entities, resulting in preliminary candidate entity scores. next, we calculate the historical frequency scores of candidate entities based on their historical kg sequences. finally, in the inference stage, the two scores mentioned above are combined using a coefficient α to obtain the final entity scores. through extensive experiments comparing it with existing state-of-the-art models, we find that our model can achieve comparable performance while having higher training efficiency and better generalization ability. furthermore, we conduct a series of empirical studies to investigate the performance of simfy and existing complex models under specific conditions. we are the first to analyze the performance of mlp and gnns in capturing unseen entities. our findings indicate that mlp performs comparably to gnn in capturing structural dependency infor-mation for the tkg reasoning task. furthermore, we investigate the question of whether historical frequency information should be incorporated into the model training process, which has not been explored before. unlike the most current mainstream methods which directly incorporate historical frequency information of entities during training, simfy utilizes conceptually simple features to model the repetitive pattern of tkgs, which is fixed in training. through empirical experiments, we validate the effectiveness of this fixed-frequency strategy adopted by simfy. the contributions of this paper are as follows. similarity matching historical frequency learning …… 𝑜 4 𝑜 2 𝑜 1 𝑜 5 𝑟 𝑟 1 𝑟 2 𝑜 3 𝑜 2 𝑜 5 𝑟 𝑟 1 𝑟 1 s 𝑜 7 𝑜 4 𝑜 1 𝑜 6 𝑜 3 𝑟 2 𝑟 3 𝑟 …… …… probability distribution 𝑜 1 𝑜 2 𝑜 3 • • • 𝑜 4 𝑜 5 ? prediction 𝑜 1 𝑜 2 • • • 𝑜 4 𝑜 2 𝑜 3 • • • 𝑜 5 𝑜 1 𝑜 4 • • • 𝑜 6 𝑠 𝑠 softmax mlp … vector representation (𝑠, 𝑟, ? , 𝑡 + ∆𝑡) query figure 2: overall model architecture of simfy. given a query (s, r, ?, t + ∆t), the similarity matching module learns the similarity score between the query and each candidate entity o, and the historical frequency learning module learns the historical frequency score between them, then the two scores are combined to generate the final probability distribution of entities. structures, and kg-bart utilizes pretrained language models augmented with knowledge graphs to not only enhance inference but also improve the generation of commonsense-reasoned text. temporal kg reasoning temporal knowledge graph inference, which takes into account the temporal evolution of events, generally falls into two categories: interpolation-based and extrapolationbased inference. interpolation-based inference aims to guess unknown facts within a known time range. ttranse (jiang et al., 2016) integrates temporal information into the transe (bordes et al., 2013) model using recursive neural networks, while hyte (dasgupta et al., 2018) designs a unique hyperplane to embed time into the entity-relation space. temp (hu et al., 2022) addresses time sparsity and variability issues by combining neural message passing and temporal dynamic methods. on the other hand, extrapolation-based inference, which forecasts unknown future facts, is garnering increasing attention. re-net @xcite employs an encoder and aggregator to model past facts, while hip @xcite utilizes temporal, structural, and repetitive information. xerte @xcite offers a novel framework for predicting future facts and cygnet @xcite introduces a creative copy-mechanism used in natural language generation tasks before. reinforce-ment learning is adopted by cluster @xcite , which infers answers from induced clues, and timetraveler @xcite , which uses historical knowledge graph snapshots for answer search. cen @xcite adopts a length-aware convolutional network to model the kg sequence dynamically and ght @xcite is the first method to introduce a transformer into the tkg reasoning task. finally, da-net @xcite and cenet @xcite propose unique event prediction models. da-net learns distributed attention to future events, while cenet distinguishes likely entities for a given query using a historical contrastive learning framework. ## our approach here we elaborate our proposed simple mlp-frequency-based model (simfy) for temporal graph reasoning. as illustrated in figure 2 , simfy mainly involves two modules: the similarity matching module and the historical frequency learning module. these two modules generate corresponding scores for candidate entities. afterward, a weight-based inference process is utilized to determine the final result of the reasoning. in the subsequent sections, we will provide a comprehensive introduction to our proposed method. ## more empirical results and analysis here, we show more empirical results and analysis to reveal the strengths of our proposed simfy.