On the Evaluation of Methods for Temporal Knowledge Graph Forecasting
Abstract
Due to its ability to incorporate and leverage time information in relational data,
Temporal Knowledge Graph (TKG) learning has become an increasingly studied research field.
With the goal of predicting the future, researchers have presented innovative methods for what is called Temporal Knowledge Graph Forecasting.
However, the experimental procedures in this line of work show inconsistencies that strongly influence empirical results and thus lead to distorted comparisons among models.
This work focuses on the evaluation of TKG Forecasting models: we describe evaluation settings commonly used in this research area and shed light on its scholarship issues.
Further, we provide a unified evaluation protocol and carry out a re-evaluation of state-of-the-art models on the most common datasets under such a setting.
Finally, we show the difference in results caused by different evaluation settings.
We believe that this work provides a solid foundation for future evaluations of TKG Forecasting models and can thus contribute to the development of this growing research area.
Introduction
Temporal Knowledge Graphs (TKG) are Knowledge Graphs (KG) where facts occur, recur or evolve over time [1].
TKG can accommodate time-evolving multi-relational data by extending facts with a timestamp to indicate that a triple is valid at this timestamp [2].
The research field of TKG Forecasting, or TKG Extrapolation, aims at predicting facts at future timesteps, based on the KG history [3].
Recently, various methods have been proposed to advance the field.
Unfortunately, and despite the progress made so far in TKG Forecasting, various reported experimental settings show discrepancies: first, the existing models are evaluated on scores computed with different filter settings; second, models for single-step prediction that predict one step to the future are lumped together with models for multi-step prediction that predict multiple steps to the future; third, multiple versions of the same datasets exist.
Last but not least, some models do use the information from the validation set for testing, whereas others do not.
These four issues can strongly influence the empirical results and significantly decrease comparability across works.
Consequently, it is very difficult to understand existing methods’ strengths or weaknesses or to identify the currently best-performing method.
In this paper, we address the aforementioned issues in the evaluation of TKG Forecasting models.
We first provide an overview of existing models for TKG Forecasting (Section 2).
We then describe common evaluation settings and compare those settings used in state-of-the-art models to highlight the inconsistencies (Section 3).
In this context, we explain the problems we discovered for each setting.
Because it is essential to evaluate models in a consistent way, we form a unified evaluation protocol using reasonable and sound evaluation settings (Section 4).
We re-evaluate state-of-the-art models on this protocol and show results for seven state-of-the-art models on five commonly used datasets (Section 5).
In addition, we provide insights into the influence of different setups on the result scores.
Our hope is to set a new standard for rigorous evaluations of new models in this growing research field.
Our contributions are:
A comprehensive discussion of evaluation settings and accompanying problems for TKG Forecasting.
The design of a unified evaluation protocol for TKG Forecasting from reasonable evaluation settings.
An extensive re-evaluation of state-of-the-art models on a consistent evaluation protocol, showing results and insights on the influence of different evaluation settings on these results.
Disclaimer : This work does not want to belittle the effort others have put into the development of methods for TKG Forecasting.
Instead, we want to point out inconsistencies that do exist and how they may impact the final results.
By suggesting a uniform evaluation protocol for TKG Forecasting, we provide a solid basis for a fair, objective, and consistent comparison of future contributions with existing methods.
Related Work and Terminology
Temporal Knowledge Graph Forecasting:
In recent years (2019-2022), researchers have proposed various methods for TKG Forecasting.
Some of them leverage Graph Neural Networks [4, 5] in combination with a sequential approach to integrate structural and sequential information in the learning process.
This includes RE-Net [6], RE-GCN [7], TANGO [8], xERTE
[2], and CEN [9].
Further, CluSTeR [10] and TimeTraveler [3], introduce Reinforcement Learning Approaches for
TKG Forecasting.
On the other hand, the model TLogic [11], is a rule-based approach.
In addition,
CyGNet [12] predicts based on the appearance and repetition of historical facts.
Please see the supplementary material for more detailed information on each method.
In our work, we analyze the evaluation discrepancies of the introduced models and evaluate the models on a joint evaluation protocol.
Evaluation of Graph-based Machine Learning Models: When conducting empirical evaluations of machine learning algorithms, various issues can arise [13].
Such problems have been reported and partially addressed in various subfields, but in the following, we limit the discussion to works in the field of Graph Machine Learning.
Shchur et al.
[14] describe the shortcomings of evaluation strategies for Graph Neural Network models for node classification.
Errica et al.
[15] focus on graph classification, providing standard practices that should be avoided for a fair comparison. Further,
Rossi et al.
[16] describe shortcomings in the evaluation of KG link prediction.
Han et al. [17] focus on the evaluation of models for TKG completion (not forecasting).
Our work is the first to study evaluation problems for TKG Forecasting.
Terminology:
A TKG is formalized as a sequence of timestamped Knowledge Graphs, G = ( G 1 , G 2 , ..., G t , ... ) .
A timestamped KG G t =
{V , R , E t } , or KG snapshot, describes the TKG at timestep t , with the set of entities V , the set of relations R , and the set of facts E t at discrete timestamp t .
Facts E t are quadruples ( s, r, o, t ) , with s, o, ∈ V , and r ∈ R , for example (Kamala Harris, visit,
France, 2021-11-10).
Entity prediction for TKG Forecasting is the task of predicting the missing object entity ( s, r, ? , t + k ) and subject entity (? , r, o, t + k ) for a query, with k ∈ N + .
Description of Evaluation Settings and Evaluation Problems
In this chapter, we subsequently focus on evaluation settings for TKG forecasting.
In each subsection, we first describe a setting, and second, describe problems that we have encountered in that setting.
In addition, Table 1, provides an overview, showing the settings each model uses by default.
We refer to the respective parts of the table in each subsection.
Further, the table contains links to the published code for each model, if available.
3.1 Filter Settings for link prediction metrics
Researchers in TKG Forecasting evaluate the models on metrics known from static link prediction, namely Mean Reciprocal Rank (MRR) and Hits@k, with k = 1 , 3 , 10 .
There are three settings which have been introduced subsequently, raw , static filter , and time-aware filter :
Raw : As introduced by Bordes et al. [18], for each test triple ( s test , r test , o test ) , remove the object ( s test , r test , ?) , and compute the score that the model assigns for each entity v ∈ V to be the object in that triple, where the set of all possible triples ( s test , r test , v ) is termed corrupted triples.
Sort the scores in descending order, and note the rank of the correct entity o test .
Repeat this by removing the subject (? , r test , o test ) .
The MRR is the mean of the reciprocal of these ranks across all queries from the test set, and Hits@k is the proportion of correct entities ranked in the top k.
Static filter : To avoid counting higher ranks from other valid predictions as errors and thus having flaws in the metrics, Bordes et al. [19] propose to remove all triples (except the triple of interest) that appear in the train, valid, and test set from the list of corrupted triples.
Time-aware filter : Han et al. [20] note that the static filter setting is inappropriate for temporal link prediction because it filters out all triples that have ever appeared from the list of corrupted triples, ignoring the time validity of facts.
As a consequence, it does not consider predictions of such triples as erroneous.
For example, if there is a test query (Barack Obama, visit, India, 2015-01-25) and if the train set contains (Barack Obama, visit, Germany, 2013-01-18), the triple (Barack Obama, visit,
Germany) is filtered out for the test query according to the static filter setting, even though it is not true for 2015-01-25 [2].
For this reason numerous works [2, 11, 8, 3, 9, 10] apply the time-aware filter setting which only filters out quadruples with the same timestamp as the test query.
In the above example, (Barack Obama, visit, Germany, t ) would only be filtered out for the given test query, if it had the timestamp t = 2015-01-25, and otherwise stay in the list of corrupted triples.
Problem 1: Different Filter Settings.
The works introduced in Section 2 do present result scores with MRR and Hits@k using the above-described filter settings.
However, not all works report results on all filter settings, which is a problem, as it decreases comparability across works.
Further, as mentioned above, the raw, and especially the static filter setting are not appropriate for TKG
Forecasting.
The first part of Table 1 illustrates the filter settings that each model reports.
3.2 Single-step and Multi-step prediction
Methods for forecasting run under two different prediction settings, single-step and multi-step prediction.
Single-step (or one-step) prediction means that the model always forecasts the next timestep [21].
The ground truth facts are fed after every timestep before predicting the subsequent timestep.
Multi-step prediction means that the model forecasts more than one future time step [21].
More specifically, in TKG Forecasting, the model predicts all timesteps from the test set, without seeing any ground truth information in between.
As described by Brownlee [21], multi-step prediction is more challenging, as the model can only leverage information from its own forecasts, and uncertainty accumulates with an increasing number of forecasted timesteps.
Problem 2:
Comparison of multi-step and single-step setting.
The models described in Sectionrun in different settings.
Some can do single-step prediction only, some can do multi-step prediction only, and some do both (see Table 1, second part).
Still, single-step models are compared to multi-step models without drawing attention to the different setups.
For example, TLogic [11] and TANGO [8] (single-step) are compared to RE-Net [6] (multi-step), and xERTE [2] to CyGNet [12].
The second part of Table 1 shows each model’s prediction setting.
3.3 Datasets
Researchers in the domain of TKG Forecasting use the following datasets: Three instances of ICEWS [22]: ICEWS05-15 [23], ICEWS14 [23], and ICEWS18 [24], where the numbers mark the respective years; further, YAGO [25] and WIKI [26], preprocessed according to Jin et al. [24], as well as GDELT
The supplementary material contains dataset statistics.
Problem 3: Multiple versions of the same dataset.
The models described in Section 2 report results on different versions of the same dataset.
For instance, three versions exist for ICEWS14.
This hinders the comparability of results across works, causing confusion and potential errors.
The third part of Table 1 shows an overview of different versions of each dataset, describing each version (marked with (a), (b), (c)) by the number of training triples.
One version of the ICEWS14 dataset (see
Table 1, version (c)) is especially problematic, as it does not contain a validation set.
Instead, the test set is used for both validation and testing.
Thus, with this setting, the test set is leaked during training.
3.4 Train, Validation, and Test Set
Researchers in TKG Forecasting split each dataset D into a training D train , validation D valid , and test set D test .
The model’s training is conducted on D train , not using information contained in D valid or D test .
D valid can be used for monitoring the training process, and selecting the best model (parameters) across epochs.
There are different options to use the validation set during testing:
(a) The model can leverage all information from D train , but not from D valid , to predict D test .
This is consistent with the setting in link prediction for static knowledge graphs. (b) The model can leverage all information from D train and from D valid , to predict D test . This means, if a model has to answer
the
query
( s, r n during testing, all quadruples from D
train and D valid
can
be
used.
This is consistent with the setting used in time-series forecasting.
Problem 4: Usage of Validation set for Testing.
For multi-step setting, during testing, some models (CygNet, TLogic) do not use the information from the validation set (option (a)), whereas others (RE-GCN, RE-Net) do use it (option (b)), see the fourth part of Table 1.
Not using the information from the validation set leads to a significantly harder task, as the model needs to forecast more steps in the future: Instead of starting to predict the next unknown timestep t +1 for the first test set sample, the model needs to already predict the timestep t + num valid + 1 , with num valid being the number of timesteps in the validation set, as an information gap between training and testing.
3.5 Problem Summary
When putting all four problems together, a dramatic picture emerges: results have been compared using different filter settings, prediction settings, dataset versions, and dataset splits.
Table 1 illustrates the scattered landscape of evaluation settings, where no two models have ever been evaluated on identical settings.
Without a uniform and standardized evaluation protocol, we will never be able to gauge true progress in the field.
Still, in existing work, the methods are compared to each other, leading to confusion and inconsistencies.
A unified evaluation protocol
To tackle the problems introduced in Section 3, it is essential to evaluate TKG models in a consistent way.
For this reason, we introduce a unified evaluation protocol with clear and reproducible choices.
Filter settings: We report results on the time-aware filter setting. As explained in Section 3.1 this setting avoids counting higher ranks from other valid predictions as errors while taking into account time validity of facts.
Single-step and Multi-Step: While both settings are valid, the comparison of results for different settings is not fair (see Section 3.2).
The setting to be used depends on the use case and on the methods’ capabilities.
If the method can predict in single- and multi-step, we re-evaluate it on both settings.
Datasets Versions: The same dataset versions should be used across works to ensure comparability.
We suggest using version (a) for each dataset (see Table 1).
We selected the dataset versions used by the authors of RE-GCN [7], mainly because these are (among) the most commonly used versions across all works.
The supplementary material shows dataset statistics.
Train, Validation, and Test Set Usage: We use the train, validation, and test sets as described in
Section 3.4, option (b), where the information from the validation set can be used for testing, to avoid time gaps between training and testing.
In addition, we make sure that the test set is never used for model selection and the datasets are split based on ordered timestamps, whereas one timestamp should not belong to two different sets.
[Table]
[Table]
- - - - - ✓ - - -
[Table]
:✓ ✓ - ✓ - - - - ✓
[Table]
✓ ✓ - ✓ - ✓ ✓ - -
[Table]
- - ✓ - - - - - -
[Table]
✓ ✓ - ✓ - ✓ ✓ ✓ -
[Table]
✓ ✓ ✓ - - ✓ ✓ ? ? Reference [7] [6] [2] [12] [11] [8] [3] [9]
[Table]
[Table]
i
[Table]
https
Net https:
TANGO
Experiments
In the following, we show the results for seven models 2 , and five datasets 3 .
The supplementary material contains additional information on specific experimental settings.
Please find the source code with
scripts for experiments and evaluation at https://github.com/nec-research/ TKG-Forecasting-Evaluation .
[Table]
TANGO
[Table]
Timetraveler
[Table]
[Table]
[Table]
[Table]
.56 xERTE
[Table]
[Table]
26 TLogic 42.
[Table]
We run the experiments on a system with one Nvidia TITAN RTX (24 GB) GPU, 512 GB Memory, and an Intel Xeon Silver 4208 CPU with 16 cores (32 threads).
To eliminate the four problems described in Section 3, we follow the evaluation protocol from
Section 4: We report results on time-aware filter settings for single-step and multi-step settings, use the dataset versions (a), and report the results with the validation set usage option (b).
We show aggregated results (mean MRR and Hits@k across all test samples) for the seven models for the datasets GDELT, YAGO, WIKI, ICEWS14, and ICEWS18 in Table 2.
The upper part for each dataset contains results in multi-step setting, and the lower part in single-step setting, where models with results for single-step prediction should not be benchmarked against methods with results of multi-step prediction.
We mark the best result for each dataset for each setting in bold .
For completeness and comparability to related work, the supplementary material reports results on raw and static filter settings.
In addition, the supplementary material contains tables with information on the reproducibility of the results that have been reported by the original works [2, 3, 6, 7, 8, 11, 12].
Figure 1 shows the MRR for three selected datasets (ICEWS18, WIKI, and GDELT) over test timestamps (snapshots) for different evaluation settings.
In the following, we will discuss important insights.
Single-step and Multi-step setting: Table 2 shows the difference in scores for single- vs. multi-step setting: Overall, scores for single-step setting are higher than for multi-step setting.
This is especially visible for the two models (TLogic and RE-GCN) that run in both settings, but also true for the other results.
Figure 1(a) - (d) shows the MRR (in % ) over snapshots in multi-step setting (left)
[Figure]
and single-step setting (right).
The figure exhibits that the MRR for multi-step prediction has a decreasing trend with increasing timestamps, in contrast to single-step prediction, which shows no such decreasing trend.
This is especially visible for the WIKI dataset in a single-step setting, which displays an increasing tendency for the MRR with increasing timestamps for the four best-performing methods.
The results reflect the statement from Section 3.2, that multi-step prediction is more challenging, and uncertainty accumulates with increasing number of forecasted timesteps, as the models can only leverage information from their own forecasts.
Thus, benchmarking models for multi-step prediction against single-step prediction is only fair for the first timestamp.
Validation Set Usage:
In Figure 1(e), we show the MRR (in % ) over snapshots in multi-step setting for TLogic and CyGNet 5 , when using the validation set for testing (Section 3.4, option (b)) vs. not using the validation set for testing (option (a)) for the dataset WIKI.
6 The figure exhibits a difference in MRR between the two settings for each model, especially in the first two snapshots with a difference in MRR of > 30 for TLogic.
This difference is caused by the information gap between the last training timestamp and the first testing timestamp.
For the case of WIKI, the number of timestamps in the validation set is num valid = 11 .
The difference decreases with increasing timestamps, because, due to the multi-step setting, there is also a rising information gap when feeding the validation set.
Thus, using the information from the validation set for testing and avoiding the information gap is crucial for fair comparison among models.
Filter Settings: Figure 1(f) shows the MRR (in % ) over snapshots in multi-step setting, exemplary for CyGNet and RE-GCN for the dataset GDELT, computed with raw, static, and time-aware filter setting, as described in Section [-3.1. 
-]3.1 7 .
It exhibits a large difference in MRR for static filter setting, vs. raw setting or time-aware filter setting, especially for CyGNet.
This is also visible for aggregated results: Where CyGNet does not have the highest MRR scores on any dataset for time-aware filter settings (see Table 2), it has the highest MRR scores on all five datasets in static filter setting (see supplementary material).
The static filter setting filters out all triples that have ever appeared from the corrupted triples, ignoring the time validity, and does not count a prediction of these triples as error.
Thus, for a given query, if a model predicts entities that have appeared in this triple at an earlier timestep, this will not be considered erroneous, even if the predicted fact is not true in the timestep of question.
The model will potentially be assigned a higher static filter score than if it would predict previously unseen facts.
Thus, the static filter setting favors models that predict repeated facts.
To summarize, we can see that no model shows the best results across all datasets.
This evidence remarks the importance of fairly comparing models on different benchmarks.
In this section, we stressed the clear differences in result scores for single-step and multi-step prediction.
In addition, we pointed out that the usage of the validation set during testing does lead to substantially higher test scores.
We also showed the significant influence of the filter setting used for score computation.
Conclusion
Summary:
In this work, we examined the evaluation of TKG Forecasting models.
We uncovered inconsistencies that strongly influence the experimental results and thus lead to distorted comparisons among models.
We described these inconsistencies caused by different dataset versions, different filter settings, the absence of comparability of single-step and multi-step prediction, and the different practices of using the validation set during testing.
To address these problems, we formed a unified evaluation protocol from reasonable evaluation settings and re-evaluated state-of-the-art methods.
We illustrated the importance of a consistent evaluation by showing the effect of different evaluation settings on the results.
Our work aims at establishing a unified evaluation protocol, stimulating discussions on the evaluation, and raising the community’s awareness of experimental issues, with the goal of advancing the research field of TKG Forecasting.
Limitation of this study: Due to computational infeasibility, we could not conduct multiple repeats for each experiment run 8 .
Even with one repetition per run, we experienced significant computation times for many models, e.g., multiple days to weeks for the dataset GDELT; thus, multiple repetitions per model and dataset were not possible.
Adding multiple repetitions to the evaluation would have further improved the robustness of our results, which are nonetheless obtained under a unified and reproducible protocol.
Future Work:
In future work, we aim to extend the proposed evaluation protocol to: First, evaluate the full predicted graph for methods that can predict full graphs (e.g., RE-Net), instead of exclusively focusing on link prediction.
This could be based on graph similarity or computing a percentage of correctly predicted triples.
Second, evaluate the change of the predicted graph snapshots over time to analyze if the predictions evolve and if they are able to capture time information.
This could be done by comparing the predictions at different time steps.
Third, include more fine-grained evaluation to answer what properties the models learned and what they did not.
This could, for example, be done using the framework KGxBoard [28], which breaks down the performance measure (e.g., Hits@10) over individual data subsets.
Acknowledgments and Disclosure of Funding
We warmly thank Federico Errica for his time and very valuable feedback.
A Supplementary Material A.1 Additional Information on Experimental Settings
In the following we describe experimental settings in general, and subsequently special settings for each model, if they are different from the default settings or otherwise noteworthy.
For each model we log the predicted scores for each quadruple from the test set D test , in both directions (subject and object prediction) in a python dictionary.
The dictionary contains as keys the query ( s, r, ? , t ) or (? , r, o, t ) , and as values the scores predicted by the model for each entity v ∈ V to belong to that query.
After running all experiments on all datasets, settings, and methods, we compute the ranks (MRR, Hits@k, (with k = 1 , 3 , 10 )) as described in Section 3.1 of the main paper based on these dictionaries.
If not stated otherwise, we use the hyperparameter settings (including the number of training epochs) reported in the respective papers.
For each model, if the published source code provides the option to manually set a random seed, we did set the same seed.
If the option was not explicitly provided in the published source code or could be reached with small ( ≤ 5 lines) modifications, during training we did not validate the models on the same filter setting as we did test them on.
Instead, we validated on the default filter setting (please see details for each model below).
The reason is that, in a preliminary experiment on selected models, we found that the filter setting has only a small influence on the best validation epoch (i.e., different settings for validation lead more often than not to the same best epoch).
Thus, we are confident that this did not significantly influence the final results.
Due to issues regarding memory consumption and very high computing time, we were not able to conduct the experiments for the dataset ICEWS05-15 for the better part of models and thus excluded this dataset from our experiments.
RE-Net [6] We run RE-Net in multi-step setting only, because the published source code does not provide the option to set the single-step option in the arguments.
Also, when we asked via e-mail and GitHub issue about how to conduct the implementation of the single-step option, unfortunately, we did not receive a concrete reply.
We train the models on the static filtered MRR, following the training procedure provided in the source code.
Due to GPU memory issues with the dataset GDELT, we run the model for this specific dataset on CPU, which leads to a very long runtime ( > 50 days of training).
For the source code to be able to run on CPU, we have to conduct modifications to the source code.
We run all other experiments for RE-Net on GPU.
RE-GCN [7] We train the models on the raw MRR, following the training procedure provided in the source code.
While RE-GCN is originally also evaluated on relation prediction, we exclude this setting in our study, as the other models do not support relation prediction.
We run RE-GCN in both settings, single-step and multi-step.
CyGNet [12] We run CyGNet on multi-step setting only, because non-trivial modifications in the source code would be necessary to run in single-step setting.
Unfortunately, the authors did not reply to our question on the concrete implementation of multi-step setting.
We train one model for each setting, raw, static, and time-aware filter.
For testing, instead of allowing the model to only use the information from triples in the train set, we allow to also take into account the triples in the validation set.
For this, we slightly adjust the original source code:
In our version, the historical vocabulary (copy-sequences) now includes all timesteps from train and validation set, instead of only the timesteps from the train set.
Please see Figure 2 for an overview of the change in testing scores for time-aware filter setting, when using the validation set (our modification) versus not using the validation set (original) during testing.
TLogic [11] For the datasets ICEWS14 and ICEWS18 we use the hyperparameters as described in the paper.
The hyperparameter that changes across datasets is the window size w .
According to the authors, the higher the window size, the better the performance, but also the higher the memory need.
This means, that generally, the smaller (and less dense) the graph, the higher the window size can be in regard to memory usage.
The datasets YAGO, WIKI, and GDELT have not been evaluated in the original paper.
For the small dataset YAGO we set the window size to w = 0 , which includes all past timesteps.
For WIKI and GDELT we experience memory issues when using the machines described in Section 5 (main paper), and even when using a machine with 2 TB Memory.
Integrating instructions kindly provided by the authors, for the datasets WIKI and GDELT we can circumvent these memory issues by decreasing the rule length to l = { 1 , 2 } , instead of l = { 1 , 2 , 3 } .
In addition, for WIKI and GDELT we set the window size to w = 200 , the value reported by Liu et al. [11] for the larger dataset ICEWS18.
For the multi-step setting, we modify the published source code.
Instead of allowing the model to only apply the rules based on occurrences of quadruples in the train set, we allow to also take into account the quadruples in the validation set.
We modify the highest timestep for the rule application to be the highest timestep from the validation set, instead of the highest timestep from the training set.
In addition, for datasets ICEWS18, WIKI and GDELT, we implement the option to set the window size of w = 200 also for multi-step prediction (instead of using all quadruples from training and validation set).
Please see Figure 2 for an overview of the impact of using the validation set (our modification) versus not using the validation set (original) during testing (rule application) on testing scores for time-aware filter setting.
TimeTraveler [3] We run TimeTraveler only in single-step setting, because, as kindly confirmed by the authors, non-trivial modifications in the source code would have been necessary to run in multi-step setting.
For GDELT, no hyperparameters were specified in the original paper.
We use the same hyperparameters as for WIKI, because this dataset is the most similar in size.
TimeTraveler is capable of doing inductive link prediction for future timesteps, i.e., prediction of triples with previously unseen nodes.
We do not specifically evaluate this capability, as it is not in the scope of our study. xERTE [2] We run xERTE only in single-step setting, because, as kindly confirmed by the authors, non-trivial modifications in the source code would have been necessary to run in multi-step setting.
In the paper, hyperparameters are not specified for the datasets WIKI and GDELT.
We use the hyperparameters as specified for the ICEWS18 dataset because ICEWS18 is most similar in size.
For each epoch during training, we log the validation results for raw, static, and time-aware filter settings.
We run separate testing for the three filter settings, where we select the trained model from the best training epoch for the respective setting.
Please note, that in most cases, the best epoch was the same across settings 9 .
We experienced a very long training time ( > 30 days) for xERTE on the GDELT dataset.
TANGO [8] We only run TANGO in single-step setting, because non-trivial modifications in the source code would be necessary to run in multi-step setting.
Unfortunately, the authors did not reply to our question on the possibility of implementing the multi-step setting, nor on the question of how to realize the long-horizontal forecasting experiment they report in their paper.
For GDELT, no hyperparameters were specified in the original paper.
We use the same hyperparameters as for WIKI, because it is the most similar in size.
We train one model for each setting, raw, static, and time-aware filter.
TANGO is capable of doing inductive link prediction for future timesteps, i.e., prediction of [Figure] triples with previously unseen nodes.
We do not specifically evaluate this capability, as it is not in the scope of our study.
A.2 Additional Experiment Results A.2.1 Usage of the Validation Set
Figure 2 shows the performance of the methods TLogic and CyGNet on all datasets in multistep setting, when not leveraging the information from the validation set D valid (option a), versus leveraging D valid (option b) (see Section 3 in main paper) during testing.
Please note that the drop in scores for the sixth timestep on the YAGO dataset is due to the dataset only having two samples in this snapshot, and all models performing bad on these two samples.
[Table] [Table]
80.42 TLogic 37.
87.20 TANGO 41.
53.06 Timetraveler 28.
multi-step setting (static filter)
ICEWS ICEWS
MRR
H@ H@
H@
MRR
H@
H@ H@
RE-GCN 48. 55.81 RE-Net 48.
55.97 CyGNet 53.
57.81 TLogic 51.
single-step setting (static filter)
ICEWS ICEWS
MRR
H@
H@ H@
MRR
H@ H@
H@
RE-GCN 52.
61.23 xERTE 46.
49.91 TLogic 57.
[Table]
A.2.2 Static and Raw Filters
Tables 3 and 4 report results on static filter setting and raw filter setting, for the five datasets GDELT,
YAGO, WIKI, ICEWS14, and ICEWS18.
Although we do not encourage evaluation on these settings, we have added the results for reasons of completeness and comparability.
Figure 3 shows the MRR over test timestamps (snapshots), for multi-step and single-step prediction for the three remaining datasets (ICEWS14, YAGO, and GDELT) that have not been shown in the main paper.
Please note that the drop in scores for the sixth timestep on the YAGO dataset is due to the dataset only having two samples in this snapshot, and all models performing badly on these two samples.
[Table] [Table]
71.93 TLogic 19.
72.09 TANGO 18.
Timetraveler 19.
multi-step setting (raw)
ICEWS ICEWS
MRR
H@
H@ H@
MRR
H@ H@
H@
RE-GCN 37.
46.99 RE-Net 36.
45.67 CyGNet 35.
43.83 TLogic 34.
single-step setting (raw)
ICEWS ICEWS
MRR
H@
H@ H@
MRR
H@ H@
H@
RE-GCN 41.
51.94 xERTE 40.
45.84 TLogic 41.
[Table]
50 TANGO
[Table]
74 Timetraveler
[Table] [Figure] [Figure] [Table]
Different usage of validation set: option (a) as described in section 3.4 in main paper.
Thus, results are not comparable.
[Table]
Different usage of validation set: option (a) as described in section 3.4 in main paper.
Thus, results are not comparable [Table] A.2.3 Result Consistency Tables 5 and 6 show the difference ∆ of scores (MRR and Hits) reported by the authors of the original papers to the results from our experiments (as reported in Table 2, 3, and 4), if computable.
As we show in Table 1, various differences in evaluation settings exist, and not all papers report results on all datasets, thus it is not possible to compute the differences for all datasets and settings for each method.
[Table]
[Table]
Different usage of validation set: option (a) as described in section 3.4 in main paper.
Thus, results are not comparable.
[Table]
Different usage of validation set: option (a) as described in section 3.4 in main paper.
Thus, results are not comparable.
[Table]
A.3 Related Work, Additional Information
Following up on the short introduction in Section 2 of the main paper, here we provide more details on each TKG Forecasting model.
Graph Neural Networks (GNNs): A large group of models leverages a GNN [4, 5] in combination with a sequential approach to integrate the structural and sequential information.
RE-Net [6] applies an autoregressive architecture.
It builds on a two-step approach to learn the temporal dependency from a sequence of graphs and the local structural dependency from the neighborhood.
The occurrence of a fact is modeled as a probability distribution conditioned on the temporal sequence of past snapshots.
RE-Net can predict full graphs.
RE-GCN [7] also models the sequence of the Knowledge Graph snapshots recurrently.
For this, it combines a convolutional graph neural network with a sequential Neural Network model.
Further, RE-GCN introduces a static graph constraint to take into account additional information like entity types.
TANGO [8] bases on neural ordinary differential equations to model the temporal sequences combined with a GNN to capture the structural information.
In addition, the authors introduce a stochastic jump method to incorporate stochastic events, i.e., triples appearing or disappearing over time.
TANGO can learn continuous representations of entities and relations.
xERTE [2] bases on so-called temporal relational attention mechanisms.
To answer a query, it extracts query-relevant subgraphs.
Further, it computes and propagates attention scores to identify the relevant evidence in the subgraphs, using a modified time-aware version of a message passing.
CEN [9] integrates a Convolutional Neural Network which can handle evolutional patterns of different lengths via an easy-to-difficult curriculum learning strategy, which learns these evolutional patterns from short to long.
The model can learn in an online setting, and thus can adapt to changes in evolutional patterns over time.
Reinforcement Learning: CluSTeR [10] introduces a two-step process: First, a Reinforcement learning agent, working with randomized beam strategy, searches and induces clue paths related to a given query.
Second, an adopted GNN and sequence method models temporal information among the clues to find answers to a query.
TimeTraveler [3] leverages a reinforcement learning model based on temporal paths.
Starting from the query’s subject node, the agent traverses outgoing edges across graph snapshots.
For this, TimeTraveler samples actions according to transition probabilities, which are based on dynamic embeddings of the query, the path history, and the candidate actions.
TimeTraveler uses a time-shaped reward based on Dirichlet distribution.
The model is able to predict in the inductive setting by integrating a newly introduced Inductive Mean representation mechanism.
Rule-Based Approaches: TLogic [11], a symbolic framework, learns so-called temporal logic rules via temporal random walks, traversing edges through the graph backward in time.
Rules contain nodes, edges, and timesteps.
TLogic applies the rules to events that happened prior to the query.
For scoring the answer candidates, it takes into account the rules’ confidence as well as time differences.
Other: CyGNet [12] predicts future facts purely based on the appearance of historical facts.
For this, to answer a query, it first computes each entity’s embedding vector.
Further, using these embeddings, it computes entity probabilities by combining predictions from a so-called "copy mode" that computes probabilities for historical events based on the repetition of facts in history and a "generation mode" that computes probabilities for every entity.
A.4 Dataset Statistics
Following up on the description in Section 3 (main paper), please see table A.4 for dataset statistics for dataset version (a), as reported by Li et al.
[Table]
A.5 Checklist for Benchmark Experiments on TKG Forecasting
In the following, we provide a checklist for benchmark experiments on TKG Forecasting.
• Are the datasets used the same version for all models?
Check e.g., number of triples in train, validation, test set.
• Are the hyperparameters set as reported in the papers?
• Is the single-step/ multi-step setting consistent across models?
• Is the validation set used during testing?
• Are you sure that the test set is not leaked during training?
• Does the model predict in both directions, ( s, r, ? , t ) and (? , r, o, t ) ?
• Are evaluation scores computed based on time-aware filtered setting?
Is the implementation to compute the evaluation scores consistent across all models?