Paper: Enhancing Temporal Knowledge Graph Completion with Global Similarity and Weighted Sampling (ICLR-2024)

Reviewer comment (Experiments):
Real-world deployment considerations are not discussed. For example, analyzing memory/compute needs and ability to handle streaming data.

Author response:
That is a great point. We have now added a table that report the memory and time complexity of each step of our approach both theoretically and in dataset ICEWS14. The table is copied below for ease of access. 

**Notation**:  $b$, $n$ and $d$ represent the batch size, number of similar entities, and the entity embedding dimensions. $|\mathcal{Q}|$ and $|\mathcal{E}|$ represent the number of entities and quadruples in the dataset respectively. 

| Methodology | Time Complexity | Memory Complexity | Per Epoch Runtime | Total Runtime |
|-----------|-----------|-----------|-----------|-----------|
| Naive Titer | - | - | ~9s | ~90m |
| Enhancement Layer | $\mathcal{O}(bnd)$ | $\mathcal{O}(bnd)$  | ~12s | ~125m|
| Weighted Sampling | $\mathcal{O}(n)$ | $\mathcal{O}(\|\mathcal{E}\|)$ | ~9s | ~110m |
| Full | $\mathcal{O}(bnd)$ | $\mathcal{O}(\|\mathcal{E}\| + bnd)$ | ~12s | ~125m|

Reviewer comment (Experiments):
While weighted sampling is an intuitive idea, the sampling function used could be further explored and justified. For example, analyzing alternative frequency-based formulas or evaluating sampling directly from a learned importance weighting.

Author response:
We thank the reviewer for this comment. We conducted experiments with various strategies, including mean, max, and min inverse frequencies of head and tail entities. During our hyperparameter selection process, the approach we ultimately chose was the one that yielded the best performance. In terms of exploring a learned importance weighting for each entity, we agree that this is a promising direction. We have mentioned this point in the future work section.

Reviewer comment (Evaluation):
In Figure 2.a, it seems that the proposed method (Full) is outperformed by the proposed method (Enhancement Layer) on HITS@10, and the proposed method (Full) is outperformed by the proposed method (Weighted Sampling), any insights on that?

Author response:
This is a great question. Our findings suggest that the enhancement layer improves performance for edges involving low and medium-degree entities, which aligns with its intended function. However, for edges involving higher-degree entities, the full methodology outperforms the individual components.

Reviewer comment (Experiments):
The comparison method is single. The author mentioned a lot of related work, but in the end only one method was selected as the basic model for the experiment. Such an experiment does not prove that the enhancement method proposed by the author is universal.

Author response:
> Q2. Have you tried more TKGC methods? For example, Cygnet, LCGE?
We are now adding another state-of-the-art model  TANGO [1] to the paper. Preliminary results show that even the addition of our enhancement layer alone improves the performance. The results are provided in the following table: 

|  Methodology |  | Current | | |  | Average | | |
|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|
|  | **Hit@10** | **Hit@3** | **Hit@1**| **MRR** | **Hit@10** | **Hit@3** | **Hit@1**| **MRR** |
TANGO |0.519 |0.355 |0.232 |0.317 |0.534 |0.395 |0.270 |0.359 | 
TANGO + Enhancement Layer |0.720 |0.555 |0.379 |0.496 |0.715 |0.557 |0.386 |0.498 |

Further details about how the enhancement layer was incorporated to TANGO is provided in our paper.


**References** 

[1] Han, Zhen, et al. "Learning neural ordinary equations for forecasting future links on temporal knowledge graphs." Proceedings of the 2021 Conference on empirical methods in natural language processing. 2021. [(Code)](https://github.com/TemporalKGTeam/TANGO)

Reviewer comment (Experiments):
Have you tried experimenting on more TKGs, such as WIKIDATA and GDELT?

Author response:
We appreciate the reviewer's time and insightful feedback. The reviewer has thoughtfully highlighted important considerations regarding the datasets: 
>Q1. Have you tried experimenting on more TKGs, such as WIKIDATA and GDELT?

While we agree that applying our approach to additional datasets would indeed be interesting, we were unfortunately unable to do so within the timeframe of this revision. However, it is important to note the inherent diversity of the two datasets currently included in our manuscript. Although ICEWS14 and ICEWS18 originate from the same ICEWS dataset, they are recognized as distinct benchmarks in the literature due to their differing distributions and densities. For example, ICEWS18 has 10 times more quadruples compared to ICEWS14 despite both of them covering a similar time duration within their respective years (see table below). The section 5.1 of our paper describes the dataset construction. We have now also revised this section to explain the dataset construction in more detail. The following table summarizes the statistics across these datasets:

| Dataset | \# Entities | \# Relations | \# Snapshots | \# Quads ($G_1$) | Ave. Quads ($G_{i> 1}$)|
|-----------|-----------|-----------|-----------|-----------|-----------|
| ICEWS14 | 7128 | 230 | 33 | ~28k/3.7k/4k| ~1k/0.3k/0.3k |
| ICEWS18 | 23039 | 230 | 16 | ~244k/45k/43k |  ~8K/1k/1k |

Reviewer comment (Experiments):
Have you tried more TKGC methods? For example, Cygnet, LCGE?

Author response:
> Q2. Have you tried more TKGC methods? For example, Cygnet, LCGE?
We are now adding another state-of-the-art model  TANGO [1] to the paper. Preliminary results show that even the addition of our enhancement layer alone improves the performance. The results are provided in the following table: 

|  Methodology |  | Current | | |  | Average | | |
|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|
|  | **Hit@10** | **Hit@3** | **Hit@1**| **MRR** | **Hit@10** | **Hit@3** | **Hit@1**| **MRR** |
TANGO |0.519 |0.355 |0.232 |0.317 |0.534 |0.395 |0.270 |0.359 | 
TANGO + Enhancement Layer |0.720 |0.555 |0.379 |0.496 |0.715 |0.557 |0.386 |0.498 |

Further details about how the enhancement layer was incorporated to TANGO is provided in our paper.


**References** 

[1] Han, Zhen, et al. "Learning neural ordinary equations for forecasting future links on temporal knowledge graphs." Proceedings of the 2021 Conference on empirical methods in natural language processing. 2021. [(Code)](https://github.com/TemporalKGTeam/TANGO)

Reviewer comment (Experiments):
Since global similarity is used to enhance the representation of entities, why not just cancel the incremental training setting and train the representations of all entities together from the beginning?

Author response:
>Q3. Since global similarity is used to enhance the representation of entities, why not just cancel the incremental training setting and train the representations of all entities together from the beginning?

To clarify, global similarity in our context, refers to identifying similar entities that may not be directly connected. It goes beyond immediate local proximity (e.g., one-hop neighborhoods) but does not imply full access to the entire graph structure from the beginning. Our data is received in a streaming fashion, and at each timestep, we compute similar entities based on the graph structure observed up to that point. The incremental training setting is essential because it allows our model to adapt to new information as it becomes available, reflecting the evolving nature of real-world data. Training all entity representations together from the beginning would require complete, upfront knowledge of the graph, which is not feasible in many practical TKG scenarios where data continuously evolves. Therefore, the incremental training approach, combined with global similarity, offers a more realistic and effective solution for TKG completion tasks, especially in dynamic environments where new entities and relationships are constantly emerging.

Reviewer comment (Theory):
Generally speaking, incremental training only reduces training costs and should not be more effective than training from scratch. If you want to use incremental training to enhance the effect of TKGC, is this starting point wrong?

Author response:
>Q4. Generally speaking, incremental training only reduces training costs and should not be more effective than training from scratch. If you want to use incremental training to enhance the effect of TKGC, is this starting point wrong?

Effectiveness can be defined differently depending on the application. In some scenarios, a model's performance on recent data is crucial, while in others, consistent performance across all tasks holds greater importance. When focusing on recent tasks, incremental training can potentially outperform a model trained from scratch, as it may be more biased to the latest data. Conversely, for overall performance across all tasks, a model trained on the entire dataset outperforms a fine-tuned model. However, this approach can be impractically inefficient for real-world applications, especially when dealing with large, continuously evolving datasets. Our goal is to strike a balance between maintaining high performance on recent tasks and overall robustness across all tasks, while also ensuring reasonable computational efficiency. Furthermore, beyond mere incremental fine-tuning, the overall performance can be enhanced through techniques such as Elastic Weight Consolidation (EWC), Experience Replay (ER), and our proposed enhancement layer.

Reviewer comment (Novelty):
In Table 1, many various recent works should be considered and discussed:  
Xu et al., 2023. Temporal knowledge graph reasoning with historical contrastive learning.  
Zhang et al., 2023. Learning Long- and Short-term Representations for Temporal Knowledge Graph Reasoning.  
Zhu et al., 2021. Learning from History: Modeling Temporal Knowledge Graphs with Sequential Copy-Generation Networks.

Author response:
We appreciate the reviewer's suggestion regarding the related work. To clarify, our experiments were designed to demonstrate the effectiveness of our framework and its ability to be used on top of any state-of-the-art TKG model. In Table 1, we compare the performance of a model before and after the application of our framework.  We have now added another state-of-the-art model  TANGO [1] to the paper. Preliminary results show that even the addition of our enhancement layer alone improves the performance. 

The results are provided in the following table: 

|  Methodology |  | Current | | |  | Average | | |
|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|
|  | **Hit@10** | **Hit@3** | **Hit@1** | **MRR** | **Hit@10** | **Hit@3** | **Hit@1** | **MRR** |
TANGO |0.519 |0.355 |0.232 |0.317 |0.534 |0.395 |0.270 |0.359 |
TANGO + Enhancement Layer |0.720 |0.555 |0.379 |0.496 |0.715 |0.557 |0.386 |0.498 |



**References** 

[1] Han, Zhen, et al. "Learning neural ordinary equations for forecasting future links on temporal knowledge graphs." Proceedings of the 2021 Conference on empirical methods in natural language processing. 2021. [(Code)](https://github.com/TemporalKGTeam/TANGO)

Reviewer comment (Novelty):
The motivation is not well established. It seems that the authors combine the two methods (global similarity and weighted sampling).

Author response:
We have modified the paper to better explain the motivation for our approach. Briefly, the motivation for integrating these two methods stems from their complementary strengths in addressing the challenges of Temporal Knowledge Graph (TKG) completion, including sparse or non-existent local connections and training biases resulting from unbalanced distribution. The enhancement layer addresses the issue of sparse or non-existent local connections for infrequent (long-tail) entities, an area where traditional TKG completion methods, which primarily relying on local neighborhood proximity for entity representation, fall short. On the other hand, weighted sampling ensures that long-tail entities are not overlooked by the model during training. Our ablation study, detailed in Section X of the paper, demonstrates how each component contributes to addressing these challenges. Figure 2.a illustrates the effectiveness of the enhancement layer for continual learning. As depicted in Figure 2.b, it also contributes to improving performance for entities with mid-degree nodes. The full method enhances hit@1 for low-degree nodes.

Reviewer comment (Novelty):
In Table 1, many various recent works should be considered and discussed:  
Xu et al., 2023. Temporal knowledge graph reasoning with historical contrastive learning.  
Zhang et al., 2023. Learning Long- and Short-term Representations for Temporal Knowledge Graph Reasoning.  
Zhu et al., 2021. Learning from History: Modeling Temporal Knowledge Graphs with Sequential Copy-Generation Networks.

Author response:
We appreciate the reviewer's suggestion regarding the related work. To clarify, our experiments were designed to demonstrate the effectiveness of our framework and its ability to be used on top of any state-of-the-art TKG model. In Table 1, we compare the performance of a model before and after the application of our framework.  We have now added another state-of-the-art model  TANGO [1] to the paper. Preliminary results show that even the addition of our enhancement layer alone improves the performance. 

The results are provided in the following table: 

|  Methodology |  | Current | | |  | Average | | |
|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|
|  | **Hit@10** | **Hit@3** | **Hit@1** | **MRR** | **Hit@10** | **Hit@3** | **Hit@1** | **MRR** |
TANGO |0.519 |0.355 |0.232 |0.317 |0.534 |0.395 |0.270 |0.359 |
TANGO + Enhancement Layer |0.720 |0.555 |0.379 |0.496 |0.715 |0.557 |0.386 |0.498 |



**References** 

[1] Han, Zhen, et al. "Learning neural ordinary equations for forecasting future links on temporal knowledge graphs." Proceedings of the 2021 Conference on empirical methods in natural language processing. 2021. [(Code)](https://github.com/TemporalKGTeam/TANGO)

Reviewer comment (Evaluation):
Another useful aspect on the proposed methodology is to add the run times and the time complexities when the layer and the sampling strategy are added.

Author response:
This is a great suggestion. We have added a table that reports the runtime and memory complexity of different variations of our model (weighted sampling, enhancement layer, full) in comparison with the naive Titer, as well as the actual per-epoch and total runtime of our method on the ICEWS14 dataset. The table is copied below for ease of access. 

**Notation**:  $b$, $n$ and $d$ represent the batch size, number of similar entities, and the entity embedding dimensions. $|\mathcal{Q}|$ and $|\mathcal{E}|$ represent the number of entities and quadruples in the dataset respectively. 

| Methodology | Time Complexity | Memory Complexity | Per Epoch Runtime | Total Runtime |
|-----------|-----------|-----------|-----------|-----------|
| Naive Titer | - | - | ~9s | ~90m |
| Enhancement Layer | $\mathcal{O}(bnd)$ | $\mathcal{O}(bnd)$  | ~12s | ~125m|
| Weighted Sampling | $\mathcal{O}(n)$ | $\mathcal{O}(\|\mathcal{E}\|)$ | ~9s | ~110m |
| Full | $\mathcal{O}(bnd)$ | $\mathcal{O}(\|\mathcal{E}\| + bnd)$ | ~12s | ~125m|