Paper: Recent Link Classification on Temporal Graphs Using Profile Builder (ICLR-2024)

Reviewer comment (Theory):
In the design of the profile encoder, the motivations for introducing metapaths are not explained. Metapaths are usually used in the graph representation learning of heterogeneous graphs, but the study seems to consider only (dynamic) homogeneous graphs. Additionally, metapaths are not illustrated in Fig. 1 (the model architecture).

Author response:
We will rework the problem statement to address the lack of clarity. Specifically, we will make clear that we are working in a continuous time setting, the source of our edge features, and the transductive nature of the problem.

Reviewer comment (Evaluation):
There are no intuitive motivations provided for the proposed metrics of edge homophily and Matthews correlation coefficient, for example, why Matthews correlation coefficient can handle the label-imbalanced issue. As a result, their advantages beyond conventional quality metrics are unclear.

Author response:
Metric uncertainty - We will add an illustrative example that highlights situations where MCC can correctly handle label imbalance but more traditional metrics cannot. In addition, we will leverage the edge homophily definition to explain aspects of our results, in analogy to the discussion of node homophily for node classification.

Reviewer comment (Experiments):
In the 1st paragraph of Section 3, the availability of graph attributes (e.g., inputs of node and edge features) is not mentioned. However, in the 2nd paragraph, edge attributes are treated as inputs of RLC. It is unclear whether graph attributes (node and edge attributes/features) are considered, and if so, whether they are assumed to be static across time steps or dynamic.

Author response:
We will rework the text to highlight the importance of our new task and more clearly explain how it differs from FLP and DNC; and we will use that to further motivate our discussion about the value of this new task. We hope that by addressing the metrics as we have planned in W1, and clearly explaining the value of RLC, we can address W4.

Reviewer comment (Writing):
The concept of 'Profile', which is significant in the proposed method (e.g., profile encoder, node profile), is not defined. There is no explanation of what profiles are in real dynamic graphs or in theoretical terms.

Author response:
We will clarify the definition of a profile in this setting, rather than depending on the colloquial definition.

Reviewer comment (Presentation):
According to Section 3, each edge in a (dynamic) graph should be associated with a time step; however, the graphs in Fig. 1 and Fig. 2 appear static. Furthermore, edge attributes are not illustrated in Fig. 1 and Fig. 2.

Author response:
The graphs in the figures are illustrative but they are dynamic based on the content of the containers. We will make this more clear in the text and figure captions

Reviewer comment (Experiments):
The experiments section is too simple, with only two baseline methods (TGN and GraphMixer), which is insufficient to validate the superiority of the proposed method. Other dynamic graph representation learning methods (e.g., TGAT, DySAT, EvolveGCN) mentioned in Section 2 could also be included.

Author response:
We will add two additional datasets, one based on video game reviews and one on NFT sales, and baselines from GraphMixer and TGAT. The NFT sales dataset is considerably larger than the ones in the current work.

Reviewer comment (Writing):
The overall presentation of the paper is poor, with inconsistent presentation noted earlier, as well as grammatical errors and typos that require careful revision, such as:  
1) 'analyze the temporal graph learning architectures divindign categorizing the methods literature into two groups'  
2) 'edges$\mathcal{E}$'  
3) 'we construct derived graphs that'  
4) 'connect a vertex acting that acts as a source to another that acts as a course through a shared destination vertex'  
5) 'on abuse-like like datasets'

Author response:
We thank the reviewer for pointing out typos and other places where the work lacks clarity. We will address those, and carefully edit the paper to catch others.

Reviewer comment (Novelty):
The reason why Graph Profiler performs better than TGN is unclear. The technical advancement of Graph Profiler is unclear.

Author response:
Our Graph Profiler method is novel in so far as, to our knowledge, the concept of building time-dependent profiles for entities based on based on the features and labels of their previous interactions  is not something we are aware of. In contrast, TGN learns a graph wide memory. We will update the text to make this point clear. Also, we believe that we have additional contributions, including the introduction of a new research task, an associated figure of merit, and baseline results. We will rework the text to further emphasize these contributions.

Reviewer comment (Experiments):
These critical design principles are different for different datasets, which makes it more like hyper-parameter tuning for specific datasets.

Author response:
On the topic of critical design principles, we will rework the text to emphasize that these principles comprise the design space for a family of models.

Reviewer comment (Experiments):
Extensive evaluation (e.g. larger datasets, other models) are needed for validating these critical design principles.

Author response:
On the topic of extensive evaluations being required, we will add two additional datasets, one based on video game reviews and one on NFT sales, and baselines from GraphMixer and TGAT. The NFT sales dataset is considerably larger than the ones in the current work.

Reviewer comment (Writing):
The motivation of the paper is unclear. If the authors want to highlight the proposed method, it might be better to explain how the design of the Graph Profiler algorithm incorporates these critical design principles.

Author response:
We have clarified the importance of the TGN modifications for motivating our design choices for our Graph Profiler model. This clarification can be found in the first paragraph of section 5.2. In summary, we explored modifications to TGN that borrowed specifics from other SOTA models, such as the fixed time encoder, to understand the importance of these model-features for our RLC task. Our development of Graph Profiler was informed by the important components of our modified TGN.

Reviewer comment (Novelty):
The model design is rather conventional and lacks novelty. It adheres to the traditional message-passing framework and introduces event and time-related elements as a simple extension.

Author response:
Our Graph Profiler method is novel in so far as, to our knowledge, the concept of building time-dependent profiles for entities based on based on the features and labels of their previous interactions  is not something we are aware of. Also, we believe that we have additional contributions, including the introduction of a new research task, an associated figure of merit, and baseline results. We will rework the text to further emphasize these contributions.

Reviewer comment (Evaluation):
While the proposed Matthews Correlation Coefficient is effective for assessing classification tasks, it may not fully account for the specific attributes of the problem, particularly in the context of temporal interaction classification. It remains unclear how well it aligns with the temporal and graph-based nature of the problem.

Author response:
We will add an illustrative example of a situation where MCC and AUC give conflicting results and provide a clear argument for MCC.

Reviewer comment (Novelty):
Is there any novelty in the method design, such as in a specific module or the whole framework?

Author response:
Our Graph Profiler method is novel in so far as, to our knowledge, the concept of building time-dependent profiles for entities based on based on the features and labels of their previous interactions  is not something we are aware of. Also, we believe that we have additional contributions, including the introduction of a new research task, an associated figure of merit, and baseline results. We will rework the text to further emphasize these contributions.

Reviewer comment (Evaluation):
Is there any specific design of the metric for temporal interaction classification?

Author response:
Additionally, we would appreciate if you could expand a little further on your question: “Is there any specific design of the metric for temporal interaction classification?” We are planning to answer it by expanding the MCC definition to explain that it applies in multi-class and imbalanced settings, but we are uncertain if this is a satisfactory answer for your question

Reviewer comment (Reproducibility):
What is the time encoder like?

Author response:
We will add more information about the time-encoder, and more generally we thank the reviewer for pointing this areas where the methods discussion is unclear, and we will work to improve that.

Reviewer comment (Writing):
The technical details on the graph profiler are hard to follow, some notation is missing or assumed. At the same time the authors explain more basic concepts such as edge homophily using more verbage.

Author response:
We thank the reviewer for pointing this areas where the methods discussion is unclear, and we will work to improve that. We will do so specifically by expanding upon details of our model, dimensions of matrices, and other model details; as well as the way in which we’ve constructed the dataset.

Reviewer comment (Evaluation):
The evaluation (since it is a novel task) is a bit weak. However, the analysis can be supplemented with more creative ways of analyzing the performance even if comparison to other algorithms is not possible.

Author response:
We will add 2 additional datasets and additional benchmark results to make the analysis a bit more involved.

Reviewer comment (Writing):
The edge homophily paragraph is dense with notation, which makes the formula hard to digest even though the idea is simple. Please include a sentence (in English) to supplement the formula when defining edge homophily, e.g. "the fraction of edges that connect nodes of the same class".

Author response:
We will expand our definition of edge homophily to be easier to understand.

Reviewer comment (Writing):
Please define the matrices you use (and their dimensions), they are sometimes only understood from the context, e.g. in the Profiler Encoder section on p.4.

Author response:
We thank the reviewer for pointing this areas where the methods discussion is unclear, and we will work to improve that. We will do so specifically by expanding upon details of our model, dimensions of matrices, and other model details; as well as the way in which we’ve constructed the dataset.

Reviewer comment (Theory):
How did you derive the formula for d_1, bottom of p. 4?

Author response:
We will review the rest of the paper to ensure that formulas like that for $d_1$ are motivated and clear.

Reviewer comment (Writing):
Some typos need to be fixed: e.g.  
- p.2 "...temporal graph learning architectures divinding categorizing..."  
- p.4 "...In our specific instance... acting that acts..."

Author response:
We will address the typos through a thorough editing process.

Reviewer comment (Experiments):
What is the significance of the TGN modifications in Table 2? They don't seem to be directly related to the proposed method.

Author response:
We have clarified the importance of the TGN modifications for motivating our design choices for our Graph Profiler model. This clarification can be found in the first paragraph of section 5.2.

Reviewer comment (Evaluation):
The modeling decisions, such as the choice between learnable or fixed time encoding, appear ad-hoc and contingent on specific datasets. It would be beneficial to elucidate insights or provide general guidance for determining an optimal combination on new datasets. For instance, what factors contribute to the observed performance variation, and is there a rationale for the less effective performance of learnable time-encoding on the Wikipedia dataset?

Author response:
- We will work to understand the structure of the datasets under investigation and how their structure impacts the time encoding.

Reviewer comment (Experiments):
Given that the src-dst-msg-t configuration doesn't generally yield the best results, I am wondering about the necessity of introducing seemingly redundant components like the destination encoder. Additionally, the observed performance degradation in cases where time encoding is added to src-dst raises questions. Is there any explanation for that?

Author response:
- Performed an ablation study to understand the role of destination embeddings, which can be found in Appendix C. We observe that in 5/6 datasets, the destination embeddings help, and we present an explanation for why this would be true.  
- We have explored the structure of the datasets and unfortunately have no explanation for the importance of the time encodings in the TGN framework. We do want to make clear, however, that the Graph Profiler model only ever uses learnable time encodings.

Reviewer comment (Novelty):
The presented task bears similarities to entity relationship classification in NLP. It would be interesting to discuss the similarities and differences between these two tasks.

Author response:
- We will explore the link between RLC and dynamic KG relationship classification. We thank the reviewer for pointing this connection out as the authors are unfamiliar with this literature.

Reviewer comment (Novelty):
The proposed method mainly combines several existing methods together, which makes the technical contribution of method design not very high.

Author response:
- Our Graph Profiler method is novel in so far as, to our knowledge, the concept of building time-dependent profiles for entities based on the features and labels of their previous interactions is not something we are aware of. Also, we believe that we have additional contributions, including the introduction of a new research task, an associated figure of merit, and baseline results. We will rework the text to further emphasize these contributions.

Reviewer comment (Writing):
The paper contains several typos, and certain sentences are challenging to comprehend.

Author response:
- We have given the paper a thorough edit, and we will perform one more pass of editing on Monday to catch any last typos.

Reviewer comment (Presentation):
The captions in the graph are too small to read.

Author response:
- To address the figure captions, we will remake the figures such that they are easier to read.