Paper: Double Equivariance for Inductive Link Prediction for Both New Nodes and New Relation Types (ICLR-2024)

Reviewer comment (Evaluation):
The proposed framework requires significant preprocessing and expensive encoding costs. This may be attributed to three factors: preprocessing costs, encoding for each relation, and separate scoring for each candidate entity.

Author response:
**A1:** In Appendix F.2 and G, we have discussed the limitation of our proposed ISDEA architecture due to its computation cost. The primary source of complexity in ISDEA is the process of enumerating across all relations and computing distances between every pair of nodes. However, we believe separate scoring for each candidate entity is commonly used by existing literatures such as DistMult (Yang et al., ICLR 2015), RGCN (Schlichtkrull et al., ESWC 2018), GraIL (Teru et al., ICML 2020), etc..

To address this, as discussed in the Common Reviewer Response, we have developed ISDEA+, which improves the time complexity by $R$ over ISDEA (where $R$ is the number of relation types). In our experiments, we observe ISDEA+ is between 20x to 120x faster than ISDEA (old).

Reviewer comment (Evaluation):
The 1 vs. 50 evaluation poses a risk as negative samples obtained from negative sampling are mostly easily distinguishable. This setup may not be sufficient to cover real-world scenarios.

Author response:
**A2:** The choice was purely due to computational constraints of our proposed ISDEA (old) since we wanted at least 5 runs for each experiment. We have simply followed the procedure proposed by GraIL (Teru et al., ICML 2020), which is among the first works to apply GNNs for inductive link prediction over new nodes (as GraIL, we perform evaluation against 50 random negative samples). This choice allows some balance between computational cost and meaningful performance insights. 

In our rebuttal, we introduced an improved version of ISDEA, named ISDEA+. This enhancement allows us to handle larger negative samples during evaluation. Here are the results we have managed to gather in the limited time available. In particular, we rerun the node prediction task on all tasks of PediaTypes with 500 random negative node samples, and collected the Hits@10 results in the following table:

| Models     | EN-FR          | FR-EN          | EN-DE          | DE-EN          | DB-WD           | WD-DB          | DB-YG          | YG-DB           |
|------------|----------------|----------------|----------------|----------------|-----------------|----------------|----------------|-----------------|
| NBFNet     | $65.73\pm3.17$ | $73.36\pm1.84$ | $58.93\pm2.89$ | $32.15\pm3.14$ | $33.75\pm2.87$  | $61.64\pm1.36$ | $34.07\pm1.49$ | $51.00\pm1.90$ |
| InGram     | $67.42\pm2.42$ | $48.76\pm8.10$ | $46.84\pm4.07$ | $54.08\pm1.35$ | $24.87\pm10.38$ | $49.41\pm5.41$ | $15.69\pm4.11$ | $23.73\pm7.12$  |
| DEq-InGram | $76.33\pm1.27$ | $64.84\pm6.17$ | $59.67\pm2.43$ | $66.30\pm2.42$ | $35.70\pm12.10$ | $65.55\pm4.04$ | $24.11\pm4.73$ | $37.76\pm12.01$ |
| ISDEA+     | $68.27\pm3.62$ | $50.76\pm2.57$ | $66.83\pm0.48$ | $53.52\pm0.67$ | $56.97\pm0.85$  | $65.75\pm0.78$ | $20.72\pm0.35$ | $26.67\pm1.76$  |


From these results, we can observe that even with an enlarged number of negative samples, we still observe the same model behaviors as negative sample size 50. The NBFNet model is still able to achieve relatively good performance in the node prediction task while its model architecture is not able to capture the relationship among the new relation types. Both of our proposed models, ISDEA+ (and DEq-InGram), still achieve better performance than the baseline NBFNet and InGram in most scenarios.

Reviewer comment (Evaluation):
While ISDEA appears suitable for relation prediction, its performance on node prediction is not very good.

Author response:
**A3:** Please see new results of ISDEA+ in Section 5.1 of the revised manuscript, which now outperforms DEq-InGram in nearly all scenarios. The improvement in performance of DEq-InGram over InGram confirms our theory that double equivariance allows neural networks to perform zero-shot double inductive link prediction tasks.

PS: The node prediction performance aligns with findings from Jambor et al., EACL 2021, where a model's success in node prediction tasks may not fully reflect its capabilities in predicting the type of link, but simply that of being able to predict homogeneous links (in a way agnostic to its type). For instance, RMPI and NBFNet have good performance at predicting nodes but significantly worse performance at predicting edge types.

Reviewer comment (Evaluation):
I am concerned about the efficiency of the proposed framework. Could you report training and inference times on some datasets?

Author response:
**A4:** Here we present a run time comparison for ISDEA+, ISDEA, DEq-InGram, InGram and NBFNet on the node prediction task on EN-FR in our PediaTypes dataset, and a much larger transductive FB15K-237 dataset. Due to varied mini-batch training sizes for different models, we report the training time per minibatch step (i.e., per gradient update step) for the training time comparison. Here are the results:

**Training and inference on the EN-FR in PediaTypes**
- 4962 entities, 122 relations, 30876 train triplets, 3326 validation triplets, 2485 test triplets

| Model      | Training Time per Minibatch Step | Test Inference Time | Max GPU Memory Usage |
|------------|----------------------------------|---------------------|----------------------|
| ISDEA+     | 0.05s                            | 6.53s               | 1.9GB                |
| ISDEA      | 2.56s                            | 359.45s             | 9.8GB                |
| Deq-InGram | 0.09s                            | 10.39s              | 3.0GB                |
| InGram     | 0.09s                            | 5.60s               | 3.0GB                |
| NBFNet     | 0.02s                            | 5.14s               | 5.5GB                |

**Training and inference on the transductive FB15k-237 dataset**
- 14541 entities, 237 relations, 272115 train triplets, 17535 validation triplets, 20466 test triplets

| Model      | Training Time per Minibatch Step  | Test Inference Time | Max GPU Memory Usage |
|------------|-----------------------------------|---------------------|----------------------|
| ISDEA+     | 0.08s                             | 27.60s              | 5GB                  |
| ISDEA      | N/A                               | N/A                 | OOM                  |
| Deq-InGram | 0.57s                             | 99.67s              | 13GB                 |
| InGram     | 0.57s                             | 62.60s              | 13GB                 | 
| NBFNet     | 0.07s                             | 62.23s              | 14GB                 |

We can see that ISDEA+ significantly improves ISDEA, and is among the fastest models with least memory consumption.

Reviewer comment (Experiments):
It should be clarified that the multilingual KGs in the OpenEA library share the same schema. So many of the relations in these KGs overlap.

Author response:
**A5:** We appreciate your observation on this aspect. Indeed, the multilingual KGs within the OpenEA library consist of several pairs of knowledge graphs, each representing similar domains but in different languages. For example, entities like “http://dbpedia.org/resource/E678522” in English and “http://fr.dbpedia.org/resource/E415873” in French represent the same concept, although with different linguistic labels. It is accurate that these KGs predominantly use English for relation labels, which causes an overlap in relations. However, in our experimental setup, we treat relations as if they were in different languages and do not leverage this overlapping information during model training. Moreover, in other KGs used in our PediaTypes and WikiTopics datasets, such overlapping relations are not present. We have included this clarification in the revised submission, in the description of our datasets. Thank you for pointing this out.

Reviewer comment (Theory):
The formalization in Section 2 assumes the existence of bijections (nodes-to-nodes, relations-to-relations) in training and test graphs. Basically, the framework posits the double equivariance only when training and test graphs have exactly the same number of nodes and edge types - which practically does not happen very often. On the other hand, the constructed datasets PediaTypes and WikiTopics all have different numbers of nodes and relations at training and test time (so there is no bijection possible). Could you please comment on the seeming discrepancy between the theory and what is measured in the experiments?

Author response:
Thank you for the insightful question, which allows us to clarify our method (we added a comment to the revised submission). In all our WikiTopics datasets the training and test datasets have **different** number of relations (Figure 6 in the Appendix F.1.3). ISDEA+ works with a different number of relations between train and test. There is no need to have a bijective mapping between the relations in training and test. GNNs are node-equivariant and work on test graphs with varying numbers of nodes. ISDEA+ is double (relation+node) equivariant and works on test graphs with varying numbers of nodes and relation types. Isomorphic pairs are used to help us understand the property of ISDEA+, but ISDEA+ can be applied to non-isomorphic graphs. In standard graph tasks, a GNN tends to make similar embeddings for nodes that have similar structure (they need not be isomorphic) as long as doing that does not negatively affect the prediction task. And we showcase ISDEA+ also has such capabilities.

Reviewer comment (Evaluation):
Section 5.2: “_relatively easier task of node prediction_” - I do not quite agree with this statement. The results might suggest the task is easier simply because you take 50 random negatives among _thousands_ of nodes in the inference graph, so those negatives are likely to be _easy_ negatives. On the other hand, the number of relations in the datasets is 50-150 in PediaTypes and <50 in WikiTopics, so the negative relation samples are likely to be harder. It was found that evaluation on small number of negative entities overestimates the performance, so I would hypothesize the numbers (and task impression) would change when the architecture would scale to ranking all nodes in the inference graph.

Author response:
Thank you for raising this important point! That is true. We will qualify our point, which is essentially similar to Jambor et al., EACL 2021, where a model's success in node prediction tasks may not fully reflect its capabilities in predicting the type of link, but simply that of being able to predict homogeneous links (in a way agnostic to its type). For instance, RMPI and NBFNet have good performance at predicting nodes but significantly worse performance at predicting edge types. 

In our rebuttal, we introduced an improved version of ISDEA (old), named ISDEA+. This enhancement allows us to handle larger negative samples during evaluation. Here are the results we have managed to gather in the limited time available. In particular, we rerun the node prediction task on all tasks of PediaTypes with 500 random negative node samples, and collected the Hits@10 results in the following table:


| Models     | EN-FR          | FR-EN          | EN-DE          | DE-EN          | DB-WD           | WD-DB          | DB-YG          | YG-DB           |
|------------|----------------|----------------|----------------|----------------|-----------------|----------------|----------------|-----------------|
| NBFNet     | $65.73\pm3.17$ | $73.36\pm1.84$ | $58.93\pm2.89$ | $32.15\pm3.14$ | $33.75\pm2.87$  | $61.64\pm1.36$ | $34.07\pm1.49$ | $51.00\pm1.90$ |
| InGram     | $67.42\pm2.42$ | $48.76\pm8.10$ | $46.84\pm4.07$ | $54.08\pm1.35$ | $24.87\pm10.38$ | $49.41\pm5.41$ | $15.69\pm4.11$ | $23.73\pm7.12$  |
| DEq-InGram | $76.33\pm1.27$ | $64.84\pm6.17$ | $59.67\pm2.43$ | $66.30\pm2.42$ | $35.70\pm12.10$ | $65.55\pm4.04$ | $24.11\pm4.73$ | $37.76\pm12.01$ |
| ISDEA+     | $68.27\pm3.62$ | $50.76\pm2.57$ | $66.83\pm0.48$ | $53.52\pm0.67$ | $56.97\pm0.85$  | $65.75\pm0.78$ | $20.72\pm0.35$ | $26.67\pm1.76$  |


From these results, we can observe that even with an enlarged number of negative samples, we still observe the same model behaviors as negative sample size 50. The NBFNet model is still able to achieve relatively good performance in the node prediction task while its model architecture is not able to capture the relationship among the new relation types. Both of our proposed models, ISDEA+ (and DEq-InGram), still achieve better performance than the baseline NBFNet and InGram in most scenarios.

Reviewer comment (Reproducibility):
What are the input features to standard GNN architectures reported in the experiments under GraphConv / GAT / GIN? Initialization of nodes with all ones or with random vectors?

Author response:
Thank you for this important question. We discussed this at the bottom of page 5: since our graphs in PediaTypes and WikiTopics do not have node features, we initialize them as all-one features to feed into all model architectures.

Reviewer comment (Theory):
Since DEq-InGram is distributionally double equivariant (by means of averaging several runs with different random relation vectors initializations), would averaging NBFNet results across several runs with random relation initialization count as distributionally double equivariant as well?

Author response:
The short answer is yes, but the model would essentially only be able to predict that a relation exists between u and v, not the type of relation. Our results with NBFNet show essentially that. The averaged NBFNet can predict a link exists (node prediction) but can't predict the relation type.

Reviewer comment (Writing):
The writing of the paper is problematic and concepts are often unclear. The text is very repetitive and contains many redundancies (i.e., the contribution of the paper is highlighted three times in the first page with paraphrased sentences), but when it comes to formal definitions, it does not make a rigorous treatment.

Author response:
**A1:** Thank you for the feedback. We have revised the introduction to make it less repetitive.

Reviewer comment (Presentation):
Figure 1 is crowded and does not explain much. It is unclear why the relations are typed using conjunctions at this point, and how the logical description given in the beginning of page 3 corresponds to this figure.

Author response:
**A2:** Great feedback! Thank you! In the revised version we have changed Figure 1 to help clarify the task. Now it shows the graph being trained solely on the Sports Wikipedia KG and tested on the Corporations Wikipedia KG. Note that the relations observed on the domain Sports are all different from the relations on the domain Corporations. This pre-trained zero-shot task better illustrates our task. We also point the reviewer to the updated benchmark task in Section 5.2, where we describe our newly proposed pre-trained zero-shot meta-learning task. We relegated the logical connections to Appendix B, where we discuss the connection between ISDEA+ and universally quantified Horn clauses. We removed the connection to UQER formulas from the main paper to improve clarity.

Reviewer comment (Theory):
The naming of "doubly inductive" is somewhat problematic, because the inductive prediction is either on the relation or on one of the entities at a time, but not both according to Definition 1.

Author response:
**A4:** Thank you for the feedback. We believe there is a misunderstanding of the term "doubly inductive link prediction task" (Definition 2.1). The inductive setting of our task aims to predict missing links in test graphs with both completely new nodes and completely new relation types, as seen in our PediaType and WikiTopic datasets (and illustrated in Figure 1). We have debated whether the term *fully inductive* would be more adequate, but we can see that also leading to even further potential misunderstandings. The task is expressed mathematically that the set of training and test nodes are disjoint, $\mathcal{V}^{(te)} \cap \mathcal{V}^{(tr)} = \emptyset$, and that the set of training and test relation types are disjoint $\mathcal{R}^{(te)} \cap \mathcal{R}^{(tr)} = \emptyset$. Perhaps the misunderstanding originates from our last sentence in Definition 2.1, “we aim to predict both missing relations for the given head and tail nodes $(i, ?, j)$ and missing nodes for a given relation $(i, k, ?)$.", which we now removed in the revision to avoid this misunderstanding.

Reviewer comment (Novelty):
The new architectures introduced in the paper appear to be somewhat incremental. IDSEA is a variant of DSS-GNN operating on relation-induced subgraphs, whereas DEq-InGram is a simple modification of InGram with bagging.

Author response:
**A6:** Thank you for your feedback. It seems that there is a misunderstanding. The original DSS-GNN is not double equivariant, since it is a method for creating representations for bags of subgraphs over a single group equivariance (node permutations). Moreover, we have now made the method ISDEA+ between 20x and 120x faster, which took significant effort. We also show that double equivariance is a key property for the task of zero-shot link prediction without side information. Inspired by one of Reviewer’s VGiN questions, we added new experimental results to the main paper, where we expanded our proposed benchmark to better explain the power of double equivariant models in a self-supervised pre-trained zero-shot meta-learning task. Please refer to ''Common Reviewer Response'' and for Section 5.2 and Appendix F.1.3 in the revised submission.  

There is also a small misunderstanding about DEq-InGram: We perform Monte Carlo averaging, not bootstrapping aggregation (the averaging is not over bootstrapping). This averaging provably makes the model double equivariant, which significantly improves its performance. The new results show that ISDEA+ is still significantly more consistent and better than even DEq-InGram. We have made this point more salient in the text.

Reviewer comment (Evaluation):
IDSEA seems to perform consistently worse than DEq-InGram in the task of node prediction on PediaTypes, which does not seem to match what the theory suggests and is not discussed in the paper.

Author response:
**A7:** Please see new results of ISDEA+ in Section 5.1 of the revised manuscript, which now outperforms DEq-InGram in nearly all scenarios. The improvement in performance of DEq-InGram over InGram confirms our theory that double equivariance allows neural networks to perform zero-shot double inductive link prediction tasks.

PS: The node prediction performance aligns with findings from Jambor et al., EACL 2021, where a model's success in node prediction tasks may not fully reflect its capabilities in predicting the type of link, but simply that of being able to predict homogeneous links (in a way agnostic to its type). For instance, RMPI and NBFNet have good performance at predicting nodes but significantly worse performance at predicting edge types.

Reviewer comment (Experiments):
The paper predominantly focuses on scenarios where the train and test graphs share a similar distribution. However, there exists a range of tasks involving unseen nodes and relations where the distribution significantly differs between the training and testing phases. Further experimental validation on these tasks is required.

Author response:
**A8:** Thank you for your question. We are uncertain what the reviewer means by "similar distributions". Assuming the reviewer is referring to the distribution of graphs, the WikiTopics dataset presents a scenario where we intentionally train the model on graphs from one domain and test it on graphs from a completely different domain, such as training on Art and testing on Sports. Arts and Sports not only have nearly no common relation types, they also have a different number of relation types. Hence, their distribution should be different. Please let us know if this answers your concerns.

Reviewer comment (Experiments):
In the experiments, the authors do not compare with standard relational GNNs such as RGCN, CompGCN, NBFNets, etc.

Author response:
**A9:** We thank the reviewer for pointing out these works, but again there seems to be a misunderstanding that we hope will be cleared with the new Figure 1 in the revised submission. Please note that RGCN, CompGCN and NBFNet cannot perform the task where they are trained on, say, Arts domains, and as asked to zero-shot predict links on, say, Sport domains. Please refer to the second paragraph in Section 4 and Appendix E for the detailed distinction between the GNN-based inductive approaches and our method. We have added CompGCN in our related work section in the appendix.

The doubly inductive link prediction task (Definition 2.1) involves both entirely new nodes and new relation types in the test graph. In contrast, the mentioned papers concentrate on different tasks. Specifically, RGCN and CompGCN were initially designed for transductive tasks, while NBFNet is designed for inductive link prediction for only new nodes. Following InGram, we modified the SOTA relational GNN model NBFNet with minimal changes to handle unseen relation types at test time, whose results are shown in Table 1 and Appendix F.1.2. The results demonstrate that our proposed method significantly outperforms the adaptation of NBFNet used by InGram, highlighting the unique challenges and efficiency of our approach in this task setting.

Reviewer comment (Evaluation):
The differences between ISDEA, DEq-InGram, and InGram in terms of their runtime are not clarified.

Author response:
**A10:** Here we present a run time comparison for ISDEA+, ISDEA, DEq-InGram, InGram and NBFNet on the node prediction task on EN-FR in our PediaTypes dataset, and a much larger transductive FB15K-237 dataset. Due to varied mini-batch training sizes for different models, we report the training time per minibatch step (i.e., per gradient update step) for the training time comparison. Here are the results:

**Training and inference on the EN-FR in PediaTypes**

-   4962 entities, 122 relations, 30876 train triplets, 3326 validation triplets, 2485 test triplets
   
| Model | Training Time per Minibatch Step | Test Inference Time | Max GPU Memory Usage |
|------------|----------------------------------|---------------------|----------------------|
| ISDEA+ | 0.05s | 6.53s | 1.9GB |
| ISDEA(old) | 2.56s | 359.45s | 9.8GB |
| Deq-InGram | 0.09s | 10.39s | 3.0GB |
| InGram | 0.09s | 5.60s | 3.0GB |
| NBFNet | 0.02s | 5.14s | 5.5GB |

**Training and inference on the transductive FB15k-237 dataset**

-   14541 entities, 237 relations, 272115 train triplets, 17535 validation triplets, 20466 test triplets
    
| Model | Training Time per Minibatch Step | Test Inference Time | Max GPU Memory Usage |
|------------|-----------------------------------|---------------------|----------------------|
| ISDEA+ | 0.08s | 27.60s | 5GB |
| ISDEA(old) | N/A | N/A | OOM |
| Deq-InGram | 0.57s | 99.67s | 13GB |
| InGram | 0.57s | 62.60s | 13GB |
| NBFNet | 0.07s | 62.23s | 14GB |

We can see that ISDEA+ significantly improves ISDEA, and is among the fastest model with least memory consumption.

Reviewer comment (Evaluation):
Since both DEq-InGram and InGram produce distributionally double equivariant representations, it is unclear why there is a substantial performance gap between these models on both datasets.

Author response:
**A11:** There appears to be a misunderstanding. InGram is a distributionally double equivariant representation (as per Lemma 3.2) and is sensitive to permutations. In contrast, DEq-InGram, is an (approximate) double-equivariant representation constructed based on Theorem 2.8, which is designed to be (approximately) permutation invariant. This difference in handling permutations contributes to the substantial performance gap between the two models. These results underscore the significance of double equivariant representations in doubly inductive link prediction tasks. Still, the new ISDEA+ consistently outperforms InGram and DEq-InGram in the revised submission.