From: NeurIPS.cc/2022/Conference/Program_Chairs
title: Paper Decision
decision: Accept

From: NeurIPS.cc/2022/Conference/Paper10172/Area_Chair_izqM
metareview: the paper proposes ReFactorGNNs, a unified view on factorization-based models (FMs) and graph neural networks (GNNs). It shows how to how FMs can be cast as GNNs by reformulating the gradient descent
procedure as message-passing operations. The reviewers agree that the idea is "intuitive and yet very clever at the same time", as one of the reviewers puts it. Connecting FMs just as DistMul with GNNs is great, and the paper is well written and presented. The one more negative reviewer expresses a low confidence and provides an informed outsider view on the paper, which is actually rather positive. 
I fully agree though I would also like to provide some more links to related work:
https://arxiv.org/abs/1509.08535
https://forum.stanford.edu/events/posterslides/MessagePassingforMatrixFactorization.pdf
http://pfister.ee.duke.edu/papers/Kim-istc10.pdf
Moreover, another indication is that transformers are quite expressive, and transformers are GNNs. I I would like to encourage the authors to provide a more high-level view on the problem, and then argue that the presented approach makes this intuition very concrete. Moreover, it would be great to discuss more future work. For instance, it seems possible now to learn new FM approaches automatically from data. Kind of "FM discovery" or "Learning to Factorize". This is all very exciting. 
recommendation: Accept
confidence: Certain
award: No

From: NeurIPS.cc/2022/Conference/Paper10172/Authors
title: Thank you!
comment: We thank the reviewer for their time, and we really appreciate the increment in the rating. If there is anything that the reviewer feels is still a gap, we are happy to discuss further!

From: NeurIPS.cc/2022/Conference/Paper10172/Authors
title: Thank you!
comment: We thank the reviewer for their time, and we are very grateful for the increment in the confidence score.

From: NeurIPS.cc/2022/Conference/Paper10172/Reviewer_jDqr
title: Thank you for the answers
comment: I would like to thank the authors for their thoughtful and detailed comments. I am satisfied with the responses and happy to raise my rating to 6.

From: NeurIPS.cc/2022/Conference/Paper10172/Reviewer_rRLn
title: Thank You
comment: The authors answer the questions posed by the original review in great detail. While the given score, which was Accept, remains, the confidence into the validity of the review and its acceptance were increased following this rebuttal.

From: NeurIPS.cc/2022/Conference/Paper10172/Authors
title: Thank you
comment: We would like to thank all three reviewers for their time and efforts and will address the comments below.

From: NeurIPS.cc/2022/Conference/Paper10172/Authors
title: Response to Reviewer jDqr complexity comparison
comment: Extending the complexity analysis in NBFNet [2] to all the triplets, we include a detailed table for complexity comparison here. We will also include it in our final version. 


|                                  | Parameter Complexity | Memory Complexity |                                                | Time Complexity                  |                                                |
| -------------------------------- | -------------------- | ----------------- | ---------------------------------------------- |:-------------------------------- | ---------------------------------------------- |
|                                  |                      | Training          | Inference (Per Forward Pass Over Entire Graph) | Training                         | Inference (Per Forward Pass Over Entire Graph) |
| GAT                              | $O(Ld^2)$            | $O(L\|V\|d)$      | $O(L\|V\|d)$                                   | $O(L\|V\|d^2 + L\|T\|d)$         | $O(L\|V\|d^2 + L\|T\|d)$                       |
| RGCN                             | $O(L\|R\|d^2)$       | $O(L\|V\|d)$      | $O(L\|V\|d)$                                   | $O(L\|T\|d^2 + L\|V\|d^2)$       | $O(L\|T\|d^2 + L\|V\|d^2)$                     |
| NBFNet (Pairwise Representation) | $O(L\|R\|d^2)$       | $O(L\|V\|\|T\|d)$ | $O(L\|V\|\|T\|d)$                              | $O(L\|T\|^2d + L \|T\|\|V\|d^2)$ | $O(L\|T\|^2d + L \|T\|\|V\|d^2)$               |
| ReFactorGNN                      | $O(\|R\|d)$          | $O(\|V\|d)$       | $O(L\|V\|d)$                                   | $O(\|T\|\|V\|d)$                 | $O(L\|T\|\|V\|d)$                              |

From: NeurIPS.cc/2022/Conference/Paper10172/Authors
title: Response to Reviewer rRLn -- Complexity comparison table
comment: Extending the complexity analysis in NBFNet [6] to all the triplets, we include a detailed table for complexity comparison here. We will also include it in our final version. 

|                                  | Parameter Complexity | Memory Complexity |                                                | Time Complexity                  |                                                |
| -------------------------------- | -------------------- | ----------------- | ---------------------------------------------- |:-------------------------------- | ---------------------------------------------- |
|                                  |                      | Training          | Inference (Per Forward Pass Over Entire Graph) | Training                         | Inference (Per Forward Pass Over Entire Graph) |
| GAT                              | $O(Ld^2)$            | $O(L\|V\|d)$      | $O(L\|V\|d)$                                   | $O(L\|V\|d^2 + L\|T\|d)$         | $O(L\|V\|d^2 + L\|T\|d)$                       |
| RGCN                             | $O(L\|R\|d^2)$       | $O(L\|V\|d)$      | $O(L\|V\|d)$                                   | $O(L\|T\|d^2 + L\|V\|d^2)$       | $O(L\|T\|d^2 + L\|V\|d^2)$                     |
| NBFNet (Pairwise Representation) | $O(L\|R\|d^2)$       | $O(L\|V\|\|T\|d)$ | $O(L\|V\|\|T\|d)$                              | $O(L\|T\|^2d + L \|T\|\|V\|d^2)$ | $O(L\|T\|^2d + L \|T\|\|V\|d^2)$               |
| ReFactorGNN                      | $O(\|R\|d)$          | $O(\|V\|d)$       | $O(L\|V\|d)$                                   | $O(\|T\|\|V\|d)$                 | $O(L\|T\|\|V\|d)$                              |

From: NeurIPS.cc/2022/Conference/Paper10172/Authors
title: Response to Reviewer M1H8
comment: 

**Significance**: 

We appreciate that your assessment about the significance of this work stems from your view on the whole field of KG research, not our contribution itself. However, of course we would make a different bet, and put our money confidently on option D—as confident as one can ever be about what research will look like in 5 years, seeing massive changes in the last two years alone. 

Our guess relies on the fact that we think our work is both of high practical significance as it provides a simple recipe for strong empirical results, and opens up a channel for further insights and innovations by linking two important approaches to reasoning with graphs. It also relies on our expectation that KGs will remain highly relevant in the foreseeable future. There are different ways we use to arrive at this.

First, we think there are clear use cases of KGs that will not go away, regardless of progress in LMs. A highly structured and normalised representation of data (i.e. triplets in a KG) has emerged in many fields as a natural choice, as it lends itself to easy visualisation, human intervention and querying. Example domains are Biology, Medicine, Recommender Systems, Finance and Legal Services [1-5].

Second, we see signals in industry such as the recent series B round of RelationAI [6] that there still exists an ample (and possibly increasing) appetite for Knowledge Graphs in practice. Of course, we are seeing even more funding and attention channelled into LM companies, but our claim is not that KG research will match the current and expected future impact of LM research, but that KG research will remain a significant and important area of the AI landscape.     

Third, we expect to see many synergies between KG and LM research in the future, and some work in either field to become useful for the other. For this perspective, it’s useful to consider that both language models and KG models are effectively models of symbol tuples, one of length 3 and the other of variable lengths. As such, findings in the more controlled 3-tuple setting might become useful in understanding increasingly complex natural language models, retrieval architectures in NLP can be integrated into KG methods etc. So even if there was a doom of KGs, insights of papers in KG research may hence still become impactful.   

Fourth, from a purely bibliographic point of view, we can’t see signs of slowdown. While these are crude metrics, looking at, say, the number of mentions of “knowledge graph” on google scholar over time, or the number of citations for standard papers in the area (like ConvE), suggests an increasing interest in the field. It is possible that, even if we don’t have a trend now, it might emerge any time from now until the end of that 5 year window (although the closer it is to the end of that window, the less effect the doom of the area will have on the expected citation count by then). But it is worthwhile pointing out that the rise of language models as well as retrieval augmentation is already a bit older (with GPT3 released in 2020 and RAG in the same year, for example).

And lastly, even if we assume that KGs are doomed, our proposed ReFactorGNN can be applied to other domains with multi-relational graphs, and we are confident that multi-relational graphs are here to stay.  


[1] Industry-scale Knowledge Graphs: Lessons and Challenges. https://dl.acm.org/doi/10.1145/3331166

[2] Toward better drug discovery with knowledge graph. https://pubmed.ncbi.nlm.nih.gov/34649044/

[3] A Literature-Based Knowledge Graph Embedding Method for Identifying Drug Repurposing Opportunities in Rare Diseases. https://pubmed.ncbi.nlm.nih.gov/31797619/

[4] A review of biomedical datasets relating to drug discovery: A knowledge graph perspective. https://arxiv.org/abs/2102.10062

[5] Drug repurposing for COVID-19 via knowledge graph completion. https://arxiv.org/abs/2010.09600

[6] RelationalAI Raises $122M to Redefine How Intelligent Data Apps Are Built. https://finance.yahoo.com/news/relationalai-raises-122m-redefine-intelligent-105500610.html

From: NeurIPS.cc/2022/Conference/Paper10172/Authors
title: Response to Reviewer M1H8
comment: Thank you for reviewing our paper including the appendix and the code. We would like to clarify a few things.

**Quality**: Thank you for inspecting our appendix and code. Our current code is still an early version. We plan to open source our codebase in the near future and will improve the code documentation as suggested.

**Clarity**: If there is a lack of clarity we would be happy to improve our presentation. Could you please provide further details? For example, what is the notation that was hard to follow?

**Originality**: We acknowledge the contribution of [28] in that they built a theoretical link between structural GNNs and node (positional) embeddings using invariant theory. Our work is fundamentally different from [28] in several aspects. 

First of all, in [28], authors focus on “simple” graphs, while KGs are multi-relational graphs. The models in [28] are not directly applicable to KGs. Specifically, models in [28] do not have any relational parameters and thus can not model different relationships between entities. The proposed ReFactor GNNs achieve SoTA on the task of inductive KGC [5], while it is not straightforward how to use FMs or CGNNs [28] for this task.

Second, our motivation is driven by the progress of KGC. The KGC community has recently experienced a resurgence of factorization-based models (FMs) , where training strategies, including learning objectives and optimizer choice, are shown to be critical [1-3]. Inspired by this, our work is devoted to understanding FMs from the perspective of the optimization process, which previous works in KBC did not explore. Since there is an increasing attention to GNN-based KBC models but empirical gains appear to be limited [3, 4], we try to bridge GNN-based and FM-based KBC models to better understand both paradigms. **As far as we know, we are the first to interpret the optimization over factorization-based KBC as message-passing**. We believe this interpretation opens up a new research direction where we can devise novel architectures at the intersection of message-passing neural nets and factorization modes.

[1] Knowledge Base Completion: Baseline strikes back (Again). https://arxiv.org/abs/2005.00804

[2] You CAN Teach an Old Dog New Tricks! On Training Knowledge Graph Embeddings. https://openreview.net/forum?id=BkxSmlBFvr

[3] A Re-evaluation of Knowledge Graph Completion Methods. https://aclanthology.org/2020.acl-main.489/

[4] Modeling Relational Data with Graph Convolutional Networks. https://arxiv.org/abs/1703.06103

[5] Inductive Relation Prediction by Subgraph Reasoning. https://arxiv.org/abs/1911.06962

[28] On the Equivalence between Positional Node Embeddings and Structural Graph Representations. https://arxiv.org/abs/1910.00452

From: NeurIPS.cc/2022/Conference/Paper10172/Authors
title: Response to Reviewer jDqr continue
comment: **Q2: It would be great to have some comment on the expressiveness of the ReFACTOR GNNs.**

To our knowledge, there are two separate notions of expressiveness for FMs and GNNs: 

- In the FM community, a key theoretical concept is full-expressiveness [1]. A FM is said to be fully expressive if, for any given graph $\mathcal{T}$ over entities $\mathcal{E}$ and relations $\mathcal{R}$, it can fully reconstruct the input adjacency tensor with a embedding size bounded by $\min(|\mathcal{E}||\mathcal{R}|, |\mathcal{T}| + 1)$. We can generalise this expressiveness analysis to FM-GNN models (ReFactor GNNs). In the $L \to \infty$ limit, ReFactor GNNs are as fully expressive as the underlying FMs. In fact, DistMult FM is not fully expressive (because of the symmetry); however a RefactorGNN based, e.g. on ComplEx can reach full expressiveness for $L \to \infty$. Please see also our reply to Q2 regarding the scoring function design choice. 

- For GNNs, established results concern the separation power of induced representations in terms of Weisfeiler-Leman (WL) isomorphism tests [5, 6]. However none of these results are directly applicable to our setting (e.g. they only consider one relationship). Nevertheless, if we consider our RefactorGNN model in a one-relationship setting, following the formalism of [6], we note that the Refactor Layer function cannot be written in Guarderd Tensor Language since at each layer it computes a global term $n[v]$. This implies that at worst RefactorGNN has a separation power upper bound comparable to the 1-WL test. We leave full analysis to future work, but will complement the conclusion section mentioning this promising direction. 

We are not aware of explicit connections between the two above notions of expressiveness. We think there is some possibility that we can bridge them, which itself will be a very interesting research direction, but would require a very substantial amount of additional work and presentation space and is thus beyond the scope of this paper.

We also kindly ask you to point us to any relevant work which we might have missed.

Alternatively, if we adopt the notion of expressiveness introduced in [5], we can also increase the expressiveness of ReFactor GNNs by adding more parameters to the message, the aggregation and the update operator. For example, introducing additional MLPs to transform the input node features or include non-linearity in the GNN update operator. This would be a natural way to increase the expressiveness of ReFactor GNNs.

Another method for increasing expressive power for link prediction task only is to extend ReFactor GNNs from node-wise to pairwise (Sec 2.2 in our paper) representations like GraIL[3] and NBFNet [2], which is more computationally intensive, but yields more powerful as node representations are not standalone but adapted to a specific query.

We will include the above comments in the revised paper.

[1] SimplE Embedding for Link Prediction in Knowledge Graphs. https://arxiv.org/abs/1802.04868

[2] Neural Bellman-Ford Networks: A General Graph Neural Network Framework for Link Prediction. https://arxiv.org/abs/2106.06935

[3] Inductive Relation Prediction by Subgraph Reasoning. https://arxiv.org/abs/1911.06962

[4] Are we really making much progress? Revisiting, benchmarking, and refining heterogeneous graph neural networks. https://keg.cs.tsinghua.edu.cn/jietang/publications/KDD21-Lv-et-al-HeterGNN.pdf

[5] How Powerful are Graph Neural Networks? https://openreview.net/pdf?id=ryGs6iA5Km

[6] Expressiveness and Approximation Properties of Graph Neural Networks. https://arxiv.org/abs/2204.04661

From: NeurIPS.cc/2022/Conference/Paper10172/Authors
title: Response to Reviewer jDqr
comment: Thank you for reviewing our paper. We will improve our work based on your suggestions and address your concerns as follows. 

**W1: The fact that the results presented are for $L=\infty$ after explicitly saying that they "periodically clear the node state cache every $L$ full batches" seems a bit strange. Is there any reason for not presenting resulting for $L$ other than $\infty$?**

We apologise for not making it clear about the usages of $L$ in our experiments. Since ReFactor GNNs interpolate FM and GNN, our experimental design focused on whether they can take the best of the both worlds. For the reported results, our **transductive** experiments (sec 5.1, Tab 1) used $L=\infty$ since we wanted to see if ReFactor GNNs ($L=\infty$) can reach the performance of the FMs; on the other hand, in our **inductive experiments**, we used ReFactor GNNs with $L \in \{1, 2, 3, 6, 9\}$ (sec 5.2, Fig 2 & 3, Tab 4 & 5) since we wanted to test their performances in inductive settings akin to standard GNNs. 

More on the role of $L$: In general, $L$ determines the number of effective message-passing layers in ReFactor GNNs. As $L$ increases, ReFactor GNNs cover more hops of neighbourhoods while at the same time reducing inductive applicability due to over-smoothing and computational requirements. Implementation-wise, $L$ is controlled by how frequently we clear the node state cache. In the extreme case of $L=\infty$, **where we never clear the node state cache during training**, the final cached node states will be used for inference directly (instead of running $L$ layers of message-passing on-the-fly to produce the node state). Note that if we have new nodes (inductive setup) we do not have cached node states, therefore this latter inference regime (with $L=\infty$) is inherently transductive.

We will revise Sec 4 & 5 accordingly to incorporate the above comments

**W2: There should be some comparisons on the link prediction task especially with RGCN.**

We ran additional experiments as suggested with RGCN. Since RGCN is with a much higher parameter complexity (linear in the number of relations), we had to perform the experiments with a smaller embedding size to fit it into our GPU memory. We will add the numbers to Table 1, Table 4 and 5 in our revised paper. For now, we list some results of RGCN here. For transductive KGC (sec 5.1), RGCN with roughly the same amount of parameters reached 0.33 MRR while DistMult or ReFactor GNN ($L=\infty$) reached 0.43 on CoDEx-S. For inductive KGC (sec 5.2), on FB15K237_v1_ind, a 3-layer RGCN obtains 0.256 Hits@10 which is worse than GAT (0.333) and ReFactor GNN (0.673).  Our findings align with [4], where simple homogeneous GNNs, e.g, GAT, with proper hyper-parameter settings, are shown to outperform Heterogeneous GNNs, e.g. RGCN, across various scenarios.

**Q1: What is the running time of the proposed ReFactor GNNs?**

By running time, we assume you are referring to inference time cost. We included the numbers in Appendix E. For our experiments, we used our two Tesla V100 GPUs with Pytorch-Lightning DDP. For inductive experiments, take FB15K237_v1_Ind as an example. The inference over the graph (1,093 nodes, 360 relations, \~4k triplets, including reciprocal triplets)  to answer all the validation/test queries (\~400 triplets, including recirporal triplets) took about 1s of wall-clock time. Wall-clock times with other hardware configurations and implementation will vary.

Going beyond wall-clock time, we can extend the complexity analysis in NBFNet [2] to consider a full pass over all the triplets in a given graph. 

Scalability w.r.t. to the number of layers – Let $L$ denote the number of message-passing layers. Since ReFactor GNNs tie weights across layers, the parameter complexity of ReFactor GNNs is O(1), while it is O(L) for standard GNNs (such as GATs, GCNs, and RGCNs). Since ReFactor GNNs adopt layer-wise training (enabled via the external memory for node state caching), the training memory footprint is also O(1), while it is O(L) for standard GNNs. 

Scalability w.r.t the embedding size. Let $d$ denote the embedding size. ReFactor GNNs scale linearly with $d$, while for most GNNs in the literature, the parameter and time complexities scale quadratically with $d$.

Scalability w.r.t the number of edges $\mathcal{T}$ in the graph – Denote the entity set as $\mathcal{E}$, the relation set as $\mathcal{R}$, and the triplets as $\mathcal{T}$. NBFNet[2] requires $O(LT^2d + LTVd^2)$ inference runtime complexity since the message-passing is done for every source node and query relation -- quadratic w.r.t the number of triplets T while ReFactor GNNs are of linear complexity w.r.t T. 

See below for the detailed complexity comparison table.

From: NeurIPS.cc/2022/Conference/Paper10172/Authors
title: Response to Reviewer rRLn continue
comment: **Q2: In which way do the design choices (DistMult, optimizer etc.) affect the key result?**

Thank you for the insightful question. As you rightfully mention, the two main design choices in Th. 1 are: 1) the score function $\Gamma$, and  2) the optimization dynamics over the node embeddings. In the paper, we chose DistMutl and GD because of their mathematical simplicity, leading to easier-to-read formulas. We can obtain extensions of Th 1 to general smooth scoring functions $\Gamma:.\mathcal{E} \times R \times \mathcal{E} \to \mathcal{R}$ by replacing occurrences of the gradient of DistMult by a generic $\nabla \Gamma$ (the gradient of DistMult w.r.t. $\phi[v]$  at $(v, r, w)$  is simply $g(r) \odot \phi[w]$). For instance, in Eq. (8) we would have $q_m = \nabla_{\phi[v]} \Gamma(v, r, w)$ if $(r, w) \in \mathcal{N}^+[v]$. Eq. (9) would remain the same, while Eq. (10) can likewise be expressed as a function of $\nabla \Gamma$ rather than the gradient of DistMult.

Theorem 1 is also extensible to different standard choices of first-order optimizers such as GD with momentum, RMSprop and Adam. In appendix A.1 we did this for AdaGrad. In general, we can interpret any auxiliary variable introduced by the optimizer (e.g. the velocity) as an additional part of the entities and relations representations on which message passing happens. However the specific equations would depend on the optimizer’s dynamics and would be hard to formally generalise. 

We will add a formulation of theorem 1 for general score functions to the appendix as well as an extended discussion on the role of the optimization dynamics. 

Empirically, in our experiments, we use AdaGrad since it is a commonly used optimiser for Knowledge Graph Completion (KGC) and results in better empirical performance as noted in [2,3,4,5] compared to other optimisers like Adam. In our ablation experiments for transductive link prediction, replacing AdaGrad with SGD led to a significant performance drop (0.325/0.303 on FB15K237). Beyond our work, the study of why AdaGrad works better than other optimisers for KGC is itself a very interesting question for the community. 

We will also amend our paper to include discussion on this empirical part.


**Q3:  What type of guarantees could we possibly give to ReFactorGNN?**

As mentioned in our response to Q1, one possibility is to use theory/findings for FMs to analyse ReFactor GNNs or even standard GNNs. Since ReFactor GNNs are effectively the same as the underlying FMs as $L \to \infty$, they can be as “fully-expressive” as the corresponding FMs. We apologise for the brief response, but this is a rather broad question. We invite the reviewer to provide more details. 

[1] SimplE Embedding for Link Prediction in Knowledge Graphs. https://arxiv.org/abs/1802.04868

[2] Canonical Tensor Decomposition for Knowledge Base Completion. https://arxiv.org/abs/1806.07297

[3] Knowledge Graph Completion via Complex Tensor Factorization. https://arxiv.org/abs/1702.06879

[4] You CAN Teach an Old Dog New Tricks! On Training Knowledge Graph Embeddings. https://openreview.net/forum?id=BkxSmlBFvr

[5] Efficient energy-based embedding models for link prediction in knowledge graphs. https://link.springer.com/article/10.1007/s10844-016-0414-7

[6] How Powerful are Graph Neural Networks? https://openreview.net/pdf?id=ryGs6iA5Km

[7] Expressiveness and Approximation Properties of Graph Neural Networks. https://arxiv.org/abs/2204.04661

[8] Neural Bellman-Ford Networks: A General Graph Neural Network Framework for Link Prediction. https://arxiv.org/abs/2106.06935

From: NeurIPS.cc/2022/Conference/Paper10172/Authors
title: Response To Reviewer rRLn
comment: Thank you for your valuable feedback and comments! We will incorporate your suggestions in the updated version of the paper.

Weaknesses: We acknowledge that investigating the expressiveness and tractability of ReFactor GNNs is an important research topic. For ReFactor GNNs, our current choice of DistMult and SGD/AdaGrad is mainly based on their mathematical simplicity and good empirical performances for downstream link prediction tasks. In future work, we will consider more variants in the spectrum of FM-GNN models. Please see our responses below for further details.

**Q1: How could we classify/analyse the spectrum of FM-GNN models?**

We believe there should be ample room for analysing the spectrum of FM-GNN models by combining research on FMs and GNNs. As you suggest, we can start by looking at expressiveness and tractability.

To our knowledge, there are two separate notions of expressiveness for FMs and GNNs: 

- In the FM community, a key theoretical concept is full-expressiveness [1]. A FM is said to be fully expressive if, for any given graph $\mathcal{T}$ over entities $\mathcal{E}$ and relations $\mathcal{R}$, it can fully reconstruct the input adjacency tensor with a embedding size bounded by $\min(|\mathcal{E}||\mathcal{R}|, |\mathcal{T}| + 1)$. We can generalise this expressiveness analysis to FM-GNN models (ReFactor GNNs). In the $L \to \infty$ limit, ReFactor GNNs are as fully expressive as the underlying FMs. In fact, DistMult FM is not fully expressive (because of the symmetry); however a RefactorGNN based, e.g. on ComplEx can reach full expressiveness for $L \to \infty$. Please see also our reply to Q2 regarding the scoring function design choice. 

- For GNNs, established results concern the separation power of induced representations in terms of Weisfeiler-Leman (WL) isomorphism tests [6, 7]. However none of these results are directly applicable to our setting (e.g. they only consider one relationship). Nevertheless, if we consider our RefactorGNN model in a one-relationship setting, following the formalism of [7], we note that the Refactor Layer function cannot be written in Guarderd Tensor Language since at each layer it computes a global term $n[v]$. This implies that at worst RefactorGNN has a separation power upper bound comparable to the 1-WL test. We leave full analysis to future work, but will complement the conclusion section mentioning this promising direction. 

We are not aware of explicit connections between the two above notions of expressiveness. We think there is some possibility that we can bridge them, which itself will be a very interesting research direction, but would require a very substantial amount of additional work and presentation space and is thus beyond the scope of this paper. We also kindly ask you to point us to any relevant work which we might have missed.

As for tractability, we assume you mean scalability. We apologise if you meant something else. Here, we can analyse the scalability of ReFactor GNNs as the the number of layers $L$ and the number of triplets/edges in the graph $|\mathcal{T}|$ increase. We will include the following analysis in our revised paper.

Scalability w.r.t. to the number of layers – Let $L$ denote the number of message-passing layers. Since ReFactor GNNs tie weights across layers, the parameter complexity of ReFactor GNNs is O(1), while it is O(L) for standard GNNs (such as GATs, GCNs, and RGCNs). Since ReFactor GNNs adopt layer-wise training (enabled via the external memory for node state caching), the training memory footprint is also O(1), while it is O(L) for standard GNNs. 

Scalability w.r.t the embedding size. Let $d$ denote the embedding size. ReFactor GNNs scale linearly with $d$, while for most GNNs in the literature, the parameter and time complexities scale quadratically with $d$.

Scalability w.r.t the number of edges $\mathcal{T}$ in the graph – Denote the entity set as $\mathcal{E}$, the relation set as $\mathcal{R}$, and the triplets as $\mathcal{T}$. NBFNet [8] requires $O(LT^2d + LTVd^2)$ inference runtime complexity since the message-passing is done for every source node and query relation -- quadratic w.r.t the number of triplets $\mathcal{T}$ while ReFactor GNNs are of linear complexity w.r.t $\mathcal{T}$. 

See below for the complexity comparison table

From: NeurIPS.cc/2022/Conference/Paper10172/Reviewer_rRLn
rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
confidence: 3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
summary: [Update after Rebuttal: Increased confidence, Accept]

[REMARK: This is an emergency review, that is, the reviewer did neither select autonomously into this paper out of interest/expertise nor was the reviewer granted with sufficient time (with respect to standard time frames) to appropriately provide a review. Please take the review with caution, the confidence level will be adjusted accordingly.]

The presented paper proposes a bridge between the otherwise disjoint model classes of factorization-based models (FM) and the currently popular graph neural networks (GNN). The motivation for this lies in the fact that the success stories of said models remains disjoint as well i.e., GNNs are successful widely but in tasks such as multi-relational link prediction they fall short to FMs. The authors bridge aforementioned gap by providing a reformulation (Theorem 3.1) that shows how one can identify Message Passing a la GNN within FM. To corroborate on their theoretical result, the authors provide a first model living within the identified (or expected) spectrum of FM-GNN models and give an empirical account of said model (referred to as ReFactor GNN) showing that the addendum to plain GNN indeed improves the shortcomings.
strengths_and_weaknesses: **Strengths**

Studying the shortcomings of GNN in advent of their rise as dominant model class for structured-input predictions is an important step towards improving the model class to cope with more classical tasks such as knowledge graph completion. Bridging the gap to other existing model classes such as FMs, helps to broaden the understanding of what qualitative differences between our modelling choices of interest exist. It also enables (a foundation to) compilation back-and-forth and therefore allows for flexible use of existing tools. The presentation of the ideas is detailed and complimented by several modalities including the rigorous mathematization and a key figure. The empirical results suggest a sensible view on FM-GNNs and the idea of a spectrum of such models (which might have been identified by this work) leaves room for further investigations.

**Weaknesses**

IMHO there is only so much a contribution can propose and certainly the key results presented should not be deluted, therefore, the following weaknesses are rather extensions to be considered upon the existing work. Considering GNN literature, several theoretical findings have been made regarding their expressivity, tractability, etc. and in a similar fashion analyzing the presented initial FM-GNN, ReFactor GNN, would benefit our understanding of where to place the model within the zoo of existing methods. Furthermore, the focus has been on DistMult but also on optimizers such as SGD or AdaGrad (as also explicitly mentioned by the authors), yet, there exists the impression of more general schemes (the FM-GNN spectrum) and thereby it would be sensible to already provide a disclaimer on how the design choices already (strongly or weakly) influence the subsequent results.
questions: The questions are derived mostly from aspects mentioned in the "Weaknesses" section. I'd invite the authors to answer these questions to overcome the observed weaknesses for improving the paper to ultimately increase the chances of both contribution and visibility to the community. The list of questions is unordered and they range in quality and speed of expected answer (some are minor, quickly answerable whereas others are more fundamental, crucial, maybe difficult to answer):
* How could we classify/analyze the spectrum of FM-GNN models?
* In which way do the design choices (DistMult, optimizer etc.) affect the key result?
* What type of guarantees could we possibly give to ReFactorGNN?
limitations: No contradictions or any sort of relevant mistake have been detected in the paper. The clarity of the paper is a clear advantage, even if (likely unavoidable) dense at times. Existing bodies of work are being referenced accordingly throughout the paper.

Sufficient details for reproduction are being provided. Also with actual code (however, the code was not reviewed). All key assumptions for the method are being pointed out explicitly. Societal impact is not being discussed explicitly, however, this was not necessary to begin with IMHO.
ethics_flag: No
soundness: 3 good
presentation: 3 good
contribution: 3 good

From: NeurIPS.cc/2022/Conference/Paper10172/Reviewer_jDqr
rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
confidence: 3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
summary: This work proposes ReFactor GNNs that makes use of the message passing property of GNNs in order to cast he hugely successful factorization based embedding techniques, here DistMult, as a GNN. The idea is to formulate gradient descent in factorization based models as message passing in GNNs. The experimental section shows that the proposed model works well on different knowledge base completion tasks.
strengths_and_weaknesses: Strengths:

1. The idea is simple and quite intuitive and yet very clever at the same time. It's great to see this connection between DistMult's and GNNs in a more formal fashion.

2. The paper is written nicely and is easily understandable.

Weaknesses:

1. The fact that the results presented are for $L=\infty$ after explicitly saying that they "periodically clear the node state cache every L full batches" seems a bit strange. Is there any reason for not presenting resulting for L other than $\infty$?

2. There should be some comparisons on the link prediction task especially with RGCN. 

I have done an emergency review for this paper and thus have lowered my confidence. Overall, I like the idea behind the paper and would like to hear more from the authors in the rebuttal. My major concern remains the experimental part of the paper.

questions: In addition to the above 2 points raised in weaknesses, I have a few more questions:

1. What is the running time of the proposed ReFactor GNNs?

2. It would be great to have some comment on the expressiveness of the ReFACTOR GNNs.
limitations: I do not foresee any potential negative societal impact of this work.
ethics_flag: No
soundness: 2 fair
presentation: 3 good
contribution: 3 good

From: NeurIPS.cc/2022/Conference/Paper10172/Reviewer_M1H8
rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
confidence: 1: Your assessment is an educated guess. The submission is not in your area or the submission was difficult to understand. Math/other details were not carefully checked.
summary: **DISCLAIMER: This is not my area of expertise, and this paper is very specific and quite technical. Nevertheless, the program chairs asked to review the submission based on my experience and broad knowledge anyway.  I do try.**

This paper proposes a new approach to the problem of knowledge graph completion. This problem resisted recent advances in geometric deep learning and the best approaches so far are still based on factorization-based models (FB).
Here the authors note that FB models can be seen as GNNs "by reformulating gradient descent on node embeddings using message-passing primitives" and use this idea to build a new hybrid model called Refactor GNN, that achieves the performance of FB models using 1/10 parameters, sometimes even fixing a new state of the art.

strengths_and_weaknesses: ### Originality
I am not able to assess whether these ideas were already existing under a different name. In particular I am not able to judge the novelty with respect to [28].

### Quality
As far as I can tell, this work is of good quality. I appreciated the uncertainty estimated using different random seeds reported in the appendix. I peeked at the code and it seems well organized, even if mostly bare of comments.

### Clarity
This paper is very specific and quite technical. The background section was easy to follow, which is a good thing, then for a non-expert it becomes quite harsh.

### Significance

Suppose I have to bet on how many citations this paper will receive in the next five years.

Following the stats of accepted papers at Neurips 2014 tracked in 2019 (as can be found at https://github.com/rubai5/MLcites) a bookmaker could give the following betting odds:

- A: Between 0 and 5 cits, proposed at 10/1.
- B: Between 5 and 25 cits, proposed at 3/1.
- C: Between 25 and 75 cits, proposed at 3/1.
- D: More than 75 cits, proposed at 6/1.

I would bet my money on A, maybe adjusting to B given the higher rate of citations in the NLP field. Actually this bet is based on my broad knowledge, and should be understood as a bet on the doom of knowledge graphs (KGs). The problem discussed here is very specific to KGs, and to me it seems that many use-cases of KGs were recently robbed by large language models, a short time ago enriched also with explicit memory and retrieval methods [RETRO].


[RETRO] Borgeaud, Sebastian, et al. "Improving language models by retrieving from trillions of tokens." arXiv preprint arXiv:2112.04426 (2021).  

questions: I do not have relevant questions to ask. My score is borderline because I am not able to judge too many aspects of this paper. 
limitations: There seems to be adequate discussion of the limitations, but I can't make an assessment.

ethics_flag: No
soundness: 3 good
presentation: 2 fair
contribution: 2 fair