Original: R E F ACTOR GNN S : Revisiting Factorisation-based Models from a Message-Passing Perspective
Revised: R E F ACTOR GNN S : Revisiting Factorisation-based Models from a Message-Passing Perspective

Original: Abstract
Revised: Abstract

Original: Factorisation-based Models (FMs), such as DistMult, have enjoyed enduring success for Knowledge Graph Completion (KGC) tasks, often outperforming Graph
Revised: Factorisation-based Models (FMs), such as DistMult, have enjoyed enduring success for Knowledge Graph Completion (KGC) tasks, often outperforming Graph

Original: Neural Networks (GNNs).
Revised: Neural Networks (GNNs).

Original: However, unlike GNNs, FMs struggle to incorporate node features and generalise to unseen nodes in inductive settings.
Revised: However, unlike GNNs, FMs struggle to incorporate node features and generalise to unseen nodes in inductive settings.

Original: Our work bridges the gap between FMs and GNNs by proposing R E F ACTOR GNN S .
Revised: Our work bridges the gap between FMs and GNNs by proposing R E F ACTOR GNN S .

Original: This new architecture draws upon both modelling paradigms, which previously were largely thought of as disjoint.
Revised: This new architecture draws upon both modelling paradigms, which previously were largely thought of as disjoint.

Original: Concretely, using a message-passing formalism, we show how FMs can be cast as GNNs by reformulating the gradient descent procedure as message-passing operations, which forms the basis of our R E F ACTOR
Revised: Concretely, using a message-passing formalism, we show how FMs can be cast as GNNs by reformulating the gradient descent procedure as message-passing operations, which forms the basis of our R E F ACTOR

Original: GNN S .
Revised: GNN S .

Original: Across a multitude of well-established KGC benchmarks, our R E F ACTOR
Revised: Across a multitude of well-established KGC benchmarks, our R E F ACTOR

Original: GNN S achieve comparable transductive performance to FMs, and state-of-the-art inductive performance while using an order of magnitude fewer parameters.
Revised: GNN S achieve comparable transductive performance to FMs, and state-of-the-art inductive performance while using an order of magnitude fewer parameters.

Original:  Introduction
Revised:  Introduction

Original: In recent years, machine learning on graphs has attracted signiﬁcant attention due to the abundance of graph-structured data and developments in graph learning algorithms.
Revised: In recent years, machine learning on graphs has attracted signiﬁcant attention due to the abundance of graph-structured data and developments in graph learning algorithms.

Original: Graph Neural
Revised: Graph Neural

Original: Networks (GNNs) have shown state-of-the-art performance for many graph-related problems, such as node classiﬁcation [16] and graph classiﬁcation [7].
Revised: Networks (GNNs) have shown state-of-the-art performance for many graph-related problems, such as node classiﬁcation [16] and graph classiﬁcation [7].

Original: Their main advantage is that they can easily be applied in an inductive setting: generalising to new nodes and graphs without re-training.
Revised: Their main advantage is that they can easily be applied in an inductive setting: generalising to new nodes and graphs without re-training.

Original: However, despite many attempts at applying GNNs for multi-relational link prediction  such as Knowledge
Revised: However, despite many attempts at applying GNNs for multi-relational link prediction tasks such as Knowledge

Original: Graph Completion [26], there are still few positive results compared to factorisation-based models (FMs) [45, 39].
Revised: Graph Completion [26], there are still few positive results compared to factorisation-based models (FMs) [45, 39].

Original: As it stands, GNNs either – after resolving reproducibility concerns – deliver signiﬁcantly lower performance [23, 36] or yield negligible performance gains at the cost of highly sophisticated architecture designs [43].
Revised: As it stands, GNNs either – after resolving reproducibility concerns – deliver signiﬁcantly lower performance [23, 36] or yield negligible performance gains at the cost of highly sophisticated architecture designs [43].

Original: A notable exception is NBFNet [49], but even here the advance comes at the price ofa high computational inference cost compared to FMs.
Revised: A notable exception is NBFNet [49], but even here improvements come at the price of high computational inference costs compared to FMs.

Original: Furthermore, it is unclear how NBFNet could incorporate node features, which – as we will see in this work – leads to remarkably lower performance in an inductive setting.
Revised: Furthermore, it is unclear how NBFNet could incorporate node features, which – as we will see in this work – leads to remarkably lower performance in an inductive setting.

Original: On the ﬂip side, FMs, despite being a simpler architecture, have been found to be very accurate for knowledge graph completion when coupled with appropriate training strategies [30] and training objectives [17, 2].
Revised: On the other hand FMs, despite being a simpler architecture, have been found to be very accurate for knowledge graph completion when coupled with appropriate training strategies [30] and training objectives [17, 2].

Original: However, they also come with shortcomings in that they, unlike GNNs, can not be applied in an inductive setting.
Revised: However, they also come with shortcomings in that they, unlike GNNs, can not be applied in an inductive setting.

Original: Given the respective strengths and weaknesses of FMs and GNNs, can we bridge these two seemingly different model categories?
Revised: Given the respective strengths and weaknesses of FMs and GNNs, can we bridge these two seemingly different model categories?

Original: While exploring this question, we make the following contributions:
Revised: While exploring this question, we make the following contributions:

Original: • By reformulating gradient descent on node embeddings using message-passing primitives,
Revised: • By reformulating gradient descent on node embeddings using message-passing primitives,

Original: we
Revised: we

Original: show a practical connection between FMs and GNNs, in that: FMs can be treated as a special instance of GNNs, but with inﬁnite neighbourhood, layer-wise training and a global normaliser.•
Revised: show a practical connection between FMs and GNNs, in that: FMs can be treated as a special instance of GNNs, but with inﬁnite neighbourhood, layer-wise training and a global normaliser.•

Original: Based on this connection, we propose a new family of architectures, referred to as R E F ACTOR GNN S , which interpolates between FMs and GNNs.
Revised: Based on this connection, we propose a new family of architectures, referred to as R E F ACTOR GNN S , which interpolates between FMs and GNNs.

Original: In essence, R E F ACTOR GNN S inductivise FMs by using a ﬁnite number of message-passing layers, and incorporating node features.
Revised: In essence, R E F ACTOR GNN S inductivise FMs by using a ﬁnite number of message-passing layers, and incorporating node features.

Original: • Through an empirical investigation across 15 well-established inductive and transductive benchmarks, we ﬁnd R E F ACTOR GNN S achieve state-of-the-art inductive performance and comparable transductive performance to FMs – despite using an order of magnitude fewer parameters.
Revised: • Through an empirical investigation across 15 well-established inductive and transductive benchmarks, we ﬁnd R E F ACTOR GNN S achieve state-of-the-art inductive performance and comparable transductive performance to FMs – despite using an order of magnitude fewer parameters.

Original:  Background
Revised:  Background

Original: Knowledge Graph Completion (KGC) [25] is a canonical task of multi-relational link prediction.
Revised: Knowledge Graph Completion (KGC) [25] is a canonical task of multi-relational link prediction.

Original: The goal is to predict missing edges given the existing edges in the knowledge graph.
Revised: The goal is to predict missing edges given the existing edges in the knowledge graph.

Original: Formally, a knowledge graph contains a set of entities (nodes) E
Revised: Formally, a knowledge graph contains a set of entities (nodes) E

Original: = { 1 , . .
Revised: = { 1 , . .

Original: . , |E|} , a set of relations (edge types)
Revised: . , |E|} , a set of relations (edge types)

Original: R = { 1 , . .
Revised: R = { 1 , . .

Original: . , |R|} , and a set of typed edges between the entities T
Revised: . , |R|} , and a set of typed edges between the entities T

Original: = { ( v i , r i , w i ) } |T
Revised: = { ( v i , r i , w i ) } |T

Original: | i =1 , where each triplet ( v i , r i , w i ) indicates a relationship of type r i ∈ R between the subject v i ∈ E and the object w i ∈ E .
Revised: | i =1 , where each triplet ( v i , r i , w i ) indicates a relationship of type r i ∈ R between the subject v i ∈ E and the object w i ∈ E .

Original: Given a node v , we denote its outgoing 1-hop neighbourhood as the set of relation-object pairs N 1+ [v] = { ( r, o ) | (v , r, o ) ∈ T } , its incoming 1-hop neighbourhood as the set of subject-relation pairs N 1 − [v] = { ( r, s ) | ( s, r, v) ∈ T } , and N 1 [ v ] = N 1+ [ v ] ∪ N 1 − [ v ] the union of the two.
Revised: Given a node v , we denote its outgoing 1-hop neighbourhood as the set of relation-object pairs N 1+ [v] = { ( r, o ) | (v , r, o ) ∈ T } , its incoming 1-hop neighbourhood as the set of subject-relation pairs N 1 − [v] = { ( r, s ) | ( s, r, v) ∈ T } , and N 1 [ v ] = N 1+ [ v ] ∪ N 1 − [ v ] the union of the two.

Original: We denote the neighbourhood of v under a speciﬁc relation r as N 1 ± [ r, v ] .
Revised: We denote the neighbourhood of v under a speciﬁc relation r as N 1 ± [ r, v ] .

Original: Entities may come with features X ∈ R |E|× K for describing them, such as textual encodings of their names and/or descriptions.
Revised: Entities may come with features X ∈ R |E|× K for describing them, such as textual encodings of their names and/or descriptions.

Original: Given a (training) knowledge graph, KGC is evaluated by answering ( v, r, ?)
Revised: Given a (training) knowledge graph, KGC is evaluated by answering ( v, r, ?)

Original: -style queries i.e. predicting the object given the subject and relation in the triplet.
Revised: -style queries i.e. predicting the object given the subject and relation in the triplet.

Original: Multi-relational link prediction models can be trained via maximum likelihood, by ﬁtting a parameterized conditional categorical distribution P θ ( w | v, r ) over the candidate objects of a relation, given the subject v and the relation type r :
Revised: Multi-relational link prediction models can be trained via maximum likelihood, by ﬁtting a parameterized conditional categorical distribution P θ ( w | v, r ) over the candidate objects of a relation, given the subject v and the relation type r :

Original: [Equation]
Revised: [Equation]

Original: Here Γ θ : E × R × E → R is a scoring function , which, given a triplet ( v, r, w ) , returns the likelihood that the corresponding edge appears in the knowledge graph.
Revised: Here Γ θ : E × R × E → R is a scoring function , which, given a triplet ( v, r, w ) , returns the likelihood that the corresponding edge appears in the knowledge graph.

Original: We illustrate our derivations using DistMult [45] as the score function Γ and defer extensions to general score functions, e.g.
Revised: We illustrate our derivations using DistMult [45] as the score function Γ and defer extensions to general score functions, e.g.

Original: ComplEx [39], to the appendix.
Revised: ComplEx [39], to the appendix.

Original: For DistMult, the score function Γ θ is deﬁned as the tri-linear dot product of the vector representations corresponding to the subject, relation, and object of the triplet:
Revised: For DistMult, the score function Γ θ is deﬁned as the tri-linear dot product of the vector representations corresponding to the subject, relation, and object of the triplet:

Original: [Equation]
Revised: [Equation]

Original: where f φ : E → R K and g ψ : R → R K are learnable maps parameterised by φ and ψ that encode entities and relation types into K -dimensional vector representations, and θ = ( φ, ψ ) .
Revised: where f φ : E → R K and g ψ : R → R K are learnable maps parameterised by φ and ψ that encode entities and relation types into K -dimensional vector representations, and θ = ( φ, ψ ) .

Original: We will refer to f and g as the entity and relation encoders , respectively.
Revised: We will refer to f and g as the entity and relation encoders , respectively.

Original: If we deﬁne the data distribution as
Revised: If we deﬁne the data distribution as

Original: P D [Equation] x ) = 1 |T | (cid:80) ( v,r,w ) ∈T δ ( v,r,w ) ( x ) , where δ ( v,r,w ) ( x ) is a Dirac delta function at ( v, r, w ) , then the objective is to learn the model parameters θ by minimising the expected negative log-likelihood
Revised: P D ( x ) = 1 |T | (cid:80) ( v,r,w ) ∈T δ ( v,r,w ) ( x ) , where δ ( v,r,w ) ( x ) is a Dirac delta function at ( v, r, w ) , then the objective is to learn the model parameters θ by minimising the expected negative log-likelihood

Original: L ( θ ) of the ground-truth entities for the queries ( v, r, ?) obtained from T :
Revised: L ( θ ) of the ground-truth entities for the queries ( v, r, ?) obtained from T :

Original: [Equation]
Revised: [Equation]

Revised: where L

Original:  During inference, we usethe distribution P θ for determining the plausibility of links not present in the training graph.
Revised: [Equation] θ [Equation] During inference, we use P θ for determining the plausibility of links not present in the training graph.

Original: 2.1 Factorisation-based Models for KGC
Revised: 2.1 Factorisation-based Models for KGC

Original: In factorisation-based models, which we assume to be DistMult, f φ and g ψ are simply parameterised as look-up tables, associating each entity and relation with a continuous distributed representation: f φ ( v ) =
Revised: In factorisation-based models, which we assume to be DistMult, f φ and g ψ are simply parameterised as look-up tables, associating each entity and relation with a continuous distributed representation: f φ ( v ) =

Original: φ [ v ] , φ ∈ R |E|× K and g ψ ( r ) =
Revised: φ [ v ] , φ ∈ R |E|× K and g ψ ( r ) =

Original: ψ [ r ] , ψ ∈ R |R|× K .
Revised: ψ [ r ] , ψ ∈ R |R|× K .

Original: 2.2 GNN-based Models for KGC
Revised: 2.2 GNN-based Models for KGC

Original: GNNs were originally proposed for node or graph classiﬁcation tasks [8, 32].
Revised: GNNs were originally proposed for node or graph classiﬁcation tasks [8, 32].

Original: To adapt them to
Revised: To adapt them to

Original: KGC, previous work has explored two different paradigms: node-wise entity representations [33] and pair-wise entity representations [37, 49].
Revised: KGC, previous work has explored two different paradigms: node-wise entity representations [33] and pair-wise entity representations [37, 49].

Original: Though the latter paradigm has shown promising results, it requires computing representations for all pairs of nodes, which can be computationally expensive for large-scale graphs with millions of entities.
Revised: Though the latter paradigm has shown promising results, it requires computing representations for all pairs of nodes, which can be computationally expensive for large-scale graphs with millions of entities.

Original: Additionally, node-wise representations allow for using a single evaluation of f φ ( v ) for multiple queries involving v .
Revised: Additionally, node-wise representations allow for using a single evaluation of f φ ( v ) for multiple queries involving v .

Original: Models based on the ﬁrst paradigm differ from pure FMs only in the entity encoder and lend themselves well for a fairer comparison with pure FMs.
Revised: Models based on the ﬁrst paradigm differ from pure FMs only in the entity encoder and lend themselves well for a fairer comparison with pure FMs.

Original: We will therefore focus on this class and leave the investigation of pair-wise representations to future work.
Revised: We will therefore focus on this class and leave the investigation of pair-wise representations to future work.

Original: Let q φ : G × X
Revised: Let q φ : G × X

Original: →
Revised: →

Original: (cid:83) S ∈ N + R S × K be a GNN encoder, where G =
Revised: (cid:83) S ∈ N + R S × K be a GNN encoder, where G =

Original: { G | G ⊆
Revised: { G | G ⊆

Original: E × R × E} is the set of all possible multi-relational graphs deﬁned over E and R , and X is the input feature space, respectively.
Revised: E × R × E} is the set of all possible multi-relational graphs deﬁned over E and R , and X is the input feature space, respectively.

Original: Then we can set f φ ( v ) =
Revised: Then we can set f φ ( v ) =

Original: q φ ( T , X )[ v ] .
Revised: q φ ( T , X )[ v ] .

Original: Following the standard message-passing framework [7, 9] used by the GNNs, we view q φ = q L ◦ ... ◦
Revised: Following the standard message-passing framework [7, 9] used by the GNNs, we view q φ = q L ◦ ... ◦

Original: q 1 as the recursive composition of L ∈ N + layers that compute intermediate representations h l for l ∈ { 1 , . . .
Revised: q 1 as the recursive composition of L ∈ N + layers that compute intermediate representations h l for l ∈ { 1 , . . .

Original: , L } (and h 0 =
Revised: , L } (and h 0 =

Original: X ) for all entities in the KG.
Revised: X ) for all entities in the KG.

Original: Each layer is made up of the following three functions:
Revised: Each layer is made up of the following three functions:

Original: A message function q l M : R K × R × R K
Revised: A message function q l M : R K × R × R K

Original: → R K that computes the message along each edge.
Revised: → R K that computes the message along each edge.

Original: Given an edge ( v, r, w ) ∈ T , q l M not only makes use of the node states h l − 1 [ v ] and h l − 1 [ w ] (as in standard GNNs) but also uses the relation r ; denote the message as
Revised: Given an edge ( v, r, w ) ∈ T , q l M not only makes use of the node states h l − 1 [ v ] and h l − 1 [ w ] (as in standard GNNs) but also uses the relation r ; denote the message as

Original: [Equation]
Revised: [Equation]

Original: An aggregation function q l A : (cid:83) S ∈ N R S
Revised: An aggregation function q l A : (cid:83) S ∈ N R S

Original: × K → R K that aggregates all messages from the 1-hop neighbourhood of a node; denote the aggregated message as
Revised: × K → R K that aggregates all messages from the 1-hop neighbourhood of a node; denote the aggregated message as

Original: [Equation]
Revised: [Equation]

Original: An update function q l U :
Revised: An update function q l U :

Original: R K
Revised: R K

Original: × R K → R K that produces the new node states h l by combining previous node states h l − 1 and the aggregated messages z l :
Revised: × R K → R K that produces the new node states h l by combining previous node states h l − 1 and the aggregated messages z l :

Original: [Equation]
Revised: [Equation]

Original: Different parameterisations of q l M , q l A , and q l U lead to different GNNs.
Revised: Different parameterisations of q l M , q l A , and q l U lead to different GNNs.

Original: For example, R-GCNs [33] deﬁne the q l M function using per-relation the linear transformations m l [ v, r, w ] = 1 N 1 [ r,v ] W lr h l − 1 [ w ] ; q l A is implemented by a summation and q l U is a non-linear transformation h l [ v ]
Revised: For example, R-GCNs [33] deﬁne the q l M function using per-relation the linear transformations m l [ v, r, w ] = 1 N 1 [ r,v ] W lr h l − 1 [ w ] ; q l A is implemented by a summation and q l U is a non-linear transformation h l [ v ]

Original: = σ ( z l [ v ] +
Revised: = σ ( z l [ v ] +

Original: W l 0 h l − 1 [ v ]) , where σ is the sigmoid function.
Revised: W l 0 h l − 1 [ v ]) , where σ is the sigmoid function.

Original: For each layer, the learnable parameters are { W lr } r ∈R and W l 0 , all of which are matrices in R K × K .
Revised: For each layer, the learnable parameters are { W lr } r ∈R and W l 0 , all of which are matrices in R K × K .

Original:  Implicit Message-Passing in FMs
Revised:  Implicit Message-Passing in FMs

Original: The sharp difference in analytical forms might give rise to the misconception that GNNs incorporate message-passing over the neighbourhood of each node (up to L -hops), while FMs do not.
Revised: The sharp difference in analytical forms might give rise to the misconception that GNNs incorporate message-passing over the neighbourhood of each node (up to L -hops), while FMs do not.

Original: In this work, we show that by explicitly considering the training dynamics of FMs, we can uncover and analyse the hidden message-passing mechanism within FMs.
Revised: In this work, we show that by explicitly considering the training dynamics of FMs, we can uncover and analyse the hidden message-passing mechanism within FMs.

Original: In turn, this will lead us to the formulation of a novel class of GNNs well suited for multi-relational link prediction tasks (Section 4).
Revised: In turn, this will lead us to the formulation of a novel class of GNNs well suited for multi-relational link prediction tasks (Section 4).

Original: Speciﬁcally, we propose to interpret the FMs’ optimisation process of their objective (3) as the entity encoder.
Revised: Speciﬁcally, we propose to interpret the FMs’ optimisation process of their objective (3) as the entity encoder.

Original: If we consider, for simplicity, a gradient descent training dynamic, then
Revised: If we consider, for simplicity, a gradient descent training dynamic, then

Original: [Equation]
Revised: [Equation]

Original: where φ t is the embedding vector at the t -th step, t ∈ N + is the total number of iterations and φ 0 is a random initialisation.
Revised: where φ t is the embedding vector at the t -th step, t ∈ N + is the total number of iterations and φ 0 is a random initialisation.

Original: GD is the gradient descent operator:
Revised: GD is the gradient descent operator:

Original: [Equation]
Revised: [Equation]

Original: where α = β |T | − 1 , with a η > 0 learning rate.
Revised: where α = β |T | − 1 , with a η > 0 learning rate.

Original: We now dissect Equation (6) in two different (but equivalent) ways.
Revised: We now dissect Equation (6) in two different (but equivalent) ways.

Original: In the ﬁrst, which we dub the edge view , we separately consider each addend of the gradient ∇ φ L .
Revised: In the ﬁrst, which we dub the edge view , we separately consider each addend of the gradient ∇ φ L .

Original: In the second, we aggregate the contributions from all the triplets to the update of a particular node.
Revised: In the second, we aggregate the contributions from all the triplets to the update of a particular node.

Original: With this latter decomposition, which we call the node view , we can explicate the message-passing mechanism at the core of the FMs.
Revised: With this latter decomposition, which we call the node view , we can explicate the message-passing mechanism at the core of the FMs.

Original: While the edge view suits a vectorised implementation better, the node view further exposes the information ﬂow among nodes, allowing us to draw an analogy to message-passing GNNs.
Revised: While the edge view suits a vectorised implementation better, the node view further exposes the information ﬂow among nodes, allowing us to draw an analogy to message-passing GNNs.

Original: 3.1 The Edge View
Revised: 3.1 The Edge View

Original: Each addend of Equation (6) corresponds to a single edge ( v, r, w ) ∈ T and contributes to the update of the representation of all nodes.
Revised: Each addend of Equation (6) corresponds to a single edge ( v, r, w ) ∈ T and contributes to the update of the representation of all nodes.

Original: The update on the representation of the subject φ [ v ] is:
Revised: The update on the representation of the subject φ [ v ] is:

Original: [Equation]
Revised: [Equation]

Original: The w → v term indicates information ﬂow from w (a neighbour of v ) to v , increasing the score of the gold triplet ( v, r, w ) .
Revised: The w → v term indicates information ﬂow from w (a neighbour of v ) to v , increasing the score of the gold triplet ( v, r, w ) .

Original: The u → v term indicates information ﬂow from global nodes, decreasing the scores of triplets ( v, r, ?)
Revised: The u → v term indicates information ﬂow from global nodes, decreasing the scores of triplets ( v, r, ?)

Original: with v as the subject and r as the predicate.
Revised: with v as the subject and r as the predicate.

Original: Similarly, for the object w ,
Revised: Similarly, for the object w ,

Original: [Equation]
Revised: [Equation]

Original: , where, again, the v → w term indicates information ﬂow from the neighbouring node v .
Revised: where, again, the v → w term indicates information ﬂow from the neighbouring node v .

Original: Finally, for the nodes other than v and w [Equation] we have
Revised: Finally, for the nodes other than v and w , we have

Original: [Equation]
Revised: [Equation]

Original: 3.2 The Node View
Revised: 3.2 The Node View

Original: To fully uncover the message-passing mechanism of FMs
Revised: To fully uncover the message-passing mechanism of FMs

Original: [Equation]
Revised: [Equation]

Original: we now focus on the gradient descent operation over a single node v ∈ E , referred to as the central node in the GNN literature.
Revised: we now focus on the gradient descent operation over a single node v ∈ E , referred to as the central node in the GNN literature.

Original: Recalling
Revised: Recalling

Original: Equation (6), we have:
Revised: Equation (6), we have:

Original: [Equation]
Revised: [Equation]

Original: which aggregates the information stemming from the updates presented in the edge view.
Revised: which aggregates the information stemming from the updates presented in the edge view.

Original: The next theorem describes how this total information ﬂow to a particular node can be recast as an instance of message passing (cf.
Revised: The next theorem describes how this total information ﬂow to a particular node can be recast as an instance of message passing (cf.

Original: Section 2.2).
Revised: Section 2.2).

Original: We defer the proof to the appendix.
Revised: We defer the proof to the appendix.

Original: Theorem 3.1 (Message passing in FMs) .
Revised: Theorem 3.1 (Message passing in FMs) .

Original: The gradient descent operator GD (Equation (7) ) on the node embeddings of a DistMult model (Equation (4) ) with the maximum likelihood objective in Equation (3) and a multi-relational graph T deﬁned over entities E induces a message-passing operator whose composing functions are:
Revised: The gradient descent operator GD (Equation (7) ) on the node embeddings of a DistMult model (Equation (4) ) with the maximum likelihood objective in Equation (3) and a multi-relational graph T deﬁned over entities E induces a message-passing operator whose composing functions are:

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: m
Revised: m

Original: [Equation]
Revised: [Equation]

Original: v
Revised: v

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: U (
Revised: U (

Original: [Equation]
Revised: [Equation]

Original: v ]
Revised: v ]

Original: [Equation]
Revised: [Equation]

Original: [ v ]
Revised: [ v ]

Original: [Equation]
Revised: [Equation]

Original: [ v ]
Revised: [ v ]

Original: [Equation]
Revised: [Equation]

Original: αz [ v ]
Revised: αz [ v ]

Original: [Equation]
Revised: [Equation]

Original: βn [ v ]
Revised: βn [ v ]

Original: [Equation]
Revised: [Equation]

Original: where, deﬁning the sets of triplets T − v = { ( s, r, o ) ∈ T : s (cid:54) = v ∧ o (cid:54) = v } ,
Revised: where, deﬁning the sets of triplets T − v = { ( s, r, o ) ∈ T : s (cid:54) = v ∧ o (cid:54) = v } ,

Original: [Equation]
Revised: [Equation]

Original: where P N 1+ [ v ] and P T − v are the empirical probability distributions associated to the respective sets.
Revised: where P N 1+ [ v ] and P T − v are the empirical probability distributions associated to the respective sets.

Original: What emerges from the equations is that each GD step contains an explicit information ﬂow from the neighbourhood of each node, which is then aggregated with a simple summation.
Revised: What emerges from the equations is that each GD step contains an explicit information ﬂow from the neighbourhood of each node, which is then aggregated with a simple summation.

Original: Through this direct information path, t GD steps cover the t -hop neighbourhood of v .
Revised: Through this direct information path, t GD steps cover the t -hop neighbourhood of v .

Original: As t goes towards inﬁnity – or in practice – as training converges, FMs capture the global graph structure.
Revised: As t goes towards inﬁnity – or in practice – as training converges, FMs capture the global graph structure.

Original: The update function (10) somewhat deviates from classic message passing frameworks as n [ v ] of Equation (11) involves global information.
Revised: The update function (10) somewhat deviates from classic message passing frameworks as n [ v ] of Equation (11) involves global information.

Original: However, we note that we can interpret this mechanism under the framework of augmented message passing [41] and, in particular, as an instance of graph rewiring .
Revised: However, we note that we can interpret this mechanism under the framework of augmented message passing [41] and, in particular, as an instance of graph rewiring .

Original: Based on Theorem 3.1 and Equation (5), we can now view φ as the transient node states h (cf.
Revised: Based on Theorem 3.1 and Equation (5), we can now view φ as the transient node states h (cf.

Original: Section 2.2) and GD on node embeddings as a message-passing layer.
Revised: Section 2.2) and GD on node embeddings as a message-passing layer.

Original: This dualism sits at the core of the ReFactor GNN model, which we describe next.
Revised: This dualism sits at the core of the ReFactor GNN model, which we describe next.

Original:  R E F ACTOR GNNs
Revised:  R E F ACTOR GNNs

Original: FMs are trained by minimising the objective (3), initialising both sets of parameters ( φ and ψ ) and performing GD until approximate convergence (or until early stopping terminates the training).
Revised: FMs are trained by minimising the objective (3), initialising both sets of parameters ( φ and ψ ) and performing GD until approximate convergence (or until early stopping terminates the training).

Original: The implications are twofold: i ) the initial value of the entity lookup table φ does not play any major role in the ﬁnal model after convergence, and ii ) if we introduce a new set of entities, the conventional wisdom is to retrain 3 the model on the expanded knowledge graph.
Revised: The implications are twofold: i ) the initial value of the entity lookup table φ does not play any major role in the ﬁnal model after convergence, and ii ) if we introduce a new set of entities, the conventional wisdom is to retrain 3 the model on the expanded knowledge graph.

Original: This is computationally rather expensive compared to the “inductive” models that require no additional training and can leverage node features like entity descriptions.
Revised: This is computationally rather expensive compared to the “inductive” models that require no additional training and can leverage node features like entity descriptions.

Original: However, as we have just seen in Theorem 3.1, the training procedure of FMs may be naturally recast as a message-passing operation, which suggests that it is possible to use FMs for inductive learning tasks.
Revised: However, as we have just seen in Theorem 3.1, the training procedure of FMs may be naturally recast as a message-passing operation, which suggests that it is possible to use FMs for inductive learning tasks.

Original: In fact, we envision that there is an entire novel spectrum of model architectures interpolating between pure FMs and (various instantiations of) GNNs.
Revised: In fact, we envision that there is an entire novel spectrum of model architectures interpolating between pure FMs and (various instantiations of) GNNs.

Original: Here we propose one simple implementation of such an architecture which we dub
Revised: Here we propose one simple implementation of such an architecture which we dub

Original: R E F ACTOR GNN S .
Revised: R E F ACTOR GNN S .

Original: Figure 1 gives an overview of R E F ACTOR GNNs.
Revised: Figure 1 gives an overview of R E F ACTOR GNNs.

Original: The R E F ACTOR Layer A R E F ACTOR GNN contains L R E F ACTOR layers, that we derive from
Revised: The R E F ACTOR Layer A R E F ACTOR GNN contains L R E F ACTOR layers, that we derive from

Original: Theorem 3.1.
Revised: Theorem 3.1.

Original: Aligning with the notations in Section 2.2, given a KG T and entity representations h l − 1 ∈ R |E|× K , the R E F ACTOR layer computes the representation of a node v as follows:
Revised: Aligning with the notations in Section 2.2, given a KG T and entity representations h l − 1 ∈ R |E|× K , the R E F ACTOR layer computes the representation of a node v as follows:

Original: [Equation]
Revised: [Equation]

Original: where the terms n l and q l M are derived from Equation (11) and Equation (8), respectively.
Revised: where the terms n l and q l M are derived from Equation (11) and Equation (8), respectively.

Original: We note that R E F ACTOR GNN S treat incoming and outgoing neighbourhoods differently instead of treating them equally as in the R-GCN, the ﬁrst GNN on multi-relational graphs [33].
Revised: We note that R E F ACTOR GNN S treat incoming and outgoing neighbourhoods differently instead of treating them equally as in the R-GCN, the ﬁrst GNN on multi-relational graphs [33].

Original: Equation (12) describes the full batch setting, which can be expensive if the KG contains many edges.
Revised: Equation (12) describes the full batch setting, which can be expensive if the KG contains many edges.

Original: Therefore, in practice, whenever the graph is big, we adopt a stochastic evaluation of the
Revised: Therefore, in practice, whenever the graph is big, we adopt a stochastic evaluation of the

Original: R E F ACTOR layer by decomposing the evaluation into several mini-batches.
Revised: R E F ACTOR layer by decomposing the evaluation into several mini-batches.

Original: We partition T into a set of computationally tractable mini-batches.
Revised: We partition T into a set of computationally tractable mini-batches.

Original: For each of them, we restrict the neighbourhoods to the subparagraph induced by it and readjust the computation of n l [ v ] to include only entities and edges present in it.
Revised: For each of them, we restrict the neighbourhoods to the subgraph induced by it and readjust the computation of n l [ v ] to include only entities and edges present in it.

Original: We leave the investigation of other stochastic strategies (e.g. by taking Monte Carlo estimations of the expectations in Equation (11)) to future work.
Revised: We leave the investigation of other stochastic strategies (e.g. by taking Monte Carlo estimations of the expectations in Equation (11)) to future work.

Original: Finally, we cascade the mini-batch evaluation to produce one full layer evaluation.
Revised: Finally, we cascade the mini-batch evaluation to produce one full layer evaluation.

Original: Training The learnable parameters of R E F ACTOR GNN S are the relation embeddings ψ , which parameterise g ( r ) in q lM , l ∈ [1 , L ] .
Revised: Training The learnable parameters of R E F ACTOR GNN S are the relation embeddings ψ , which parameterise g ( r ) in q lM , l ∈ [1 , L ] .

Original: Inspired by [5, 46], we learn ψ by layer-wise (stochastic) gradient descent.
Revised: Inspired by [5, 46], we learn ψ by layer-wise (stochastic) gradient descent.

Original: This is in contrast to conventional GNN training, where we need to backpropagate through  Typically , possibly by starting v [Figure]
Revised: This is in contrast to conventional GNN training, where we need to backpropagate through 3 Typically until convergence, possibly by partially warm-starting θ .

Original: v
Revised: v

Original: [Figure]
Revised: [Figure]

Original: vv
Revised: vvv

Original: [Figure]
Revised: [Figure]

Revised: ϕ [ v

Revised: [Figure]

Revised: g ( r

Original: [Figure]
Revised: [Figure]

Original: ) r 1 rrw r Γ v
Revised: ) r 1 rrw r Γ v

Original: Softmax
Revised: Softmax

Original: P ( v ∣ w , r ) all the layers.
Revised: P ( v ∣ w , r ) all the layers.

Original: A (full-batch) GD training dynamic for ψ can be written as ψ t +1 =
Revised: A (full-batch) GD training dynamic for ψ can be written as ψ t +1 =

Original: ψ t − η ∇L t ( ψ t ) , where L t ( ψ t ) = −|T | − 1 (cid:80) T log P
Revised: ψ t − η ∇L t ( ψ t ) , where L t ( ψ t ) = −|T | − 1 (cid:80) T log P

Original: ψ t ( w | v, r ) , with:
Revised: ψ t ( w | v, r ) , with:

Original: [Equation]
Revised: [Equation]

Original: and the node state update as
Revised: and the node state update as

Original: [Equation]
Revised: [Equation]

Original: otherwise [Equation] Implementation-wise, such a training dynamic equals to using an external memory for storing historical node states h t − 1 akin to the procedure introduced in [5].
Revised: Implementation-wise, such a training dynamic equals to using an external memory for storing historical node states h t − 1 akin to the procedure introduced in [5].

Original: The cached node states can then be queried to compute h t using Equation (12).
Revised: The cached node states can then be queried to compute h t using Equation (12).

Original: From this perspective, we periodically clear the node state cache every L full batches to force the model to predict based on on-the-ﬂy L -layer message-passing.
Revised: From this perspective, we periodically clear the node state cache every L full batches to force the model to predict based on on-the-ﬂy L -layer message-passing.

Original: After training, we obtain ψ ∗ and perform inference by running L -layer message-passing with ψ ∗ .
Revised: After training, we obtain ψ ∗ and perform inference by running L -layer message-passing with ψ ∗ .

Original: In general, L determines the number of effective message-passing layers in R E F ACTOR GNN S .
Revised: In general, L determines the number of effective message-passing layers in R E F ACTOR GNN S .

Original: A larger L enables R E F ACTOR GNN S to fuse information from more hops of neighbourhoods into the ﬁnal node representations.
Revised: A larger L enables R E F ACTOR GNN S to fuse information from more hops of neighbourhoods into the ﬁnal node representations.

Original: In the meantime, it reduces the inductive applicability of R E F ACTOR
Revised: In the meantime, it reduces the inductive applicability of R E F ACTOR

Original: GNN S due to over-smoothing and computational requirements.
Revised: GNN S due to over-smoothing and computational requirements.

Original: In the extreme case of L = ∞ , where we never clear the node state cache during training , the ﬁnal cached node states will be used for inference.
Revised: In the extreme case of L = ∞ , where we never clear the node state cache during training , the ﬁnal cached node states will be used for inference.

Original: Note that this latter inference regime is inherently transductive since there will be no cached states for new nodes.
Revised: Note that this latter inference regime is inherently transductive since there will be no cached states for new nodes.

Original:  Experiments
Revised:  Experiments

Original: We perform experiments  to answer the following questions regarding R E F ACTOR GNN S :
Revised: We perform experiments 4 to answer the following questions regarding R E F ACTOR GNN S :

Original: • Q1.
Revised: • Q1.

Original: R E F ACTOR GNN S are derived from a message-passing reformulation of FMs: do they also inherit FMs’ predictive accuracy in transductive KGC tasks?
Revised: R E F ACTOR GNN S are derived from a message-passing reformulation of FMs: do they also inherit FMs’ predictive accuracy in transductive KGC tasks?

Original: (Section 5.1) • Q2.
Revised: (Section 5.1) • Q2.

Original: R E F ACTOR GNN S “inductivise” FMs.
Revised: R E F ACTOR GNN S “inductivise” FMs.

Original: Are they more statistically accurate than other GNN baselines in inductive KGC tasks?
Revised: Are they more statistically accurate than other GNN baselines in inductive KGC tasks?

Original: (Section 5.2) • Q3.
Revised: (Section 5.2) • Q3.

Original: The term n [ v ] involves nodes that are not in the 1-hop neighbourhood.
Revised: The term n [ v ] involves nodes that are not in the 1-hop neighbourhood.

Original: Is such augmented message passing [41] necessary for good KGC performance?
Revised: Is such augmented message passing [41] necessary for good KGC performance?

Original: (Section 5.3)
Revised: (Section 5.3)

Original: For transductive experiments, we used three well-established KGC datasets: UMLS [14], CoDExS [31], and FB15K237 [38].
Revised: For transductive experiments, we used three well-established KGC datasets: UMLS [14], CoDExS [31], and FB15K237 [38].

Original: For inductive experiments, we used the inductive KGC benchmarks introduced by GraIL [37], which include 12 pairs of knowledge graphs: ( FB15K237_v i ,
Revised: For inductive experiments, we used the inductive KGC benchmarks introduced by GraIL [37], which include 12 pairs of knowledge graphs: ( FB15K237_v i ,

Original: FB15K237_v i _ind ), ( WN18RR_v i , WN18RR_v
Revised: FB15K237_v i _ind ), ( WN18RR_v i , WN18RR_v

Original: i _ind ), and ( NELL_v i , NELL_v i _ind ), where i ∈ [1 , 2 , 3 , 4] , and ( _v i , _v i _ind ) represents a pair of graphs with a shared relation vocabulary and non-overlapping entities.
Revised: i _ind ), and ( NELL_v i , NELL_v i _ind ), where i ∈ [1 , 2 , 3 , 4] , and ( _v i , _v i _ind ) represents a pair of graphs with a shared relation vocabulary and non-overlapping entities.

Original: We follow the standard KGC evaluation protocol by fully ranking all thecandidate entities and computing two metrics using the ranks of the ground-truth entities: Mean
Revised: We follow the standard KGC evaluation protocol by fully ranking all the

Original: [Table]
Revised: [Table]

Revised: candidate entities and computing two metrics using the ranks of the ground-truth entities: Mean

Original: Reciprocal Ranking (MRR) and Hit Ratios at Top K (Hits@ K ) with K ∈ [1 , 3 , 10] .
Revised: Reciprocal Ranking (MRR) and Hit Ratios at Top K (Hits@ K ) with K ∈ [1 , 3 , 10] .

Original: For the inductive
Revised: For the inductive

Original: KGC, we additionally consider the partial-ranking evaluation protocol used by GraIL for a fair comparison.
Revised: KGC, we additionally consider the partial-ranking evaluation protocol used by GraIL for a fair comparison.

Original: Empirically, we ﬁnd full ranking more difﬁcult than partial ranking, and thus more suitable for reﬂecting the differences among models on GraIL datasets – we would like to call for future work on GraIL datasets to also adopt a full ranking protocol on these datasets.
Revised: Empirically, we ﬁnd full ranking more difﬁcult than partial ranking, and thus more suitable for reﬂecting the differences among models on GraIL datasets – we would like to call for future work on GraIL datasets to also adopt a full ranking protocol on these datasets.

Original: Our transductive experiments used L = ∞ , i.e. node states cache is never cleared, as we wanted to see if R E F ACTOR GNN S ( L = ∞ ) can reach the performance of the FMs; on the other hand, in our inductive experiments, we used R E F ACTOR GNN S with L ∈ { 1 , 2 , 3 , 6 , 9 } , since we wanted to test their performances in inductive settings akin to standard GNNs.
Revised: Our transductive experiments used L = ∞ , i.e. node states cache is never cleared, as we wanted to see if R E F ACTOR GNN S ( L = ∞ ) can reach the performance of the FMs; on the other hand, in our inductive experiments, we used R E F ACTOR GNN S with L ∈ { 1 , 2 , 3 , 6 , 9 } , since we wanted to test their performances in inductive settings akin to standard GNNs.

Original: We used a hidden size offor the node representations.
Revised: We used a hidden size offor the node representations.

Original: All the models are trained using [128 , 512] in-batch negative samples and one global negative node for each positive link.
Revised: All the models are trained using [128 , 512] in-batch negative samples and one global negative node for each positive link.

Original: We performed a grid search over the other hyper-parameters and selected the best conﬁguration based on the validation MRR.
Revised: We performed a grid search over the other hyper-parameters and selected the best conﬁguration based on the validation MRR.

Original: Since training deep GNNs with full-graph message passing might be slow for large knowledge graphs, we follow the literature [10, 50, 47] to sample sub-graphs for training GNNs.
Revised: Since training deep GNNs with full-graph message passing might be slow for large knowledge graphs, we follow the literature [10, 50, 47] to sample sub-graphs for training GNNs.

Original: Considering that sampling on the ﬂy often prevents high utilisation of GPUs, we resort to a two-stage process: we ﬁrst sampled and serialised sub-graphs around the target edges in the mini-batches; we then trained the GNNs with the serialised sub-graphs.
Revised: Considering that sampling on the ﬂy often prevents high utilisation of GPUs, we resort to a two-stage process: we ﬁrst sampled and serialised sub-graphs around the target edges in the mini-batches; we then trained the GNNs with the serialised sub-graphs.

Original: To ensure we have sufﬁcient sub-graphs for training the models, we sampled for 20 epochs for each knowledge graph, i.e. 20 full passes over the full graph.
Revised: To ensure we have sufﬁcient sub-graphs for training the models, we sampled for 20 epochs for each knowledge graph, i.e. 20 full passes over the full graph.

Original: The sub-graph sampler we currently used is LADIES [50].
Revised: The sub-graph sampler we currently used is LADIES [50].

Original: 5. R E F ACTOR GNN S for Transductive Learning (Q1)
Revised: 5. R E F ACTOR GNN S for Transductive Learning (Q1)

Original: R E F ACTOR GNN S are derived from the message-passing reformulation of FMs.
Revised: R E F ACTOR GNN S are derived from the message-passing reformulation of FMs.

Original: We expect them to approximate the performance of FMs for transductive KGC tasks.
Revised: We expect them to approximate the performance of FMs for transductive KGC tasks.

Original: To verify this, we run experiments on the datasets UMLS, CoDEx-S, and FB15K237.
Revised: To verify this, we run experiments on the datasets UMLS, CoDEx-S, and FB15K237.

Original: For a fair comparison, we use Equation (2) as the decoder and consider i) lookup embedding table as the entity encoder, which forms the FM when combined with the decoder (Section 2.1), and ii) R E F ACTOR GNN S as the entity encoder.
Revised: For a fair comparison, we use Equation (2) as the decoder and consider i) lookup embedding table as the entity encoder, which forms the FM when combined with the decoder (Section 2.1), and ii) R E F ACTOR GNN S as the entity encoder.

Original: R E F ACTOR
Revised: R E F ACTOR

Original: GNN S are trained with L = ∞ , i.e. we never clear the node state cache.
Revised: GNN S are trained with L = ∞ , i.e. we never clear the node state cache.

Original: Since transductive KGC tasks do not involve new entities, the node state cache in R E F ACTOR GNN S can be directly used for link prediction.
Revised: Since transductive KGC tasks do not involve new entities, the node state cache in R E F ACTOR GNN S can be directly used for link prediction.

Original: Table 1 summarises the result.
Revised: Table 1 summarises the result.

Original: We observe that R E F ACTOR GNN S achieve a similar or slightly better performance compared to the FM.
Revised: We observe that R E F ACTOR GNN S achieve a similar or slightly better performance compared to the FM.

Original: This shows that R E F ACTOR GNN S are able to capture the essence of FMs and thus maintain strong at transductive KGC.
Revised: This shows that R E F ACTOR GNN S are able to capture the essence of FMs and thus maintain strong at transductive KGC.

Original: 5. R E F ACTOR GNN S for Inductive Learning (Q2)
Revised: 5. R E F ACTOR GNN S for Inductive Learning (Q2)

Original: Despite FMs’ good empirical performance on transductive KGC tasks, they fail to be inductive as
Revised: Despite FMs’ good empirical performance on transductive KGC tasks, they fail to be inductive as

Original: GNNs.
Revised: GNNs.

Original: According to our reformulation, this is due to the inﬁnite message-passing layers hidden in FMs’ optimisation.
Revised: According to our reformulation, this is due to the inﬁnite message-passing layers hidden in FMs’ optimisation.

Original: Discarding inﬁnite message-passing layers, R E F ACTOR GNN S enable FMs to perform inductive reasoning tasks by learning to use a ﬁnite set of message-passing layers for prediction similarly to GNNs.
Revised: Discarding inﬁnite message-passing layers, R E F ACTOR GNN S enable FMs to perform inductive reasoning tasks by learning to use a ﬁnite set of message-passing layers for prediction similarly to GNNs.

Original: Here we present experiments to verify R E F ACTOR GNN S ’s capability for inductive reasoning.
Revised: Here we present experiments to verify R E F ACTOR GNN S ’s capability for inductive reasoning.

Original: Speciﬁcally, we study the task of inductive KGC and investigate whether R E F ACTOR GNN S can generalise to unseen entities.
Revised: Speciﬁcally, we study the task of inductive KGC and investigate whether R E F ACTOR GNN S can generalise to unseen entities.

Original: Following [37], on GraIL datasets, we trained models on the original graph, and run 0-shot link prediction on the _ind test graph.
Revised: Following [37], on GraIL datasets, we trained models on the original graph, and run 0-shot link prediction on the _ind test graph.

Original: Similar as the transductive experiments, we use Equation (2) as the decoder and vary the entity encoder.
Revised: Similar to the transductive experiments, we use Equation (2) as the decoder and vary the entity encoder.

Original: We denote three-layer R E F ACTOR
Revised: We denote three-layer R E F ACTOR

Original: GNN S as ReFactor GNNs (3) and six-layer R E F ACTOR GNN S as ReFactor GNNs (6) .
Revised: GNN S as ReFactor GNNs (3) and six-layer R E F ACTOR GNN S as ReFactor GNNs (6) .

Original: We consider several baseline entity encoders: i) no-pretrain , models without any pretraining on the original graph; ii) GAT(3) , three-layer graph attention network [40]; iii) GAT(6) , six-layer graph attention network; iv) GraIL , a sub-graph-based relational GNN [37]; v) NBFNet , a pathbased GNN [49], current SoTA on GraIL datasets. In addition to randomly initialised vectors
Revised: We consider several baseline entity encoders: i) no-pretrain , models without any pretraining on the original graph; ii) GAT(3) , three-layer graph attention network [40]; iii) GAT(6) , six-layer graph attention network; iv) GraIL , a sub-graph-based relational GNN [37]; v) NBFNet , a path-

Original: [Figure]
Revised: [Figure]

Original: With Random 
Revised: With Random Features

Original: [Figure]
Revised: With Textual Features based GNN [49], current SoTA on GraIL datasets.

Original: With Textual Features as the node features, we also used textual node features, RoBERTa [20] Encodings of the entity descriptions, which are produced by SentenceBERT [29].
Revised: In addition to randomly initialised vectors as the node features, we also used textual node features, RoBERTa [20] Encodings of the entity descriptions, which are produced by SentenceBERT [29].

Original: Due to space reason, we present the results on ( FB15K237_v 1 , FB15K237_v 1 _ind ) in Figure 2.
Revised: Due to space reason, we present the results on ( FB15K237_v 1 , FB15K237_v 1 _ind ) in Figure 2.

Original: Results on other datasets are similar and can be found in the appendix.
Revised: Results on other datasets are similar and can be found in the appendix.

Original: We can see that without textual node features, R E F ACTOR GNN S perform better than GraIL (+23%); with textual node features, R E F ACTOR GNN S outperform both GraIL (+43%) and NBFNet (+10%), achieving new SoTA results on inductive KGC.
Revised: We can see that without textual node features, R E F ACTOR GNN S perform better than GraIL (+23%); with textual node features, R E F ACTOR GNN S outperform both GraIL (+43%) and NBFNet (+10%), achieving new SoTA results on inductive KGC.

Original: Performance vs Parameter Efﬁciency as #Message-Passing Layers Increases Usually, as the number of message-passing layers increases in GNNs, the over-smoothing issue occurs while the computational cost also increases exponentially.
Revised: Performance vs Parameter Efﬁciency as #Message-Passing Layers Increases Usually, as the number of message-passing layers increases in GNNs, the over-smoothing issue occurs while the computational cost also increases exponentially.

Original: R E F ACTOR GNN S avoid this by layer-wise training and sharing the weights across layers.
Revised: R E F ACTOR GNN S avoid this by layer-wise training and sharing the weights across layers.

Original: Here we compare R E F ACTOR GNN S with { 1 , 3 , 6 , 9 } message-passing layer(s) with same-depth GATs – results are summarised in Figure 3.
Revised: Here we compare R E F ACTOR GNN S with { 1 , 3 , 6 , 9 } message-passing layer(s) with same-depth GATs – results are summarised in Figure 3.

Original: We observe that increasing the number of message-passing layers in GATs does not necessarily improve the predictive accuracy – the best results were obtained with 3 message-passing layers on FB15K237_v while using 6 and 9 layers leads to performance degradation.
Revised: We observe that increasing the number of message-passing layers in GATs does not necessarily improve the predictive accuracy – the best results were obtained with 3 message-passing layers on FB15K237_v while using 6 and 9 layers leads to performance degradation.

Original: On the other hand, R E F ACTOR GNN S obtain consistent improvements when increasing #Layers from 1 to 3, 6, and 9.
Revised: On the other hand, R E F ACTOR GNN S obtain consistent improvements when increasing #Layers from 1 to 3, 6, and 9.

Original: R E F ACTOR GNN S (6 , 6) and (9 , 9) clearly outperform their GAT counterparts.
Revised: R E F ACTOR GNN S (6 , 6) and (9 , 9) clearly outperform their GAT counterparts.

Original: Most importantly, R E F ACTOR GNN S are more parameter-efﬁcient than GATs, with a constant #Parameters as #Layers increases.
Revised: Most importantly, R E F ACTOR GNN S are more parameter-efﬁcient than GATs, with a constant #Parameters as #Layers increases.

Original: 5.3 Beyond Message-Passing (Q3)
Revised: 5.3 Beyond Message-Passing (Q3)

Original: As shown by Theorem 3.1, R E F ACTOR GNN S contain not only terms capturing information ﬂow from the 1-hop neighbourhood, which falls into the classic message-passing framework, but also a term n [ v ] that involve nodes outside the 1-hop neighbourhood.
Revised: As shown by Theorem 3.1, R E F ACTOR GNN S contain not only terms capturing information ﬂow from the 1-hop neighbourhood, which falls into the classic message-passing framework, but also a term n [ v ] that involve nodes outside the 1-hop neighbourhood.

Original: The term n [ v ] can be treated as augmented message-passing on a dynamically rewired graph [41].
Revised: The term n [ v ] can be treated as augmented message-passing on a dynamically rewired graph [41].

Original: Here we perform ablation experiments to measure the impact of the n [ v ] term.
Revised: Here we perform ablation experiments to measure the impact of the n [ v ] term.

Original: Table 2 summarises the ablation results: we can see that, without the term n [ v ] , R E F ACTOR GNN S with random vectors as node features yield a 2% lower MRR, while R E F ACTOR GNN S with RoBERTa textual encodings as node features produce a 7% lower MRR.
Revised: Table 2 summarises the ablation results: we can see that, without the term n [ v ] , R E F ACTOR GNN S with random vectors as node features yield a 2% lower MRR, while R E F ACTOR GNN S with RoBERTa textual encodings as node features produce a 7% lower MRR.

Original: This suggests that augmented message-passing also plays a signiﬁcant role in
Revised: This suggests that augmented message-passing also plays a signiﬁcant role in

Original: R E F ACTOR GNN S ’ generalisation properties in downstream link prediction tasks.
Revised: [Figure]

Original: Future work might gain more insights by further dissecting the n [ v ] term.
Revised: T

Original: [Figure]
Revised: [Figure]

Original: T es t
Revised: t

Original: M RR # P a r a m s ( M ) 0.
Revised: M RR # P a r a m s ( M ) 0.

Original: Test MRR
Revised: Test MRR

Original: With Random Features
Revised: With Random Features

Original: With Textual Features 
Revised: With Textual Features with n [ v [Table] without n [

Original: [Table]
Revised: [Table]

Revised: R E F ACTOR GNN S ’ generalisation properties in downstream link prediction tasks.

Revised: Future work might gain more insights by further dissecting the n [ v ] term.

Original:  Related Work
Revised:  Related Work

Original: Multi-Relational Graph Representation Learning Multi-relational graph representation learning concernsabout graphs with various edge types.
Revised: Multi-Relational Graph Representation Learning Multi-relational graph representation learning concerns graphs with various edge types.

Original: Another relevant line of work would be representation learning over heterogeneous graphs, where node types are also considered.
Revised: Another relevant line of work would be representation learning over heterogeneous graphs, where node types are also considered.

Original: Previous work on multi-relational graph representation learning focused either on FMs [27, 39, 45, 17, 26, 3, 24, 2] or on GNN-based models [33, 44, 48, 18].
Revised: Previous work on multi-relational graph representation learning focused either on FMs [27, 39, 45, 17, 26, 3, 24, 2] or on GNN-based models [33, 44, 48, 18].

Original: Similar to a recent ﬁnding in a benchmark study over heterogeneous GNNs [21], where the best choices of GNNs for heterogeneous graphs seem to regress to simple homogeneous GNN baselines, the progress of multi-relational graph representation learning also mingles with FMs, the very classic multi-relational link predictors.
Revised: Similar to a recent ﬁnding in a benchmark study over heterogeneous GNNs [21], where the best choices of GNNs for heterogeneous graphs seem to regress to simple homogeneous GNN baselines, the progress of multi-relational graph representation learning also mingles with FMs, the very classic multi-relational link predictors.

Original: Recently, FMs were found to be signiﬁcantly more accurate than GNNs in KGC tasks, when coupled with speciﬁc training strategies [30, 12, 17].
Revised: Recently, FMs were found to be signiﬁcantly more accurate than GNNs in KGC tasks, when coupled with speciﬁc training strategies [30, 12, 17].

Original: While more advanced GNNs [49] for KBC are showing promise at the cost of extra algorithm complexity, little effort has been devoted to establish the links between plain GNNs and FMs, which are strong multi-relational link predictors despite their simplicity.
Revised: While more advanced GNNs [49] for KBC are showing promise at the cost of extra algorithm complexity, little effort has been devoted to establishing the links between plain GNNs and FMs, which are strong multi-relational link predictors despite their simplicity.

Original: Our work aims to align GNNs with FMs so that we can combine the strengths from both families of models.
Revised: Our work aims to align GNNs with FMs so that we can combine the strengths from both families of models.

Original: Relationships between FMs and GNNs A very recent work [35] builds a theoretical link between structural GNNs and node (positional) embeddings.
Revised: Relationships between FMs and GNNs A very recent work [35] builds a theoretical link between structural GNNs and node (positional) embeddings.

Original: However, on one end of the link, the second model category encompasses not merely factorisation-based models but also many practical graph neural networks, between which the connection is unknown.
Revised: However, on one end of the link, the second model category encompasses not merely factorisation-based models but also many practical graph neural networks, between which the connection is unknown.

Original: Our work instead offers a more practical link between positional node embeddings produced by FMs and positional node embeddings produced by
Revised: Our work instead offers a more practical link between positional node embeddings produced by FMs and positional node embeddings produced by

Original: GNNs, while at the same time focusing on KGC.
Revised: GNNs, while at the same time focusing on KGC.

Original: Beyond FMs in KGC, using graph signal processing theory, [34] show that matrix factorisation (MF) based recommender models correspond to ideal low-pass graph convolutional ﬁlters.
Revised: Beyond FMs in KGC, using graph signal processing theory, [34] show that matrix factorisation (MF) based recommender models correspond to ideal low-pass graph convolutional ﬁlters.

Original: They also ﬁnd inﬁnite neighbourhood coverage in MF although using a completely different approach and focusing on a different domain in contrast to our work.
Revised: They also ﬁnd inﬁnite neighbourhood coverage in MF although using a completely different approach and focusing on a different domain in contrast to our work.

Original: Message Passing Message-passing is itself a broad terminology – people generally talk about it under two different contexts.
Revised: Message Passing Message-passing is itself a broad terminology – people generally talk about it under two different contexts.

Original: Firstly, as a computational technique, message passing allowsto recursively decompose a global function into simple local, parallelisable computations [22], thus being widely used for solvingthe inference problems in a graphical model.
Revised: Firstly, as a computational technique, message passing allows recursively decomposing a global function into simple local, parallelisable computations [22], thus being widely used for solving inference problems in a graphical model.

Original: Speciﬁcally, we note that message passing based inference techniques were proposed for matrix completion based recommendation [15] and Bayesian Boolean data decomposition [28] in  pre-deep-learning era.
Revised: Speciﬁcally, we note that message passingbased inference techniques were proposed for matrix completion-based recommendation [15] and Bayesian Boolean data decomposition [28] in the pre-deep-learning era.

Original: Secondly, as a paradigm of parameterising learnable functions over graph-structured data , message-passing has recently been used to provide a uniﬁed reformulation [7] for various GNN architectures, including Graph AttentionNetworks [40], Gated Graph Neural Networks [19], and
Revised: Secondly, as a paradigm of parameterising learnable functions over graph-structured data , message-passing has recently been used to provide a uniﬁed reformulation [7] for various GNN architectures, including Graph Attention

Original:  Graph Convolutional Networks [16].
Revised: Networks [40], Gated Graph Neural Networks [19], and Graph Convolutional Networks [16].

Original: In this work, we show that FMs can also be cast as a special type of message-passing GNNs via considering the gradient descent updates [1] over node embeddings as message-passing operations between nodes.
Revised: In this work, we show that FMs can also be cast as a special type of message-passing GNNs by considering the gradient descent updates [1] over node embeddings as message-passing operations between nodes.

Original: To the best of our knowledge, our work is the ﬁrst to provide such connections between FMs and message-passing GNNs.
Revised: To the best of our knowledge, our work is the ﬁrst to provide such connections between FMs and message-passing GNNs.

Original: We show that FMs can be seen as instances of GNNs, with a characteristic feature about the nodes being considered during the message-passing process: our R E F ACTOR GNN S can be seen as using an Augmented Message-Passing process on a dynamically re-wired graph [41].
Revised: We show that FMs can be seen as instances of GNNs, with a characteristic feature about the nodes being considered during the message-passing process: our R E F ACTOR GNN S can be seen as using an Augmented Message-Passing process on a dynamically re-wired graph [41].

Original:  Conclusion & Future Work
Revised:  Conclusion & Future Work

Original: Conclusion.
Revised: Conclusion.

Original: Motivated by the need of understanding FMs and GNNs despite the sharp differences in their analytical forms, our work establishes a link between FMs and GNNs on the task of multirelational link prediction.
Revised: Motivated by the need of understanding FMs and GNNs despite the sharp differences in their analytical forms, our work establishes a link between FMs and GNNs on the task of multirelational link prediction.

Original: The reformulation of FMs as GNNs addresses the question why FMs are stronger multi-relational link predictors compared to plain GNNs.
Revised: The reformulation of FMs as GNNs addresses the question why FMs are stronger multi-relational link predictors compared to plain GNNs.

Original: Guided by the reformulation, we further propose a new variant of GNNs, R E F ACTOR GNN S , which combines the strengths of both
Revised: Guided by the reformulation, we further propose a new variant of GNNs, R E F ACTOR GNN S , which combines the strengths of both

Original: GNNs and classic FMs.
Revised: GNNs and classic FMs.

Original: Empirical experiments show that R E F ACTOR GNN S produce signiﬁcantly more accurate results than our GNN baselines on inductive link prediction tasks.
Revised: Empirical experiments show that R E F ACTOR GNN S produce signiﬁcantly more accurate results than our GNN baselines on inductive link prediction tasks.

Original: Limitations.
Revised: Limitations.

Original: Since we adopted a two-stage (sub-graph serialisation and then model training) approach instead of online sampling, there can be side effects from the low sub-graph diversity.
Revised: Since we adopted a two-stage (sub-graph serialisation and then model training) approach instead of online sampling, there can be side effects from the low sub-graph diversity.

Original: In our experiments, we used LADIES [50] for sub-graph sampling.
Revised: In our experiments, we used LADIES [50] for sub-graph sampling.

Original: Experiments with different sub-graph sampling algorithms, such as GraphSaint [47] might affect the downstream link prediction results.
Revised: Experiments with different sub-graph sampling algorithms, such as GraphSaint [47] might affect the downstream link prediction results.

Original: Furthermore, it would be interesting to analyse decoders other than DistMult, as well as additional optimisation schemes beyond SGD and AdaGrad.
Revised: Furthermore, it would be interesting to analyse decoders other than DistMult, as well as additional optimisation schemes beyond SGD and AdaGrad.

Original: We do not dive deeper into the expressiveness of
Revised: We do not dive deeper into the expressiveness of

Original: R E F ACTOR GNN S .
Revised: R E F ACTOR GNN S .

Original: Nevertheless, we offer a brief discussion in Appendix F.
Revised: Nevertheless, we offer a brief discussion in Appendix F.

Original: Future Work.
Revised: Future Work.

Original: The most direct future work would be using the insight to develop more sophisticated models at the crossroad between FMs and GNNs, e.g. by further parameterising the message/update function.
Revised: The most direct future work would be using the insight to develop more sophisticated models at the crossroad between FMs and GNNs, e.g. by further parameterising the message/update function.

Original: One implication from our work is that reformulating FMs as message-passing enables the idea of “learning to factorize”.
Revised: One implication from our work is that reformulating FMs as message-passing enables the idea of “learning to factorize”.

Original: This might broaden the usages of FMs, going beyond link prediction, to tasks such as graph classiﬁcation.
Revised: This might broaden the usages of FMs, going beyond link prediction, to tasks such as graph classiﬁcation.

Original: Another implication comes from our approach of unpacking embedding updates into a series of message-passing operations.
Revised: Another implication comes from our approach of unpacking embedding updates into a series of message-passing operations.

Original: This approach can possibly generalise to other dot-product based models that use embedding layers for processing the inputs. In this sense, our work paves the way towards understanding complicated attention-based models e.g. transformers.
Revised: This approach can be generalised to other dot-product-based models that use embedding layers for processing the inputs, e.g. transformers.

Original: Although transformers can be treated as GNNs over fully-connected graphs, where a sentence would be a graph and its tokens would be the nodes, the message-passing is limited to within each sentence under this view.

Original: We instead envision cross-sentence message-passing by reformulating the updates of the token embedding layer in transformers.

Original: In general, the direction of organising FMs, GNNs, and transformers under the same framework will allow  better understanding of all three models.
Revised: In general, the direction of organising FMs, GNNs, and transformers under the same framework will allow a better understanding of all three models.

Original: Acknowledgements
Revised: Acknowledgements

Original: We would like to thank all reviewers for their constructive comments.
Revised: We would like to thank all reviewers for their constructive comments.

Original: We thank people with whom we had some really interesting chats throughout the project: Zhaocheng Zhu, Floris Geerts, Muhan
Revised: We thank people with whom we had some really interesting chats throughout the project: Zhaocheng Zhu, Floris Geerts, Muhan

Original: Zhang, Patrick Lewis, Hanqing Zeng, Shalini Maiti, Yifei Shen, Yinglong Xia, Matthias Fey, Jing
Revised: Zhang, Patrick Lewis, Hanqing Zeng, Shalini Maiti, Yifei Shen, Yinglong Xia, Matthias Fey, Jing

Original: Zhu,  and Maximilian Nickel.
Revised: Zhu, Marc Deisenroth and Maximilian Nickel.

Original: Yihong would like to thank the UCL-Facebook AIResearch joint PhD
Revised: Yihong would like to thank the UCL-Facebook AI

Revised: Research joint PhD program for generously funding her PhD. Pasquale and Pontus were partially funded by the European Union’s Horizon 2020 research and innovation programme under grant agreement no. 875160, and by an industry grant from Cisco.