Original: O N THE E QUIVALENCE BETWEEN P OSITIONAL N ODE E MBEDDINGS AND S TRUCTURAL G RAPH R EPRESENTATIONS
Revised: O N THE E QUIVALENCE BETWEEN P OSITIONAL N ODE E MBEDDINGS AND S TRUCTURAL G RAPH R EPRESENTATIONS

Original: bsriniv@purdue.edu
Revised: bsriniv@purdue.edu

Original: A BSTRACT
Revised: A BSTRACT

Original: This work provides the ﬁrst unifying theoretical framework for node (positional) embeddings and structural graph representations, bridging methods like matrix factorization and graph neural networks.
Revised: This work provides the ﬁrst unifying theoretical framework for node (positional) embeddings and structural graph representations, bridging methods like matrix factorization and graph neural networks.

Original: Using invariant theory, we show that the relationship between structural representations and node embeddings is analogous to that of a distribution and its samples.
Revised: Using invariant theory, we show that the relationship between structural representations and node embeddings is analogous to that of a distribution and its samples.

Original: We prove that all tasks that can be performed by node embeddings can also be performed by structural representations and vice-versa.
Revised: We prove that all tasks that can be performed by node embeddings can also be performed by structural representations and vice-versa.

Original: We also show that the concept of transductive and inductive learning is unrelated to node embeddings and graph representations, clearing another source of confusion in the literature.
Revised: We also show that the concept of transductive and inductive learning is unrelated to node embeddings and graph representations, clearing another source of confusion in the literature.

Original: Finally, we introduce new practical guidelines to generating and using node embeddings, which ﬁxes signiﬁcant shortcomings of standard operating procedures used today.
Revised: Finally, we introduce new practical guidelines to generating and using node embeddings, which ﬁxes signiﬁcant shortcomings of standard operating procedures used today.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: The theory of structural graph representations is a recently emerging ﬁeld.
Revised: The theory of structural graph representations is a recently emerging ﬁeld.

Original: It creates a link between relational learning and invariant theory.
Revised: It creates a link between relational learning and invariant theory.

Original: Interestingly, or rather unfortunately, there is no uniﬁed theory connecting node embeddings —low-rank matrix approximations, factor analysis, latent semantic analysis, etc.— with structural graph representations.
Revised: Interestingly, or rather unfortunately, there is no uniﬁed theory connecting node embeddings —low-rank matrix approximations, factor analysis, latent semantic analysis, etc.— with structural graph representations.

Original: Instead, conﬂicting interpretations have manifested over the last few years, that further confound practitioners and researchers alike.
Revised: Instead, conﬂicting interpretations have manifested over the last few years, that further confound practitioners and researchers alike.

Original: For instance, consider the direction, word embeddings → structural representations , where the structural equivalence between men → king and women → queen is described as being obtained by just adding or subtracting their node embeddings (positions in the embedding space)
Revised: For instance, consider the direction, word embeddings → structural representations , where the structural equivalence between men → king and women → queen is described as being obtained by just adding or subtracting their node embeddings (positions in the embedding space)

Original: (Arora et al., 2016;
Revised: (Arora et al., 2016;

Original: Mikolov et al., 2013).
Revised: Mikolov et al., 2013).

Original: Hence, can all (positional) node embeddings provide structural relationships akin to word analogies?
Revised: Hence, can all (positional) node embeddings provide structural relationships akin to word analogies?

Original: We provide a visual example in Appendix (Section 7) using the food web of Figure 1.
Revised: We provide a visual example in Appendix (Section 7) using the food web of Figure 1.

Original: In the opposite direction, structural representations → node embeddings , graph neural networks (GNNs) are often optimized to predict edges even though their structural node representations are provably incapable of performing the task.
Revised: In the opposite direction, structural representations → node embeddings , graph neural networks (GNNs) are often optimized to predict edges even though their structural node representations are provably incapable of performing the task.

Original: For instance, the node representations of the lynx and the orca in Figure 1 are indistinguishable due to an isomorphic equivalence between the nodes, making any edge prediction task that distinguishes the edges of lynx and orca a seemly futile exercise (see Appendix (Section 7) for more details).
Revised: For instance, the node representations of the lynx and the orca in Figure 1 are indistinguishable due to an isomorphic equivalence between the nodes, making any edge prediction task that distinguishes the edges of lynx and orca a seemly futile exercise (see Appendix (Section 7) for more details).

Original: Hence, are structural representations in general —and GNNs in particular— fundamentally incapable of performing link (dyadic) and multi-ary (polyadic) predictions tasks?
Revised: Hence, are structural representations in general —and GNNs in particular— fundamentally incapable of performing link (dyadic) and multi-ary (polyadic) predictions tasks?

Original: GNNs, however, can perform node classiﬁcation tasks, which is a task not associated with positional node embeddings (see Appendix (Section 7) for a concrete visual interpretation of the differences between positional node embeddings and structural representations over node classiﬁcation and link prediction tasks).
Revised: GNNs, however, can perform node classiﬁcation tasks, which is a task not associated with positional node embeddings (see Appendix (Section 7) for a concrete visual interpretation of the differences between positional node embeddings and structural representations over node classiﬁcation and link prediction tasks).

Original: Conﬁrmation bias has seemingly appeared to thwart recent efforts to bring node embeddings and structural representations into a single overarching framework.
Revised: Conﬁrmation bias has seemingly appeared to thwart recent efforts to bring node embeddings and structural representations into a single overarching framework.

Original: Preconceived notions of the two being fundamentally different (see Appendix (Section 7)) have been reinforced in the existing literature, arguing they belong in different applications: (Positional) Node embeddings would ﬁnd applications in multi-ary relationships such as link prediction, clustering, and natural language processing and knowledge acquisition through word and entity embeddings.
Revised: Preconceived notions of the two being fundamentally different (see Appendix (Section 7)) have been reinforced in the existing literature, arguing they belong in different applications: (Positional) Node embeddings would ﬁnd applications in multi-ary relationships such as link prediction, clustering, and natural language processing and knowledge acquisition through word and entity embeddings.

Original: Structural representations[Figure] would ﬁnd applications in node classiﬁcation, graph classiﬁcation, and role discovery.
Revised: Structural representations would ﬁnd applications in node classiﬁcation, graph classiﬁcation, and role discovery.

Original: A uniﬁed theory is required if we wish to eliminate these artiﬁcial boundaries, and better cross-pollinate, node embeddings and structural representations in novel techniques.
Revised: A uniﬁed theory is required if we wish to eliminate these artiﬁcial boundaries, and better cross-pollinate, node embeddings and structural representations in novel techniques.

Original: Contributions :
Revised: Contributions :

Original: In this work we use invariant theory and axiomatic counterfactuals (causality) to develop a uniﬁed theoretical framework that clariﬁes the differences between node embeddings and  structural representations and emphasizes their correspondence.
Revised: In this work we use invariant theory and axiomatic counterfactuals (causality) to develop a uniﬁed theoretical framework that clariﬁes the differences between node embeddings and [Figure] structural representations and emphasizes their correspondence.

Original: More speciﬁcally, (a) we show that structural representations and node embeddings have the same relationship as distributions and their samples; (b) we prove that all tasks that can be performed by node embeddings can also be performed by structural representations and vice-versa.
Revised: More speciﬁcally, (a) we show that structural representations and node embeddings have the same relationship as distributions and their samples; (b) we prove that all tasks that can be performed by node embeddings can also be performed by structural representations and vice-versa.

Original: Moreover, (c) we introduce new guidelines to creating and using node embeddings, which we hope will replace the less-than-optimal standard operating procedures used today.
Revised: Moreover, (c) we introduce new guidelines to creating and using node embeddings, which we hope will replace the less-than-optimal standard operating procedures used today.

Original: Finally, (d) we show that the concepts of transductive and inductive learning —commonly used to describe relational methods— are unrelated to node embeddigns and structural representations.
Revised: Finally, (d) we show that the concepts of transductive and inductive learning —commonly used to describe relational methods— are unrelated to node embeddigns and structural representations.

Original:  P RELIMINARIES
Revised:  P RELIMINARIES

Original: This section introduces some basic deﬁnitions, attempting to keep the mathematical jargon in check as much as we can, sometimes even sacriﬁcing generality for clarity.
Revised: This section introduces some basic deﬁnitions, attempting to keep the mathematical jargon in check as much as we can, sometimes even sacriﬁcing generality for clarity.

Original: We recommend Bloem-Reddy & Teh (2019) for a more formal description of some of the deﬁnitions in this section.
Revised: We recommend Bloem-Reddy & Teh (2019) for a more formal description of some of the deﬁnitions in this section.

Original: Deﬁnition 1 (Graph) .
Revised: Deﬁnition 1 (Graph) .

Original: We consider either a directed or an undirected attributed graph, denoted by
Revised: We consider either a directed or an undirected attributed graph, denoted by

Original: G =
Revised: G =

Original: ( V, E, X , E ) , where V is a set of n = | V | vertices, E is the set of edges in V × V , with matrix X ∈ R n × k and 3-mode tensor E ∈ R n × n × k (cid:48) representing the node and edge features, respectively.
Revised: ( V, E, X , E ) , where V is a set of n = | V | vertices, E is the set of edges in V × V , with matrix X ∈ R n × k and 3-mode tensor E ∈ R n × n × k (cid:48) representing the node and edge features, respectively.

Original: The edge set has an associated adjacency matrix A ∈ { 0 , 1 } n × n .
Revised: The edge set has an associated adjacency matrix A ∈ { 0 , 1 } n × n .

Original: In order to simplify notation, we will compress E and A into a single tensor A ∈ R n × n × ( k (cid:48) +1) .
Revised: In order to simplify notation, we will compress E and A into a single tensor A ∈ R n × n × ( k (cid:48) +1) .

Original: When explicit vertex and edge features and weights are unavailable, we will consider X = 11 T and A =
Revised: When explicit vertex and edge features and weights are unavailable, we will consider X = 11 T and A =

Original: A , whereis a n × 1 vector of ones.
Revised: A , whereis a n × 1 vector of ones.

Original: We will abuse notation and denote the graph as G =
Revised: We will abuse notation and denote the graph as G =

Original: ( A , X ) .
Revised: ( A , X ) .

Original: Without loss of generality, we deﬁne number the nodes in V = { 1 , . . . , n } following the same ordering as the adjacency tensor A and the rows in X .
Revised: Without loss of generality, we deﬁne number the nodes in V = { 1 , . . . , n } following the same ordering as the adjacency tensor A and the rows in X .

Original: We denote (cid:126)S and a vector of the elements in S ∈ P (cid:63) ( V ) sorted in ascending order, where P (cid:63) ( V ) is the power set of V without the empty set.
Revised: We denote (cid:126)S and a vector of the elements in S ∈ P (cid:63) ( V ) sorted in ascending order, where P (cid:63) ( V ) is the power set of V without the empty set.

Original: One of the most important operators in our mathematical toolkit will be that of a permutation action, orbits, G -invariance, and G -equivariance:
Revised: One of the most important operators in our mathematical toolkit will be that of a permutation action, orbits, G -invariance, and G -equivariance:

Original: Deﬁnition 2 (Permutation action π ) .
Revised: Deﬁnition 2 (Permutation action π ) .

Original: A permutation action π is a function that acts on any vector, matrix, or tensor deﬁned over the nodes V , e.g., ( Z i ) i ∈ V , and outputs an equivalent vector, matrix, or tensor with the order of the nodes permuted.
Revised: A permutation action π is a function that acts on any vector, matrix, or tensor deﬁned over the nodes V , e.g., ( Z i ) i ∈ V , and outputs an equivalent vector, matrix, or tensor with the order of the nodes permuted.

Original: We deﬁne Π n as the set of all n !
Revised: We deﬁne Π n as the set of all n !

Original: such permutation actions.
Revised: such permutation actions.

Original: Deﬁnition 3 (Orbits) .
Revised: Deﬁnition 3 (Orbits) .

Original: An orbit is the result of a group action Π n acting on elements of a group correspond to bijective transformations of the space that preserve some structure of the space.
Revised: An orbit is the result of a group action Π n acting on elements of a group correspond to bijective transformations of the space that preserve some structure of the space.

Original: The orbit of an element ω ∈ Ω is the set of equivalent elements under action Π n , i.e., Π n ( x ) =
Revised: The orbit of an element ω ∈ Ω is the set of equivalent elements under action Π n , i.e., Π n ( x ) =

Original: { π ( x ) | π ∈ Π n } .
Revised: { π ( x ) | π ∈ Π n } .

Original: Deﬁnition 4 ( G -equivariant and G -invariant functions) .
Revised: Deﬁnition 4 ( G -equivariant and G -invariant functions) .

Original: Let Σ be the set of all possible attributed graphs G .
Revised: Let Σ be the set of all possible attributed graphs G .

Original: More formally, Σ is the set of all tuples ( A , X ) with all-size adjacency tensors A and cor- responding node attributes X .
Revised: More formally, Σ is the set of all tuples ( A , X ) with all-size adjacency tensors A and corresponding node attributes X .

Original: A function g : Σ → R n ×· is G -equivariant w.r.t.
Revised: A function g : Σ → R n ×· is G -equivariant w.r.t.

Original: valid permutations of the nodes V , whenever any permutation action π ∈ Π n in the Σ symmetric space is associated with the same permutation action of the nodes in the R n ×· symmetric space.
Revised: valid permutations of the nodes V , whenever any permutation action π ∈ Π n in the Σ symmetric space is associated with the same permutation action of the nodes in the R n ×· symmetric space.

Original: A function g : Σ → R · is G -invariant whenever it is invariant to any permutation action π ∈ Π n in the Σ symmetric space.
Revised: A function g : Σ → R · is G -invariant whenever it is invariant to any permutation action π ∈ Π n in the Σ symmetric space.

Original: Deﬁnition 5 (Graph orbits & graph isomorphism) .
Revised: Deﬁnition 5 (Graph orbits & graph isomorphism) .

Original: Let G =
Revised: Let G =

Original: ( A , X ) be a graph with n nodes, and let Π n ( G )
Revised: ( A , X ) be a graph with n nodes, and let Π n ( G )

Original: = { ( A (cid:48) , X (cid:48) ) :
Revised: = { ( A (cid:48) , X (cid:48) ) :

Original: ( A (cid:48) , X (cid:48) )
Revised: ( A (cid:48) , X (cid:48) )

Original: =
Revised: =

Original: ( π ( A ) , π ( X )) , ∀ π ∈ Π n } be the set of all equivalent (isomorphic) graphs under the permutation action π .
Revised: ( π ( A ) , π ( X )) , ∀ π ∈ Π n } be the set of all equivalent (isomorphic) graphs under the permutation action π .

Original: Two graphs G 1 =
Revised: Two graphs G 1 =

Original: ( A 1 , X 1 ) and G 

Revised: ( A 1 , X 1 ) and G

Original: =
Revised: =

Original: ( A 2 , X 2 ) are said isomorphic iff Π n ( G 1 ) =
Revised: ( A 2 , X 2 ) are said isomorphic iff Π n ( G 1 ) =

Original: Π n ( G 2 ) .
Revised: Π n ( G 2 ) .

Original: Deﬁnition 6 (Node orbits & node isomorphism) .
Revised: Deﬁnition 6 (Node orbits & node isomorphism) .

Original: The equivalence classes of the vertices of a graph
Revised: The equivalence classes of the vertices of a graph

Original: G under the action of automorphisms are called vertex orbits.
Revised: G under the action of automorphisms are called vertex orbits.

Original: If two nodes are in the same node orbit, we say that they are isomorphic.
Revised: If two nodes are in the same node orbit, we say that they are isomorphic.

Original: In Figure 1, the lynx and the orca are isomorphic (they have the same node orbits).
Revised: In Figure 1, the lynx and the orca are isomorphic (they have the same node orbits).

Original: We now generalize Deﬁnition 6 to subsets of nodes S ∈ P (cid:63) ( V ) , where P (cid:63) ( V ) is the power set of V without the empty set.
Revised: We now generalize Deﬁnition 6 to subsets of nodes S ∈ P (cid:63) ( V ) , where P (cid:63) ( V ) is the power set of V without the empty set.

Original: Deﬁnition 7 (Vertex subset orbits and joint isomorphism) .
Revised: Deﬁnition 7 (Vertex subset orbits and joint isomorphism) .

Original: The equivalence classes of k -sized subsets of vertices S ∈ P (cid:63) ( V ) of a graph G under the action of automorphisms between the subsets are called vertex subset orbits, k ≥ 2 .
Revised: The equivalence classes of k -sized subsets of vertices S ∈ P (cid:63) ( V ) of a graph G under the action of automorphisms between the subsets are called vertex subset orbits, k ≥ 2 .

Original: If two proper subsets S 1 , S 2 ∈ P (cid:63) ( V ) \ V are in the same vertex subset orbit, we say they are jointly isomorphic.
Revised: If two proper subsets S 1 , S 2 ∈ P (cid:63) ( V ) \ V are in the same vertex subset orbit, we say they are jointly isomorphic.

Original: Next we deﬁne the relationship between structural representations and node embeddings.
Revised: Next we deﬁne the relationship between structural representations and node embeddings.

Original:  A U NIFYING T HEORETICAL F RAMEWORK OF N ODE E MBEDDINGS AND S TRUCTURAL R EPRESENTATIONS
Revised:  A U NIFYING T HEORETICAL F RAMEWORK OF N ODE E MBEDDINGS AND S TRUCTURAL R EPRESENTATIONS

Original: How are node embeddings and structural representations related?
Revised: How are node embeddings and structural representations related?

Original: This section starts with a familiar, albeit na ¨ ıve, view of the differences between node embeddings and structural representations, preparing the groundwork to later broadening and rectifying these into precise model-free mathematical statements using invariant theory.
Revised: This section starts with a familiar, albeit na¨ıve, view of the differences between node embeddings and structural representations, preparing the groundwork to later broadening and rectifying these into precise model-free mathematical statements using invariant theory.

Original: This broadening is needed since model-free node embeddings need not be related to node closeness in the graph (or to lower dimensional projections for that matter), as it is impossible to have a model-free deﬁnition of closeness .
Revised: This broadening is needed since model-free node embeddings need not be related to node closeness in the graph (or to lower dimensional projections for that matter), as it is impossible to have a model-free deﬁnition of closeness .

Original: A familiar interpretation of node embeddings: Node embeddings are often seen as a lowerdimensional projection of the rows and columns of the adjacency matrix A from R n to R d , d < n , that preserves relative positions of the nodes in a graph (Graham & Winkler, 1985; Linial et al., 1995); for instance, in Figure 1, the lynx and the coyote would have close node embeddings, while the node embeddings of the lynx and the orca would be signiﬁcantly different.
Revised: A familiar interpretation of node embeddings: Node embeddings are often seen as a lowerdimensional projection of the rows and columns of the adjacency matrix A from R n to R d , d < n , that preserves relative positions of the nodes in a graph (Graham & Winkler, 1985; Linial et al., 1995); for instance, in Figure 1, the lynx and the coyote would have close node embeddings, while the node embeddings of the lynx and the orca would be signiﬁcantly different.

Original: Node embeddings are often seen as encoding the fact that the lynx and the coyote are part of a tightly-knit community, while the lynx and orca belong to distinct communities.
Revised: Node embeddings are often seen as encoding the fact that the lynx and the coyote are part of a tightly-knit community, while the lynx and orca belong to distinct communities.

Original: The structural representation of a node, on the other hand, shows which nodes have similar roles (structural similarities) on a graph; for instance, the lynx and the orca in Figure 1 must have the same structural representation, while the lynx and the coyote likely have different structural representations.
Revised: The structural representation of a node, on the other hand, shows which nodes have similar roles (structural similarities) on a graph; for instance, the lynx and the orca in Figure 1 must have the same structural representation, while the lynx and the coyote likely have different structural representations.

Original: The lynx, like the orca, is a top predator in the food web while the coyote is not a top predator.
Revised: The lynx, like the orca, is a top predator in the food web while the coyote is not a top predator.

Original: The incompatibility of the familiar interpretation with the theory of structural graph representations: The above interpretation of node embeddings must be tied to a model that deﬁnes closeness .
Revised: The incompatibility of the familiar interpretation with the theory of structural graph representations: The above interpretation of node embeddings must be tied to a model that deﬁnes closeness .

Original: Structural graph representations are model-free.
Revised: Structural graph representations are model-free.

Original: Hence, we need a model-free deﬁnition of node embedding to connect it with structural representations.
Revised: Hence, we need a model-free deﬁnition of node embedding to connect it with structural representations.

Original: Unfortunately, one cannot deﬁne closeness without a model.
Revised: Unfortunately, one cannot deﬁne closeness without a model.

Original: Hence, in the remainder of this paper, we abandon this familiar interpretation in favor of a model-free deﬁnition.
Revised: Hence, in the remainder of this paper, we abandon this familiar interpretation in favor of a model-free deﬁnition.

Original: Roadmap:
Revised: Roadmap:

Original: In what follows, we restate some existing model-free deﬁnitions of structural graph representation and introduce some new ones.
Revised: In what follows, we restate some existing model-free deﬁnitions of structural graph representation and introduce some new ones.

Original: Then, we introduce a model-free deﬁnition of node embeddings.
Revised: Then, we introduce a model-free deﬁnition of node embeddings.

Original: We will retain the terminology node embedding for historical reasons, even though our node embedding need not be an embedding (a projection into lower dimensional space).
Revised: We will retain the terminology node embedding for historical reasons, even though our node embedding need not be an embedding (a projection into lower dimensional space).

Original: 3. O N S TRUCTURAL R EPRESENTATIONS
Revised: 3. O N S TRUCTURAL R EPRESENTATIONS

Original: In what follows we use the terms link and edge interchangeably.
Revised: In what follows we use the terms link and edge interchangeably.

Original: Proofs are left to the Appendix.
Revised: Proofs are left to the Appendix.

Original: Deﬁnition 8 (Structural node representations) .
Revised: Deﬁnition 8 (Structural node representations) .

Original: The structural representation of node v ∈ V in a graph G =
Revised: The structural representation of node v ∈ V in a graph G =

Original: ( A , X ) is the G -invariant representation Γ( v, A , X ) ,
Revised: ( A , X ) is the G -invariant representation Γ( v, A , X ) ,

Original: where Γ : V × Σ → R d , d ≥ 1 , such that ∀ u ∈ V , Γ( u, A , X ) =
Revised: where Γ : V × Σ → R d , d ≥ 1 , such that ∀ u ∈ V , Γ( u, A , X ) =

Original: Γ( π ( u ) , π ( A ) , π ( X )) for all permutation actions ∀ π ∈ Π n .
Revised: Γ( π ( u ) , π ( A ) , π ( X )) for all permutation actions ∀ π ∈ Π n .

Original: Moreover, for any two isomorphic nodes u, v ∈ V , Γ( u, A , X ) =
Revised: Moreover, for any two isomorphic nodes u, v ∈ V , Γ( u, A , X ) =

Original: Γ( v, A , X ) .
Revised: Γ( v, A , X ) .

Original: Deﬁnition 9 (Most-expressive structural node representations Γ (cid:63) ) .
Revised: Deﬁnition 9 (Most-expressive structural node representations Γ (cid:63) ) .

Original: A structural representation of a node v ∈ V , Γ (cid:63) ( v, A , X ) , is most-expressive iff, ∀ u ∈ V , there exists a bijective measurable map between Γ (cid:63) ( u, A , X ) and the orbit of node u in G =
Revised: A structural representation of a node v ∈ V , Γ (cid:63) ( v, A , X ) , is most-expressive iff, ∀ u ∈ V , there exists a bijective measurable map between Γ (cid:63) ( u, A , X ) and the orbit of node u in G =

Original: ( A , X ) (Deﬁnition 7).
Revised: ( A , X ) (Deﬁnition 7).

Original: Trivially, by Deﬁnitions 6 and 9, two graphs G 1 =
Revised: Trivially, by Deﬁnitions 6 and 9, two graphs G 1 =

Original: ( A 1 , X 1 ) and G 

Revised: ( A 1 , X 1 ) and G

Original: =
Revised: =

Original: ( A 2 , X 2 ) are isomorphic (Deﬁnition 5) iff the most-expressive structural node representations (Γ (cid:63) ( u, A 1 , X 1 ))
Revised: ( A 2 , X 2 ) are isomorphic (Deﬁnition 5) iff the most-expressive structural node representations (Γ (cid:63) ( u, A 1 , X 1 ))

Original: u ∈ V and (Γ (cid:63) ( v, A 2 , X 2 )) v ∈ V are the same up to a valid permutation π ∈ Π n of the nodes.
Revised: u ∈ V and (Γ (cid:63) ( v, A 2 , X 2 )) v ∈ V are the same up to a valid permutation π ∈ Π n of the nodes.

Original: We now describe the relationship between structural node representations and node isomorphism.
Revised: We now describe the relationship between structural node representations and node isomorphism.

Original: Lemma 1.
Revised: Lemma 1.

Original: Two nodes v, u ∈ V , have the same most-expressive structural representations Γ (cid:63) ( v, A , X ) =
Revised: Two nodes v, u ∈ V , have the same most-expressive structural representations Γ (cid:63) ( v, A , X ) =

Original: Γ (cid:63) ( u, A , X ) iff u and v are isomorphic nodes in G =
Revised: Γ (cid:63) ( u, A , X ) iff u and v are isomorphic nodes in G =

Original: ( A , X ) .
Revised: ( A , X ) .

Original: Having described representation of nodes, we now generalize these representations to subsets of V .
Revised: Having described representation of nodes, we now generalize these representations to subsets of V .

Original: Deﬁnition 10 (Joint structural representation Γ ) .
Revised: Deﬁnition 10 (Joint structural representation Γ ) .

Original: A joint structural representation Γ : P (cid:63) ( V ) × Σ → R d , d ≥ 1 , where P (cid:63) ( V ) is the power set of V excluding the empty set.
Revised: A joint structural representation Γ : P (cid:63) ( V ) × Σ → R d , d ≥ 1 , where P (cid:63) ( V ) is the power set of V excluding the empty set.

Original: Furthermore, Γ is Ginvariant over all node subsets, i.e., ∀ S ∈ P (cid:63) ( V ) and ∀ ( A , X ) ∈ Σ , it must be that Γ( (cid:126)S, A , X ) =
Revised: Furthermore, Γ is Ginvariant over all node subsets, i.e., ∀ S ∈ P (cid:63) ( V ) and ∀ ( A , X ) ∈ Σ , it must be that Γ( (cid:126)S, A , X ) =

Original: Γ( π ( (cid:126)S ) , π ( A ) , π ( X )) for all permutation actions ∀ π ∈ Π n .
Revised: Γ( π ( (cid:126)S ) , π ( A ) , π ( X )) for all permutation actions ∀ π ∈ Π n .

Original: Moreover, for any two isomorphic subsets S, S (cid:48) ∈ P (cid:63) ( V ) , Γ( (cid:126)S, A , X ) =
Revised: Moreover, for any two isomorphic subsets S, S (cid:48) ∈ P (cid:63) ( V ) , Γ( (cid:126)S, A , X ) =

Original: Γ( (cid:126)S (cid:48) , A , X ) .
Revised: Γ( (cid:126)S (cid:48) , A , X ) .

Original: We now mirror Deﬁnition 9 in our generalization of Γ :
Revised: We now mirror Deﬁnition 9 in our generalization of Γ :

Original: Deﬁnition 11 (Most-expressive joint structural representations Γ (cid:63) ) .
Revised: Deﬁnition 11 (Most-expressive joint structural representations Γ (cid:63) ) .

Original: A structural representation Γ (cid:63) ( (cid:126)S, A , X ) of a non-empty subset S ∈ P (cid:63) ( V ) , of a graph ( A , X ) ∈ Σ , is most-expressive iff, there exists a bijective measurable map between Γ (cid:63) ( (cid:126)U, A , X ) and the orbit of U in G (Deﬁnition 7), ∀ U ∈ P (cid:63) ( V ) and ∀ ( A , X ) ∈ Σ .
Revised: A structural representation Γ (cid:63) ( (cid:126)S, A , X ) of a non-empty subset S ∈ P (cid:63) ( V ) , of a graph ( A , X ) ∈ Σ , is most-expressive iff, there exists a bijective measurable map between Γ (cid:63) ( (cid:126)U, A , X ) and the orbit of U in G (Deﬁnition 7), ∀ U ∈ P (cid:63) ( V ) and ∀ ( A , X ) ∈ Σ .

Original: Note, however, the failure to represent the link ( lynx , coyote ) in Figure 1 using the mostexpressive node representations of the lynx and the coyote .
Revised: Note, however, the failure to represent the link ( lynx , coyote ) in Figure 1 using the mostexpressive node representations of the lynx and the coyote .

Original: A link needs to be represented by a joint representation of two nodes.
Revised: A link needs to be represented by a joint representation of two nodes.

Original: For instance, we can easily verify from Deﬁnition 11 that Γ (cid:63) (( lynx,coyote ) , A , X ) (cid:54) =
Revised: For instance, we can easily verify from Deﬁnition 11 that Γ (cid:63) (( lynx,coyote ) , A , X ) (cid:54) =

Original: Γ (cid:63) (( orca,coyote ) , A , X ) , even though Γ (cid:63) ( lynx , A , X ) =
Revised: Γ (cid:63) (( orca,coyote ) , A , X ) , even though Γ (cid:63) ( lynx , A , X ) =

Original: Γ (cid:63) ( orca , A , X ) .
Revised: Γ (cid:63) ( orca , A , X ) .

Original: Next we show that joint prediction tasks only require joint structural representations.
Revised: Next we show that joint prediction tasks only require joint structural representations.

Original: But ﬁrst we need to show that any causal model deﬁned through axiomatic counterfactuals (Galles & Pearl, 1998) can be equivalently deﬁned through noise outsourcing (Austin, 2008), a straightforward result that we were unable to ﬁnd in the literature.
Revised: But ﬁrst we need to show that any causal model deﬁned through axiomatic counterfactuals (Galles & Pearl, 1998) can be equivalently deﬁned through noise outsourcing (Austin, 2008), a straightforward result that we were unable to ﬁnd in the literature.

Original: Lemma 2 (Causal modeling through noise outsourcing) .
Revised: Lemma 2 (Causal modeling through noise outsourcing) .

Original: Deﬁnition 1 of Galles & Pearl (1998) gives a causal model as a triplet
Revised: Deﬁnition 1 of Galles & Pearl (1998) gives a causal model as a triplet

Original: [Equation]
Revised: [Equation]

Original: where U is a set of exogenous variables, V [Equation] is a set of endogenous variables, and F is a set of functions, such that in the causal model, v (cid:48)
Revised: where U is a set of exogenous variables, V (cid:48) is a set of endogenous variables, and F is a set of functions, such that in the causal model, v (cid:48)

Original: i =
Revised: i =

Original: f ( (cid:126)pa i , u ) is the realization of random variable V (cid:48)
Revised: f ( (cid:126)pa i , u ) is the realization of random variable V (cid:48)

Original: i ∈ V (cid:48) and a sequence of random variables (cid:126)PA i with PA i ⊆
Revised: i ∈ V (cid:48) and a sequence of random variables (cid:126)PA i with PA i ⊆

Original: V \ V (cid:48)
Revised: V \ V (cid:48)

Original: i as the endogenous variable parents of variable V (cid:48)
Revised: i as the endogenous variable parents of variable V (cid:48)

Original: i as given by a directed acyclic graph.
Revised: i as given by a directed acyclic graph.

Original: Then, there exists a pure random noise (cid:15) and a set of (measurable) functions { g u } u ∈ U such that for V i ∈ V (cid:48) , V (cid:48)
Revised: Then, there exists a pure random noise (cid:15) and a set of (measurable) functions { g u } u ∈ U such that for V i ∈ V (cid:48) , V (cid:48)

Original: i can be equivalently deﬁned as v (cid:48)
Revised: i can be equivalently deﬁned as v (cid:48)

Original: i a.s. =
Revised: i a.s. =

Original: f ( (cid:126)pa i , g u ( (cid:15) u )) , where (cid:15) u has joint distribution ( (cid:15) u ) ∀ u
Revised: f ( (cid:126)pa i , g u ( (cid:15) u )) , where (cid:15) u has joint distribution ( (cid:15) u ) ∀ u

Original: ∈ U a.s. =
Revised: ∈ U a.s. =

Original: g (cid:48) ( (cid:15) ) for some Borel measurable function g (cid:48) and a random variable (cid:15) ∼ Uniform (0 , 1) .
Revised: g (cid:48) ( (cid:15) ) for some Borel measurable function g (cid:48) and a random variable (cid:15) ∼ Uniform (0 , 1) .

Original: The latter deﬁnes M via noise outsourcing (Austin, 2008).
Revised: The latter deﬁnes M via noise outsourcing (Austin, 2008).

Original: The proof of Lemma 2 is given in the Appendix.
Revised: The proof of Lemma 2 is given in the Appendix.

Original: Lemma 2 deﬁnes the causal model entirely via endogenous variables, deterministic functions, and a pure random noise random variable.
Revised: Lemma 2 deﬁnes the causal model entirely via endogenous variables, deterministic functions, and a pure random noise random variable.

Original: We are now ready for our theorem showing that joint prediction tasks only require (most-expressive) joint structural representations.
Revised: We are now ready for our theorem showing that joint prediction tasks only require (most-expressive) joint structural representations.

Original: Theorem 1.
Revised: Theorem 1.

Original: Let S ⊆ P (cid:63) ( V ) be a set of non-empty subsets of the vertices V .
Revised: Let S ⊆ P (cid:63) ( V ) be a set of non-empty subsets of the vertices V .

Original: Let Y ( S , A , X ) =
Revised: Let Y ( S , A , X ) =

Original: ( Y ( (cid:126) S, A , X ))
Revised: ( Y ( (cid:126)S, A , X ))

Original: S ∈S be a sequence of random variables deﬁned over the sets S ∈ S of a graph
Revised: S ∈S be a sequence of random variables deﬁned over the sets S ∈ S of a graph

Original: G =
Revised: G =

Original: ( A , X ) , that are invariant to the ordering of (cid:126)S, S ∈ S , such that Y ( (cid:126)S 1 , A , X ) d
Revised: ( A , X ) , that are invariant to the ordering of (cid:126)S, S ∈ S , such that Y ( (cid:126)S 1 , A , X ) d

Original: = Y ( (cid:126)S 2 , A , X ) for any two jointly isomorphic subsets S 1 , S 2 ∈ S (Deﬁnition 7), where d = means equality in their marginal distributions.
Revised: = Y ( (cid:126)S 2 , A , X ) for any two jointly isomorphic subsets S 1 , S 2 ∈ S (Deﬁnition 7), where d = means equality in their marginal distributions.

Original: Then, there exists a measurable function ϕ such that, Y ( S , A , X ) a.s. =
Revised: Then, there exists a measurable function ϕ such that, Y ( S , A , X ) a.s. =

Original: ( ϕ (Γ (cid:63) ( (cid:126)S, A , X ) , (cid:15) S ))
Revised: ( ϕ (Γ (cid:63) ( (cid:126)S, A , X ) , (cid:15) S ))

Original: S ∈S , where (cid:15) S is the random noise that deﬁnes the exogenous variables of
Revised: S ∈S , where (cid:15) S is the random noise that deﬁnes the exogenous variables of

Original: Lemma 2, with joint distribution p (( (cid:15) S (cid:48) )
Revised: Lemma 2, with joint distribution p (( (cid:15) S (cid:48) )

Original: ∀ S (cid:48) ∈S ) independent of A and X .
Revised: ∀ S (cid:48) ∈S ) independent of A and X .

Original: Theorem 1 extends Theorem 12 of Bloem-Reddy & Teh (2019) in multiple ways: (a) to all subsets of nodes, S ∈ P (cid:63) ( V ) , (b) to include causal language, and, most importantly, (c) to showing that any prediction task that can be deﬁned over S , requires only a most-expressive joint structural representation over S .
Revised: Theorem 1 extends Theorem 12 of Bloem-Reddy & Teh (2019) in multiple ways: (a) to all subsets of nodes, S ∈ P (cid:63) ( V ) , (b) to include causal language, and, most importantly, (c) to showing that any prediction task that can be deﬁned over S , requires only a most-expressive joint structural representation over S .

Original: For instance, any task with | S | = 2 predicting a missing link ( u, v ) on a graph G =
Revised: For instance, any task with | S | = 2 predicting a missing link ( u, v ) on a graph G =

Original: ( A , X ) , requires only the most-expressive structural representation Γ (cid:63) (( u, v ) , A , X ) .
Revised: ( A , X ) , requires only the most-expressive structural representation Γ (cid:63) (( u, v ) , A , X ) .

Original: Note that, in order to predict directed edges, we must use Γ (cid:63) (( u, v ) , A , X ) to also predict the edge’s direction: → , ← , ↔ , but a detailed procedure showing how to predict directed edges is relegated to a future journal version of this paper.
Revised: Note that, in order to predict directed edges, we must use Γ (cid:63) (( u, v ) , A , X ) to also predict the edge’s direction: → , ← , ↔ , but a detailed procedure showing how to predict directed edges is relegated to a future journal version of this paper.

Original: Theorem 1 also includes node tasks for | S | = 1 , hyperedge tasks for 2 < | S | < n , and graph-wide tasks for S = V .
Revised: Theorem 1 also includes node tasks for | S | = 1 , hyperedge tasks for 2 < | S | < n , and graph-wide tasks for S = V .

Original: Remark 1 (GNNs and link prediction) .
Revised: Remark 1 (GNNs and link prediction) .

Original: Even though structural node representations of GNNs are not able to predict edges, GNNs are often still optimized to predict edges (e.g., (Hamilton et al., 2017a;
Revised: Even though structural node representations of GNNs are not able to predict edges, GNNs are often still optimized to predict edges (e.g., (Hamilton et al., 2017a;

Original: Xu et al., 2018)) in transfer learning tasks.
Revised: Xu et al., 2018)) in transfer learning tasks.

Original: This optimization objective guarantees that any small topological differences between two nearly-isomorphic nodes without an edge will be ampliﬁed, while differences between nodes with an edge will be minimized.
Revised: This optimization objective guarantees that any small topological differences between two nearly-isomorphic nodes without an edge will be ampliﬁed, while differences between nodes with an edge will be minimized.

Original: Hence, the topological differences in a close-knit community will be minimized in the representation.
Revised: Hence, the topological differences in a close-knit community will be minimized in the representation.

Original: This procedure is an interesting way to introduce homophily in structural representations and should work well for node classiﬁcation tasks in homophilic networks (where node classes tend to be clustered).
Revised: This procedure is an interesting way to introduce homophily in structural representations and should work well for node classiﬁcation tasks in homophilic networks (where node classes tend to be clustered).

Original: We now turn our attention to node embeddings and their relationship with joint representations.
Revised: We now turn our attention to node embeddings and their relationship with joint representations.

Original: 3. O N N ODE E MBEDDINGS
Revised: 3. O N N ODE E MBEDDINGS

Original: Deﬁnition 12 (Node Embeddings) .
Revised: Deﬁnition 12 (Node Embeddings) .

Original: The node embeddings of a graph G =
Revised: The node embeddings of a graph G =

Original: ( A , X ) are deﬁned as joint samples of random variables ( Z i ) i ∈ V | A , X ∼ p ( ·| A , X ) ,
Revised: ( A , X ) are deﬁned as joint samples of random variables ( Z i ) i ∈ V | A , X ∼ p ( ·| A , X ) ,

Original: Z i ∈ R d , d ≥ 1 , where p ( ·| A , X ) is a G -equivariant probability distribution on A and X , that is, π ( p ( ·| A , X ))
Revised: Z i ∈ R d , d ≥ 1 , where p ( ·| A , X ) is a G -equivariant probability distribution on A and X , that is, π ( p ( ·| A , X ))

Original: =
Revised: =

Original: p ( ·| π ( A ) , π ( X )) for any permutation π ∈ Π n .
Revised: p ( ·| π ( A ) , π ( X )) for any permutation π ∈ Π n .

Original: Essentially, Deﬁnition 12 says that the probability distribution p ( Z | A , X ) of a node embedding Z must be G -equivariant on A and X .
Revised: Essentially, Deﬁnition 12 says that the probability distribution p ( Z | A , X ) of a node embedding Z must be G -equivariant on A and X .

Original: This is the only property we require to deﬁne a node embedding.
Revised: This is the only property we require to deﬁne a node embedding.

Original: Next, we show that the node embeddings given by Deﬁnition 12 cover a wide range of embedding methods in the literature.
Revised: Next, we show that the node embeddings given by Deﬁnition 12 cover a wide range of embedding methods in the literature.

Original: Corollary 1.
Revised: Corollary 1.

Original: The node embeddings in Deﬁnition 12 encompass embeddings given by matrix and tensor factorization methods —such as Singular Value Decomposition (SVD), Non-negative Matrix
Revised: The node embeddings in Deﬁnition 12 encompass embeddings given by matrix and tensor factorization methods —such as Singular Value Decomposition (SVD), Non-negative Matrix

Original: Factorization (NMF), implicit matrix factorization (a.k.a. word2vec)–, latent embeddings given by
Revised: Factorization (NMF), implicit matrix factorization (a.k.a. word2vec)–, latent embeddings given by

Original: Bayesian graph models —such as Probabilistic Matrix Factorizations (PMFs) and variants—, variational autoencoder methods and graph neural networks that use random lighthouses to extract node embedddings.
Revised: Bayesian graph models —such as Probabilistic Matrix Factorizations (PMFs) and variants—, variational autoencoder methods and graph neural networks that use random lighthouses to extract node embedddings.

Original: The proof of Corollary 1 is in the Appendix, along with the references of each of the methods mentioned in the corollary.
Revised: The proof of Corollary 1 is in the Appendix, along with the references of each of the methods mentioned in the corollary.

Original: The output of some of the methods described in Corollary 1 is deterministic, and for those, the probability density p ( Z | A , X ) is a Dirac delta.
Revised: The output of some of the methods described in Corollary 1 is deterministic, and for those, the probability density p ( Z | A , X ) is a Dirac delta.

Original: In practice, however, even deterministic methods use algorithms whose outputs depend on randomized initial conditions, which will also satisfy Corollary 1.
Revised: In practice, however, even deterministic methods use algorithms whose outputs depend on randomized initial conditions, which will also satisfy Corollary 1.

Original: We now show that permutation equivariance implies two isomorphic nodes (or two subsets of nodes) must have the same marginal distributions over Z :
Revised: We now show that permutation equivariance implies two isomorphic nodes (or two subsets of nodes) must have the same marginal distributions over Z :

Original: Lemma 3.
Revised: Lemma 3.

Original: The permutation equivariance of p in Deﬁnition 12 implies that, if two proper subsets of nodes S 1 , S 2 ∈ P (cid:63) ( V ) \ V are isomorphic, then their marginal node embedding distributions must be the same up to a permutation, i.e., p (( Z i ) i ∈
Revised: The permutation equivariance of p in Deﬁnition 12 implies that, if two proper subsets of nodes S 1 , S 2 ∈ P (cid:63) ( V ) \ V are isomorphic, then their marginal node embedding distributions must be the same up to a permutation, i.e., p (( Z i ) i ∈

Original: S 1 | A , X ) =
Revised: S 1 | A , X ) =

Original: π ( p (( Z j ) j ∈ S 2 | A , X )) for some appropriate permutation π ∈ Π n .
Revised: π ( p (( Z j ) j ∈ S 2 | A , X )) for some appropriate permutation π ∈ Π n .

Original: Hence, the critical difference between the structural node representation vector (Γ( v, A , X )) v ∈ V in Deﬁnition 8 and node embeddings Z in Deﬁnition 12, is that the vector (Γ( v, A , X )) v ∈ V must be G -equivariant while Z need not be —even though Z ’s distribution must be G -equivariant.
Revised: Hence, the critical difference between the structural node representation vector (Γ( v, A , X )) v ∈ V in Deﬁnition 8 and node embeddings Z in Deﬁnition 12, is that the vector (Γ( v, A , X )) v ∈ V must be G -equivariant while Z need not be —even though Z ’s distribution must be G -equivariant.

Original: This seemly trivial difference has tremendous consequences, which we explore in the reminder of this section.
Revised: This seemly trivial difference has tremendous consequences, which we explore in the reminder of this section.

Original: Next, we show that node embeddings Z cannot have any extra information about G that is not already contained in a most-expressive structural representation Γ (cid:63) .
Revised: Next, we show that node embeddings Z cannot have any extra information about G that is not already contained in a most-expressive structural representation Γ (cid:63) .

Original: Theorem 2 (The statistical equivalence between node embeddings and structural representations) .
Revised: Theorem 2 (The statistical equivalence between node embeddings and structural representations) .

Original: Let Y ( S , A , X ) =
Revised: Let Y ( S , A , X ) =

Original: ( Y ( (cid:126)S, A , X ))
Revised: [Equation] Y [Equation] , A , X ))

Original: S ∈S be as in Theorem 1.
Revised: S ∈S be as in Theorem 1.

Original: Consider a graph G =
Revised: Consider a graph G =

Original: ( A , X ) ∈ Σ .
Revised: ( A , X ) ∈ Σ .

Original: Let Γ (cid:63) ( (cid:126)S, A , X ) be a most-expressive structural representation of nodes S ∈ P (cid:63) ( V ) in G .
Revised: Let Γ (cid:63) ( (cid:126)S, A , X ) be a most-expressive structural representation of nodes S ∈ P (cid:63) ( V ) in G .

Original: Then,
Revised: Then,

Original: [Equation]
Revised: [Equation]

Original: for any node embedding matrix Z that satisﬁes Deﬁnition 12, where A ⊥⊥ B C means A is independent of C given B .
Revised: for any node embedding matrix Z that satisﬁes Deﬁnition 12, where A ⊥⊥ B C means A is independent of C given B .

Original: Finally, ∀ ( A , X ) ∈ Σ , there exists a most-expressive node embedding Z (cid:63) | A , X such that,
Revised: Finally, ∀ ( A , X ) ∈ Σ , there exists a most-expressive node embedding Z (cid:63) | A , X such that,

Original: [Equation]
Revised: [Equation]

Original: for some appropriate collection of functions { f ( k ) ( · ) } k = [Equation] ...,n .
Revised: for some appropriate collection of functions { f ( k ) ( · ) } k =1 ,...,n .

Original: The proof of Theorem 2 is given in the Appendix.
Revised: The proof of Theorem 2 is given in the Appendix.

Original: It uses structural causal models (Galles &
Revised: It uses structural causal models (Galles &

Original: Pearl, 1998; Halpern, 2000) in their measure theory form via the noise outsourcing (transfer) theorem (Austin, 2008; Kallenberg, 2006) to redeﬁne Z a.s. =
Revised: Pearl, 1998; Halpern, 2000) in their measure theory form via the noise outsourcing (transfer) theorem (Austin, 2008; Kallenberg, 2006) to redeﬁne Z a.s. =

Original: ψ ( A , X , (cid:15) ) , where ψ is a G -equivariant deterministic function of A , X , and a pure random noise (cid:15) that is independent of A and X .
Revised: ψ ( A , X , (cid:15) ) , where ψ is a G -equivariant deterministic function of A , X , and a pure random noise (cid:15) that is independent of A and X .

Original: The mostexpressive embedding Z (cid:63) | A , X extends the insight used to make GNNs more expressive in Murphy et al.
Revised: The mostexpressive embedding Z (cid:63) | A , X extends the insight used to make GNNs more expressive in Murphy et al.

Original: to a more general procedure.
Revised: to a more general procedure.

Original: Theorem 2 implies that, for any graph prediction task, node embeddings carry no information beyond that of structural representations.
Revised: Theorem 2 implies that, for any graph prediction task, node embeddings carry no information beyond that of structural representations.

Original: A less attentive reader may think this creates an apparent paradox, since one cannot predict a property Y (( lynx,coyote ) , A food web , X food web ) in Figure 1 from structural node embeddings, since Γ( lynx , A , X )
Revised: A less attentive reader may think this creates an apparent paradox, since one cannot predict a property Y (( lynx,coyote ) , A food web , X food web ) in Figure 1 from structural node embeddings, since Γ( lynx , A , X )

Original: =
Revised: =

Original: Γ( orca , A , X ) .
Revised: Γ( orca , A , X ) .

Original: The resolution of the paradox is to note that Theorem 2 describes the prediction of a link through a pairwise structural representation Γ(( lynx,coyote ) , A food web , X food web ) , and we may not be able to do the same task with structural node representations alone.
Revised: The resolution of the paradox is to note that Theorem 2 describes the prediction of a link through a pairwise structural representation Γ(( lynx,coyote ) , A food web , X food web ) , and we may not be able to do the same task with structural node representations alone.

Original: An interesting question for future work is how well can we learn distributions (representations) from (node embeddings) samples, extending (Kamath et al., 2015) to graph representations.
Revised: An interesting question for future work is how well can we learn distributions (representations) from (node embeddings) samples, extending (Kamath et al., 2015) to graph representations.

Original: Other equally important consequences of Theorem 2 are: (a) any sampling approach obtaining node embeddings Z is valid as long as the distribution is G -equivariant
Revised: Other equally important consequences of Theorem 2 are: (a) any sampling approach obtaining node embeddings Z is valid as long as the distribution is G -equivariant

Original: (Deﬁnition 12), noting that isomorphic nodes must have the same marginal distributions (per Lemma 3).
Revised: (Deﬁnition 12), noting that isomorphic nodes must have the same marginal distributions (per Lemma 3).

Original: (b) Interestingly, convex optimization methods for matrix factorization can be seen as variance-reduction techniques with no intrinsic value beyond reducing variance.
Revised: (b) Interestingly, convex optimization methods for matrix factorization can be seen as variance-reduction techniques with no intrinsic value beyond reducing variance.

Original: (c) Methods that give unique node embeddings —if the embedding of any two isomorphic nodes are different— are provably incorrect when used to predict graph relationships since they are permutation-sensitive.
Revised: (c) Methods that give unique node embeddings —if the embedding of any two isomorphic nodes are different— are provably incorrect when used to predict graph relationships since they are permutation-sensitive.

Original: Remark 2 (Some GNN methods give node embeddings not structural representations) .
Revised: Remark 2 (Some GNN methods give node embeddings not structural representations) .

Original: The random edges added by GraphSAGE (Hamilton et al., 2017a) and GIN (Xu et al., 2018) random walks make these methods node embeddings rather than structural node representations, according to
Revised: The random edges added by GraphSAGE (Hamilton et al., 2017a) and GIN (Xu et al., 2018) random walks make these methods node embeddings rather than structural node representations, according to

Original: Deﬁnition 12.
Revised: Deﬁnition 12.

Original: To transform them back to structural node representations, one must average over all such random walks.
Revised: To transform them back to structural node representations, one must average over all such random walks.

Original: The following corollaries describe other consequences of Theorem 2:
Revised: The following corollaries describe other consequences of Theorem 2:

Original: Corollary 2.
Revised: Corollary 2.

Original: The link prediction task between any two nodes u, v ∈ V depends only on the mostexpressive tuple representation Γ (cid:63) (( u, v ) , A , X ) .
Revised: The link prediction task between any two nodes u, v ∈ V depends only on the mostexpressive tuple representation Γ (cid:63) (( u, v ) , A , X ) .

Original: Moreover, Γ (cid:63) (( u, v ) , A , X ) always exists for any graph ( A , X ) and nodes ( u, v ) .
Revised: Moreover, Γ (cid:63) (( u, v ) , A , X ) always exists for any graph ( A , X ) and nodes ( u, v ) .

Original: Finally, given most-expressive node embeddings Z (cid:63) , there exists a function f such that Γ (cid:63) (( u, v ) , A , X ) =
Revised: Finally, given most-expressive node embeddings Z (cid:63) , there exists a function f such that Γ (cid:63) (( u, v ) , A , X ) =

Original: E Z (cid:63) [ f ( Z (cid:63)u , Z (cid:63)v )] , ∀ u, v .
Revised: E Z (cid:63) [ f ( Z (cid:63)u , Z (cid:63)v )] , ∀ u, v .

Original: A generalization of Corollary 2 is also possible, where Theorem 2 is used to allow us to create joint representations from simpler node embedding sampling methods.
Revised: A generalization of Corollary 2 is also possible, where Theorem 2 is used to allow us to create joint representations from simpler node embedding sampling methods.

Original: Corollary 3.
Revised: Corollary 3.

Original: Sample Z according to Deﬁnition 12.
Revised: Sample Z according to Deﬁnition 12.

Original: Then, we can learn a k -node structural representation of a subset of k nodes S ∈ P (cid:63) ( V ) , | S |
Revised: Then, we can learn a k -node structural representation of a subset of k nodes S ∈ P (cid:63) ( V ) , | S |

Original: =
Revised: =

Original: k , simply by learning a function f ( k ) whose average Γ( (cid:126) S, A , X ) =
Revised: k , simply by learning a function f ( k ) whose average Γ( (cid:126)S, A , X ) =

Original: E [ f ( k ) (( Z v ) v ∈ S )] can be used to predict Y ( (cid:126) S, A , X ) .
Revised: E [ f ( k ) (( Z v ) v ∈ S )] can be used to predict Y ( (cid:126)S, A , X ) .

Original: The proof of Corollary 3 is in the Appendix.
Revised: The proof of Corollary 3 is in the Appendix.

Original: Finally, we show that the concepts of transductive and inductive learning are unrelated to the notions of node embeddings and structural representations.
Revised: Finally, we show that the concepts of transductive and inductive learning are unrelated to the notions of node embeddings and structural representations.

Original: Corollary 4.
Revised: Corollary 4.

Original: Transductive and inductive learning are unrelated to the concepts of node embeddings and structural representations.
Revised: Transductive and inductive learning are unrelated to the concepts of node embeddings and structural representations.

Original: Corollary 4 clears a confusion that, we believe, arises because traditional applications of node embeddings use a single Monte Carlo sample of Z | A , X to produce a structural representation (e.g., (Mikolov et al., 2013)).
Revised: Corollary 4 clears a confusion that, we believe, arises because traditional applications of node embeddings use a single Monte Carlo sample of Z | A , X to produce a structural representation (e.g., (Mikolov et al., 2013)).

Original: Inherently, a classiﬁer learned with such a poor structural representation may fail to generalize over the test data, and will be deemed transductive .
Revised: Inherently, a classiﬁer learned with such a poor structural representation may fail to generalize over the test data, and will be deemed transductive .

Original: Corollary 5.
Revised: Corollary 5.

Original: Merging a node embeddings sampling scheme with GNNs can increase the structural representation power of GNNs.
Revised: Merging a node embeddings sampling scheme with GNNs can increase the structural representation power of GNNs.

Original: Corollary 5 is a direct consequence of Theorem 2, with Murphy et al.
Revised: Corollary 5 is a direct consequence of Theorem 2, with Murphy et al.

Original: (2019) showing RP-GNN as a concrete method to do so.
Revised: (2019) showing RP-GNN as a concrete method to do so.

Original:  R ESULTS
Revised:  R ESULTS

Original: This section focuses on applying the lessons learned in Section 3 in four tasks, divided into two common goals.
Revised: This section focuses on applying the lessons learned in Section 3 in four tasks, divided into two common goals.

Original: The goal of the ﬁrst three tasks is to show that, as described in Theorem 2, node embeddings can be used to create expressive structural embeddings of nodes, tuples, and triads.
Revised: The goal of the ﬁrst three tasks is to show that, as described in Theorem 2, node em- beddings can be used to create expressive structural embeddings of nodes, tuples, and triads.

Original: These representations are then subsequently used to make predictions on downstream tasks with varied node set sizes.
Revised: These representations are then subsequently used to make predictions on downstream tasks with varied node set sizes.

Original: The tasks also showcase the added value of using multiple node embeddings (Monte
Revised: The tasks also showcase the added value of using multiple node embeddings (Monte

Original: Carlo) samples to estimate structural representations, both during training and testing.
Revised: Carlo) samples to estimate structural representations, both during training and testing.

Original: Moreover, showcasing Theorem 1 and the inability of node representations to capture joint structural representations, these tasks show that structural node representations are useless in prediction tasks over more than one node, such as links and triads.
Revised: Moreover, showcasing Theorem 1 and the inability of node representations to capture joint structural representations, these tasks show that structural node representations are useless in prediction tasks over more than one node, such as links and triads.

Original: The goal of fourth task is to showcase how multiple Monte Carlo samples of node embeddings are required to observe the fundamental relationship between structural representations and node embeddings predicted by Theorem 2.
Revised: The goal of fourth task is to showcase how multiple Monte Carlo samples of node embeddings are required to observe the fundamental relationship between structural representations and node embeddings predicted by Theorem 2.

Original: An important note: Our proposed theoretical framework is not limited to the way we generate node embeddings.
Revised: An important note: Our proposed theoretical framework is not limited to the way we generate node embeddings.

Original: For example, our theoretical framework can use SVD in an inductive setting, where we train a classiﬁer in one graph and test in a different graph, which was thought not possible previously with SVD.
Revised: For example, our theoretical framework can use SVD in an inductive setting, where we train a classiﬁer in one graph and test in a different graph, which was thought not possible previously with SVD.

Original: SVD with our theoretical framework is denoted MC-SVD, to emphasize the importance of Monte Carlo sampling in building better structural representations.
Revised: SVD with our theoretical framework is denoted MC-SVD, to emphasize the importance of Monte Carlo sampling in building better structural representations.

Original: Alternatively, more expressive node embeddings can be obtained, as we show next.

Original: 4. C OLLIDING G RAPH N EURAL N ETWORKS (CGNN S )

Original: In this section we propose a new variational auto-encoder procedure to obtain node embeddings using neural networks, denoted Colliding Graph Neural Networks (CGNNs).

Original: Our sole reason to propose a new auto-encoder method is because we want to test the expressiveness of node embedding auto-encoders —and, unfortunately, existing auto-encoders, such as Grover et al.

Original: (2019), do not properly account for the dependencies introduced by the colliding variables in the graphical model.

Original: In our experiments, shown later, we aggregate multiple node embedding sampled from CGNN to obtain structural representations of the corresponding nodes and node sets. Node Embedding Auto-encoder. In CGNN’s, we adopt a latent variable approach to learn node embeddings. Corresponding to each evidence feature vector X
Revised: Alternatively, more expressive node embeddings can be obtained using Colliding Graph Neural Networks (CGNN), as we show in the Appendix (Section 10 and Section 11)

Original: i, · ∈ R k ∀ i ∈ V , we introduce a latent variable Z i, · ∈ R k .

Original: In addition, our graphical model also consists of observed variables

Original: A i,j, · ∈ R d ∀ i, j ∈ V .

Original: These are related through the joint distribution p

Original: ( A , X | Z )

Original: = (cid:81) i,j ∈ V × V p

Original: ( A i,j, · | Z i, · , Z j, · ) (cid:81)

Original: h ∈ V p

Original: ( X h, · | Z h, · ) , which is summarized by the Bayesian network in Figure 5 in the Appendix.

Original: Note that A i,j, · is a collider, since it is observed and inﬂuenced by two hidden variables, Z i, · , Z j, · .

Original: A neural network is used to learn the joint probability via MCMC, in an unsupervised fashion.

Original: The model learns the parameters of the MCMC transition kernel via an unrolled Gibbs Sampler, a templated recurrent model (an MLP with shared weights across Gibbs sampling steps), partially inspired by Fan & Huang (2017).

Original: The unrolled Gibbs Sampler, starts with a normal distribution of the latent variables Z (0) i, · , ∀ i ∈ V, with each Z (0) i, · ∼ N ( 0 , I ) independently, where I is the identity matrix.

Original: Subsequently at time steps t = 1 , 2 , . .

Original: . , in accordance with the graphical model, each variable Z ( t ) i, · is sequentially sampled from its true distribution conditioned on all observed edges of its corresponding node i , in addition to the most-up-to-date latent variables

Original: Z ’s associated with its immediate neighbors.

Original: The reparametrization trick (Kingma & Welling, 2013) allows us to backpropogate through the unrolled

Original: Gibbs Sampler.

Original: Algorithm 1 in the Appendix details our method.

Original: Consequently, this procedure has an effect on the run-time of this technique, which we alleviate by performing Parallel Gibbs

Original: Sampling by constructing parallel splashes (Gonzalez et al., 2011).

Original: Our unsupervised objective is reconstructing the noisy adjacency matrix.

Original: 4. Q UANTITATIVE R ESULTS
Revised: 4. Q UANTITATIVE R ESULTS

Original: In what follows, we evaluate structural representations estimated from four node embedding techniques, namely GIN (Xu et al., 2018), RP-GIN (Murphy et al., 2019), 1-2-3 GNN (Morris et al., 2019), MC-SVD and CGNN.
Revised: In what follows, we evaluate structural representations estimated from four node embedding techniques, namely GIN (Xu et al., 2018), RP-GIN (Murphy et al., 2019), 1-2-3 GNN (Morris et al., 2019), MC-SVD and CGNN.

Original: We classify GIN, RP-GIN and 1-2-3 GNN as node embedding techniques, as they employ the unsupervised learning procedure of Hamilton et al.
Revised: We classify GIN, RP-GIN and 1-2-3 GNN as node embedding techniques, as they employ the unsupervised learning procedure of Hamilton et al.

Original: (2017a).
Revised: (2017a).

Original: These were chosen because of their potential extra link and triad representation power over traditional structural representation GNNs.
Revised: These were chosen because of their potential extra link and triad representation power over traditional structural representation GNNs.

Original: All node embedding methods are evaluated by their effect in estimating good structural representation for downstream task accuracy.
Revised: All node embedding methods are evaluated by their effect in estimating good structural representation for downstream task accuracy.

Original: We partition G =
Revised: We partition G =

Original: ( A , X ) into three non-overlapping induced subgraphs, namely G train =
Revised: ( A , X ) into three non-overlapping induced subgraphs, namely G train =

Original: ( A train , X train ) , G val =
Revised: ( A train , X train ) , G val =

Original: ( A val , X val ) and G test =
Revised: ( A val , X val ) and G test =

Original: ( A test , X test ) , which we use for training, validation and testing, respectively.
Revised: ( A test , X test ) , which we use for training, validation and testing, respectively.

Original: In learning all four node embedding techniques, we only make use of the graphs G train and G val .
Revised: In learning all four node embedding techniques, we only make use of the graphs G train and G val .

Original: All the four models used here have never seen the test graph G test before test time —i.e., all our node embedding methods, used in the framework of Theorem 2, behave like inductive methods.
Revised: All the four models used here have never seen the test graph G test before test time —i.e., all our node embedding methods, used in the framework of Theorem 2, behave like inductive methods.

Original: Monte Carlo joint representations during an unsupervised learning phase: A key component of our optimization is learning joint representations from node embeddings —as per Theorem 2.
Revised: Monte Carlo joint representations during an unsupervised learning phase: A key component of our optimization is learning joint representations from node embeddings —as per Theorem 2.

Original: For this, at each gradient step (in practice, we do at each epoch), we perform a Monte Carlo sample of the node embeddings Z | A , X .
Revised: For this, at each gradient step (in practice, we do at each epoch), we perform a Monte Carlo sample of the node embeddings Z | A , X .

Original: This, procedure optimizes a proper upper bound on the empirical loss, if the loss is the negative log-likelihood, cross-entropy, or a square loss.
Revised: This, procedure optimizes a proper upper bound on the empirical loss, if the loss is the negative log-likelihood, cross-entropy, or a square loss.

Original: The proof is trivial by
Revised: The proof is trivial by

Original: Jensen’s inequality.
Revised: Jensen’s inequality.

Original: For GIN, RP-GIN and 1-2-3 GNN, we add random edges to the graph following a random walk at each epoch (Hamilton et al., 2017a).
Revised: For GIN, RP-GIN and 1-2-3 GNN, we add random edges to the graph following a random walk at each epoch (Hamilton et al., 2017a).

Original: For the MC-SVD procedure, we use the left eigenvector matrix obtained by: (1) a random seed, (2) a random input permutation of the adjacency matrix, and (3) a single optimization step, rather than running SVD until it converges.
Revised: For the MC-SVD procedure, we use the left eigenvector matrix obtained by: (1) a random seed, (2) a random input permutation of the adjacency matrix, and (3) a single optimization step, rather than running SVD until it converges.

Original: We also have results with MC-SVD † , which is the same procedure as before, but runs SVD until convergence —noting that the latter is likely to give deterministic results in large real-world graphs.
Revised: We also have results with MC-SVD † , which is the same procedure as before, but runs SVD until convergence —noting that the latter is likely to give deterministic results in large real-world graphs.

Original: Monte Carlo joint representations during a supervised learning phase: During the supervised phase, we ﬁrst estimate a structural joint representation ˆ Γ( (cid:126) S, A , X ) as the average of m ∈ { 1 , 5 , 20 }
Revised: Monte Carlo joint representations during a supervised learning phase: During the supervised phase, we ﬁrst estimate a structural joint representation ˆΓ( (cid:126)S, A , X ) as the average of m ∈ { 1 , 5 , 20 }

Original: Monte Carlo samples of a permutation-invariant function (Murphy et al., 2018;
Revised: Monte Carlo samples of a permutation-invariant function (Murphy et al., 2018;

Original: Zaheer et al., 2017)
Revised: Zaheer et al., 2017)

Original: (sum-pooling followed by an MLP) applied to a sampled node embedding ( Z v ) v ∈ S | A , X .
Revised: (sum-pooling followed by an MLP) applied to a sampled node embedding ( Z v ) v ∈ S | A , X .

Original: Then, using ˆΓ( (cid:126)S, A , X ) , we predict the corresponding target variable Y ( S, A , X ) of each task using an
Revised: Then, using ˆΓ( (cid:126)S, A , X ) , we predict the corresponding target variable Y ( S, A , X ) of each task using an

Original: MLP.
Revised: MLP.

Original: The node sets of our tasks S ⊆ V , have sizes | S | ∈ { 1 , 2 , 3 } , corresponding to node classiﬁcation, link prediction, and triad prediction tasks, respectively.
Revised: The node sets of our tasks S ⊆ V , have sizes | S | ∈ { 1 , 2 , 3 } , corresponding to node classiﬁcation, link prediction, and triad prediction tasks, respectively.

Original: Datasets: We consider four graph datasets used by Hamilton et al.
Revised: Datasets: We consider four graph datasets used by Hamilton et al.

Original: (2017a), namely Cora, Citeseer,
Revised: (2017a), namely Cora, Citeseer,

Original: Pubmed (Namata et al., 2012; Sen et al., 2008) and PPI (Zitnik & Leskovec, 2017).
Revised: Pubmed (Namata et al., 2012; Sen et al., 2008) and PPI (Zitnik & Leskovec, 2017).

Original: Cora, Citeseer and Pubmed are citation networks, where vertices represent papers, edges represent citations, and vertex features are bag-of-words representation of the document text.
Revised: Cora, Citeseer and Pubmed are citation networks, where vertices represent papers, edges represent citations, and vertex features are bag-of-words representation of the document text.

Original: The PPI (protein-protein interaction) dataset is a collection of multiple graphs representing the human tissue, where vertices represent proteins, edges represent interactions across them, and node features include genetic and immunological information.
Revised: The PPI (protein-protein interaction) dataset is a collection of multiple graphs representing the human tissue, where vertices represent proteins, edges represent interactions across them, and node features include genetic and immunological information.

Original: Train, validation and test splits are used as proposed by Yang et al.
Revised: Train, validation and test splits are used as proposed by Yang et al.

Original: (2016) (see Table 3 in the Appendix).
Revised: (2016) (see Table 3 in the Appendix).

Original: Further dataset details can be found in the Appendix.
Revised: Further dataset details can be found in the Appendix.

Revised: [Table]

Original: Node classiﬁcation task: This task predicts node classes for each of the four datasets.
Revised: [Figure] [Table] [Figure] [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Original: In this task, structural node representations are enough.
Revised: [Figure] [Table] [Figure] [Table]

Original: The structural node representation is used to classify nodes into different classes using an MLP, whose weights are trained in a supervised manner using
Revised: [Figure] [Table] [Figure]

Original: [Table]
Revised: [Table]

Original:  the same splits as described above.
Revised: Node classiﬁcation task: This task predicts node classes for each of the four datasets. In this task, structural node representations are enough. The structural node representation is used to classify nodes into different classes using an MLP, whose weights are trained in a supervised manner using the same splits as described above.

Original: In Cora, Citeseer and Pubmed, each vertex belongs only to a single class, whereas in the PPI graph dataset, nodes could belong to multiple classes.
Revised: In Cora, Citeseer and Pubmed, each vertex belongs only to a single class, whereas in the PPI graph dataset, nodes could belong to multiple classes.

Original: Link prediction task: Here, we predict a small fraction of edges and non-edges in the test graph, as well as identify all false edges and non-edges (which were introduced as a corruption of the original graph) between different pairs of nodes in the graph.
Revised: Link prediction task: Here, we predict a small fraction of edges and non-edges in the test graph, as well as identify all false edges and non-edges (which were introduced as a corruption of the original graph) between different pairs of nodes in the graph.

Original: Speciﬁcally, we use joint tuple representations Γ(( u, v ) , A , X ) , for u, v ∈ V , as prescribed by Theorem 2.
Revised: Speciﬁcally, we use joint tuple representations Γ(( u, v ) , A , X ) , for u, v ∈ V , as prescribed by Theorem 2.

Original: Since, datasets are sparse in nature, and a trivial ‘non-edges’ predictor would result in a very high accuracy, we balance the train and validation and test splits to contain an equal number of edges and non-edges.
Revised: Since, datasets are sparse in nature, and a trivial ‘non-edges’ predictor would result in a very high accuracy, we balance the train and validation and test splits to contain an equal number of edges and non-edges.

Original: Triad prediction task: This task involves the prediction of triadic interaction as well as identiﬁcation of possible fake interactions in the data between the three nodes under consideration.
Revised: Triad prediction task: This task involves the prediction of triadic interaction as well as identiﬁcation of possible fake interactions in the data between the three nodes under consideration.

Original: In this case, we use joint triadic representations Γ(( u, v, h ) , A , X ) , for u, v, h ∈ V , as prescribed by Theorem 2.
Revised: In this case, we use joint triadic representations Γ(( u, v, h ) , A , X ) , for u, v, h ∈ V , as prescribed by Theorem 2.

Original: Here, we ensure that edge corruptions are dependent.
Revised: Here, we ensure that edge corruptions are dependent.

Original: We treat the graphs as being undirected in accordance with previous literature, and predict the number of true (uncorrupted) edges between the three nodes.
Revised: We treat the graphs as being undirected in accordance with previous literature, and predict the number of true (uncorrupted) edges between the three nodes.

Original: Again, to handle the sparse nature of the graphs, we use a balanced dataset for train, validation, and test.
Revised: Again, to handle the sparse nature of the graphs, we use a balanced dataset for train, validation, and test.

Original: In Table 1 we present Micro-F1 scores for all four models over the three tasks.
Revised: In Table 1 we present Micro-F1 scores for all four models over the three tasks.

Original: First, we note how more Monte Carlo samples at test time tend to increase test accuracy.
Revised: First, we note how more Monte Carlo samples at test time tend to increase test accuracy.

Original: In node classiﬁcation tasks , we note that structural node representations from CGNN node embeddings signiﬁcantly outperform other methods in two of the three datasets (the harder tasks).
Revised: In node classiﬁcation tasks , we note that structural node representations from CGNN node embeddings signiﬁcantly outperform other methods in two of the three datasets (the harder tasks).

Original: In link prediction tasks , the low accuracy of GNN-based methods (close to random) showcases the little extra-power of GIN and RPGIN sampling schemes have over the the inability of structural node representations to predict links.
Revised: In link prediction tasks , the low accuracy of GNN-based methods (close to random) showcases the little extra-power of GIN and RPGIN sampling schemes have over the the inability of structural node representations to predict links.

Original: Surprisingly, in triads predictions , the accuracy of GNN-based methods is much above random in some datasets, but still far from other node embedding methods.
Revised: Surprisingly, in triads predictions , the accuracy of GNN-based methods is much above random in some datasets, but still far from other node embedding methods.

Original: In link and triad prediction tasks ,
Revised: In link and triad prediction tasks ,

Original: MC-SVD and CGNN share the lead with MC-SVD winning on Pubmed and PPI, and CGNN being signiﬁcantly more accurate on Cora.
Revised: MC-SVD and CGNN share the lead with MC-SVD winning on Pubmed and PPI, and CGNN being signiﬁcantly more accurate on Cora.

Original: Although, the 1-2-3 GNN is based on the two-WeisfeilerLehman (2-WL) algorithm (pairwise Weisfeiler-Lehman algorithm (F¨urer, 2017)), which provides tuple representations that can be exploited towards link prediction, it is however an approximation primarily designed for graph classiﬁcation tasks.
Revised: Although, the 1-2-3 GNN is based on the two-WeisfeilerLehman (2-WL) algorithm (pairwise Weisfeiler-Lehman algorithm (F¨urer, 2017)), which provides tuple representations that can be exploited towards link prediction, it is however an approximation primarily designed for graph classiﬁcation tasks.

Original: Unfortunately, the 1-2-3 GNN performs quite poorly on all our tasks (node classiﬁcation, link and triad prediction), indicating the need for a task-speciﬁc approximation of 2-WL GNN’s.
Revised: Unfortunately, the 1-2-3 GNN performs quite poorly on all our tasks (node classiﬁcation, link and triad prediction), indicating the need for a taskspeciﬁc approximation of 2-WL GNN’s.

Original: Results for Citeseer, in the Appendix, show similar results.
Revised: Results for Citeseer, in the Appendix, show similar results.

Original: 4. Q UALITATIVE R ESULTS
Revised: 4. Q UALITATIVE R ESULTS

Original: We now investigate the transformation of node embedding into node and link structural representations.
Revised: We now investigate the transformation of node embedding into node and link structural representations.

Original: Theorem 2 shows that the average of a function over node embedding Monte Carlo samples gives node and link embedding.
Revised: Theorem 2 shows that the average of a function over node embedding Monte Carlo samples gives node and link embedding.

Original: In this experiment, we empirically test Theorem 2, by creating[Figure] structural representations from the node embedding random matrix Z , deﬁned as the left eigenvector matrix obtained through SVD (ran until convergence), with the sources of randomness being due to a random permutation of the adjacency matrix given as input to the SVD method and the random seed it uses.
Revised: In this experiment, we empirically test Theorem 2, by creating structural representations from the node embedding random matrix Z , deﬁned as the left eigenvector matrix obtained through SVD (ran until convergence), with the sources of randomness being due to a random permutation of the adjacency matrix given as input to the SVD method and the random seed it uses.

Original: Consider m such embedding matrices Monte Carlo samples, Z ( m ) =
Revised: Consider m such embedding matrices Monte Carlo samples, Z ( m ) =

Original: { Z ( i ) } mi =1 .
Revised: { Z ( i ) } m i =1 .

Original:  Structural node representations from node embeddings: According to Theorem 2, the average E [ Z v, · | A ] is a valid structural representation of node v ∈ V in the adjacency matrix A of Figure 1.
Revised: [Figure] Structural node representations from node embeddings: According to Theorem 2, the average E [ Z v, · | A ] is a valid structural representation of node v ∈ V in the adjacency matrix A of Figure 1.

Original: To test this empirically, we consider the unbiased estimator ˆ µ ( v, Z ( m ) )
Revised: To test this empirically, we consider the unbiased estimator ˆ µ ( v, Z ( m ) )

Original: = 1 m (cid:80) mi
Revised: = 1 m (cid:80) mi

Original: =1 Z ( i ) v, · , v ∈ V , where lim m →∞ ˆ µ ( v, Z ( m ) )
Revised: =1 Z ( i ) v, · , v ∈ V , where lim m →∞ ˆ µ ( v, Z ( m ) )

Original: a.s. =
Revised: a.s. =

Original: E [ Z v, · | A ] .
Revised: E [ Z v, · | A ] .

Original: Figure 2a shows the Euclidean distance between the empirical structural representations ˆ µ ( orca , Z ( m ) ) and ˆ µ ( lynx , Z ( m ) ) as a function of m ∈ [1 , 200] .
Revised: Figure 2a shows the Euclidean distance between the empirical structural representations ˆ µ ( orca , Z ( m ) ) and ˆ µ ( lynx , Z ( m ) ) as a function of m ∈ [1 , 200] .

Original: As expected, because these two nodes are isomorphic, (cid:107) ˆ µ ( orca , Z ( m ) )
Revised: As expected, because these two nodes are isomorphic, (cid:107) ˆ µ ( orca , Z ( m ) )

Original: − ˆ µ ( lynx , Z ( m ) )
Revised: − ˆ µ ( lynx , Z ( m ) )

Original: (cid:107) → 0 as m grows, with m = 100 giving reasonably accurate results.
Revised: (cid:107) → 0 as m grows, with m = 100 giving reasonably accurate results.

Original: Structural link representations from node embeddings:
Revised: Structural link representations from node embeddings:

Original: According to Theorem 2, the average E [ f (2) ( Z u, · , Z v, · ) | A ] of a function f (2) is a valid structural representation of a link with nodes u, v ∈ V in the adjacency matrix A of Figure 1.
Revised: According to Theorem 2, the average E [ f (2) ( Z u, · , Z v, · ) | A ] of a function f (2) is a valid structural representation of a link with nodes u, v ∈ V in the adjacency matrix A of Figure 1.

Original: As an example, we use f (2) ( a, b ) = (cid:107) a − b (cid:107) , and deﬁne the unbiased estimator ˆ µ ( u, v, Z ( m ) )
Revised: As an example, we use f (2) ( a, b ) = (cid:107) a − b (cid:107) , and deﬁne the unbiased estimator ˆ µ ( u, v, Z ( m ) )

Original: = 1 m (cid:80) mi =1 (cid:107)
Revised: = 1 m (cid:80) mi =1 (cid:107)

Original: Z ( i ) u − Z
Revised: Z ( i ) u − Z

Original: ( i ) v (cid:107) , ∀ u, v ∈ V , where lim m →∞ ˆ µ ( u, v, Z ( m ) )
Revised: ( i ) v (cid:107) , ∀ u, v ∈ V , where lim m →∞ ˆ µ ( u, v, Z ( m ) )

Original: a.s. =
Revised: a.s. =

Original: E [ (cid:107) Z u, · − Z v, · (cid:107)| A ] .
Revised: E [ (cid:107) Z u, · − Z v, · (cid:107)| A ] .

Original: Figure 2b shows the impact of increasing the number of Monte Carlo samples m over the empirical structural representation of links.
Revised: Figure 2b shows the impact of increasing the number of Monte Carlo samples m over the empirical structural representation of links.

Original: We observe that although the empirical node representations of the orca and the lynx seem to converge to the same value, lim m →∞ ˆ µ ( orca , Z ( m ) )
Revised: We observe that although the empirical node representations of the orca and the lynx seem to converge to the same value, lim m →∞ ˆ µ ( orca , Z ( m ) )

Original: =
Revised: =

Original: lim m →∞ ˆ µ ( lynx , Z ( m ) ) , their empirical joint representations with the coyote converge to different values, lim m →∞ ˆ µ ( lynx, coyote , Z ( m ) )
Revised: lim m →∞ ˆ µ ( lynx , Z ( m ) ) , their empirical joint representations with the coyote converge to different values, lim m →∞ ˆ µ ( lynx, coyote , Z ( m ) )

Original: (cid:54) =
Revised: (cid:54) =

Original: lim m →∞ ˆ µ ( orca, coyote , Z ( m ) ) , as predicted by Theorem 2.
Revised: lim m →∞ ˆ µ ( orca, coyote , Z ( m ) ) , as predicted by Theorem 2.

Original: Also note a similar (but weaker) trend for lim m →∞ ˆ µ ( orca, lynx , Z ( m ) )
Revised: Also note a similar (but weaker) trend for lim m →∞ ˆ µ ( orca, lynx , Z ( m ) )

Original: (cid:54) =
Revised: (cid:54) =

Original: lim m →∞ ˆ µ ( orca, coyote , Z ( m ) ) , showing these three tuples to be structurally different.
Revised: lim m →∞ ˆ µ ( orca, coyote , Z ( m ) ) , showing these three tuples to be structurally different.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: Node Embeddings vs Structural Representations: Prior works have categorized themselves as one of either node embedding methods or methods which learn structural representations.
Revised: Node Embeddings vs Structural Representations: Prior works have categorized themselves as one of either node embedding methods or methods which learn structural representations.

Original: This artiﬁcial separation, consequently led to little contemplation of the relation between the two, restricting each of these approaches to a certain subsets of downstream tasks on graphs.
Revised: This artiﬁcial separation, consequently led to little contemplation of the relation between the two, restricting each of these approaches to a certain subsets of downstream tasks on graphs.

Original: Node embeddings were arguably ﬁrst deﬁned in 1904, through Spearman’s common factors.
Revised: Node embeddings were arguably ﬁrst deﬁned in 1904, through Spearman’s common factors.

Original: Ever since, there has never been a universal deﬁnition of node embedding: node embeddings were simply the product of a particular method.
Revised: Ever since, there has never been a universal deﬁnition of node embedding: node embeddings were simply the product of a particular method.

Original: This literature features a myriad of methods, e.g., matrix factorization (Belkin &
Revised: This literature features a myriad of methods, e.g., matrix factorization (Belkin &

Original: Niyogi, 2002; Cao et al., 2015; Ahmed et al., 2013; Ou et al., 2016), implicit matrix factorization (Mikolov et al., 2013; Arora et al., 2009; Chen et al., 2012; Perozzi et al., 2014; Grover & Leskovec, 2016), Bayesian factor models (Mnih & Salakhutdinov, 2008; Gopalan et al., 2014a), and some types of neural networks (You et al., 2019; Liang et al., 2018; Tang et al., 2019; Grover et al., 2019).
Revised: Niyogi, 2002; Cao et al., 2015; Ahmed et al., 2013; Ou et al., 2016), implicit matrix factorization (Mikolov et al., 2013; Arora et al., 2009; Chen et al., 2012; Perozzi et al., 2014; Grover & Leskovec, 2016), Bayesian factor models (Mnih & Salakhutdinov, 2008; Gopalan et al., 2014a), and some types of neural networks (You et al., 2019; Liang et al., 2018; Tang et al., 2019; Grover et al., 2019).

Original: Arguably, the most common interpretation of node embeddings borrows from deﬁnitions of graph (node) embeddings in metric spaces: a measure of relative node closeness (Abraham et al., 2006;
Revised: Arguably, the most common interpretation of node embeddings borrows from deﬁnitions of graph (node) embeddings in metric spaces: a measure of relative node closeness (Abraham et al., 2006;

Original: Bourgain, 1985; Cand`es & Recht, 2009; Graham & Winkler, 1985; Kleinberg, 2007; Linial et al., 1995; Rabinovich & Raz, 1998; Recht et al., 2010; Shaw et al., 2011).
Revised: Bourgain, 1985; Cand`es & Recht, 2009; Graham & Winkler, 1985; Kleinberg, 2007; Linial et al., 1995; Rabinovich & Raz, 1998; Recht et al., 2010; Shaw et al., 2011).

Original: Even in non-metric methods, such as word2vec (Mikolov et al., 2013) and Glove (Pennington et al., 2014), the embeddings have properties similar to those of metric spaces (Nematzadeh et al., 2017).
Revised: Even in non-metric methods, such as word2vec (Mikolov et al., 2013) and Glove (Pennington et al., 2014), the embeddings have properties similar to those of metric spaces (Nematzadeh et al., 2017).

Original: Note that the deﬁnition of close varies from method to method, i.e., it is model-dependent .
Revised: Note that the deﬁnition of close varies from method to method, i.e., it is model-dependent .

Original: Still, this interpretation of closeness is the reason why their downstream tasks are often link prediction and clustering.
Revised: Still, this interpretation of closeness is the reason why their downstream tasks are often link prediction and clustering.

Original: However, once the literature started deﬁning relative node closeness with respect to structural neighborhood similarities (e.g., Henderson et al.
Revised: However, once the literature started deﬁning relative node closeness with respect to structural neighborhood similarities (e.g., Henderson et al.

Original: ; Ribeiro et al.
Revised: ; Ribeiro et al.

Original: ; Donnat et al.
Revised: ; Donnat et al.

Original: , node embeddings and structural representations became more strangely entangled.
Revised: , node embeddings and structural representations became more strangely entangled.

Original: Structural representations have an increasing body of literature focused on node and whole-graph classiﬁcation tasks.
Revised: Structural representations have an increasing body of literature focused on node and whole-graph classiﬁcation tasks.

Original: Theoretically, these works abandon metric spaces in favor of a group-theoretic description of graphs (Bloem-Reddy & Teh, 2019; Chen et al., 2019; Kondor & Trivedi, 2018;
Revised: Theoretically, these works abandon metric spaces in favor of a group-theoretic description of graphs (Bloem-Reddy & Teh, 2019; Chen et al., 2019; Kondor & Trivedi, 2018;

Original: Maron et al., 2019; Murphy et al., 2019), with connections to ﬁnite exchangeability and prior work on multilayer perceptrons (Wood & Shawe-Taylor, 1996).
Revised: Maron et al., 2019; Murphy et al., 2019), with connections to ﬁnite exchangeability and prior work on multilayer perceptrons (Wood & Shawe-Taylor, 1996).

Original: Graph neural networks (GNNs) (e.g., (Duvenaud et al., 2015; Gilmer et al., 2017; Kipf & Welling, 2016a; Hamilton et al., 2017a;
Revised: Graph neural networks (GNNs) (e.g., (Duvenaud et al., 2015; Gilmer et al., 2017; Kipf & Welling, 2016a; Hamilton et al., 2017a;

Original: Xu et al., 2018; Scarselli et al., 2008) among others) exploit this approach in tasks such as node and wholegraph classiﬁcation.
Revised: Xu et al., 2018; Scarselli et al., 2008) among others) exploit this approach in tasks such as node and wholegraph classiﬁcation.

Original: Morris et al.
Revised: Morris et al.

Original: (2019) proposes a higher-order Weisfeller-Lehman GNN (WL- kGNN), which is shown to get better accuracy in graph classiﬁcation tasks than traditional (WL-1)
Revised: (2019) proposes a higher-order Weisfeller-Lehman GNN (WL- kGNN), which is shown to get better accuracy in graph classiﬁcation tasks than traditional (WL-1)

Original: GNNs.
Revised: GNNs.

Original: Unfortunately, Morris et al.
Revised: Unfortunately, Morris et al.

Original: (2019) focused only on graph-wide tasks, missing the fact that
Revised: (2019) focused only on graph-wide tasks, missing the fact that

Original: WL-2 GNN should be able to also perform link prediction tasks (Theorem 1), unlike WL-1 GNNs.
Revised: WL-2 GNN should be able to also perform link prediction tasks (Theorem 1), unlike WL-1 GNNs.

Original: More recently, graph neural networks have also been employed towards relational reasoning as well as matrix completion tasks (Schlichtkrull et al., 2018; Zhang & Chen, 2018; Battaglia et al., 2018;
Revised: More recently, graph neural networks have also been employed towards relational reasoning as well as matrix completion tasks (Schlichtkrull et al., 2018; Zhang & Chen, 2018; Battaglia et al., 2018;

Original: Berg et al., 2017; Monti et al., 2017).
Revised: Berg et al., 2017; Monti et al., 2017).

Original: However, these GNN’s, in general, learn node embeddings rather than structural node representations, which are then exploited towards link prediction.
Revised: However, these GNN’s, in general, learn node embeddings rather than structural node representations, which are then exploited towards link prediction.

Original: GNNlike architectures have been used to simulate dynamic programming algorithms (Xu et al., 2019), which is unrelated to graphs and outside the scope of this work.
Revised: GNNlike architectures have been used to simulate dynamic programming algorithms (Xu et al., 2019), which is unrelated to graphs and outside the scope of this work.

Original: To the best of our knowledge, our work is the ﬁrst to provide the theoretical foundations connecting node embeddings and structural representations.
Revised: To the best of our knowledge, our work is the ﬁrst to provide the theoretical foundations connecting node embeddings and structural representations.

Original: A few recent works have classiﬁed node embedding and graph representation methods arguing them to be fundamentally different (e.g., Hamilton et al.
Revised: A few recent works have classiﬁed node embedding and graph representation methods arguing them to be fundamentally different (e.g., Hamilton et al.

Original: (2017b)
Revised: (2017b)

Original: ; Rossi et al.
Revised: ; Rossi et al.

Original: ; Wu et al.
Revised: ; Wu et al.

Original: ; Zhou et al.
Revised: ; Zhou et al.

Original: Rather, our work shows that these are actually equivalent for downstream classiﬁcation tasks, with the difference being that one is a
Revised: Rather, our work shows that these are actually equivalent for downstream classiﬁcation tasks, with the difference being that one is a

Original: Monte Carlo method (embedding) and the other one is deterministic (representation).
Revised: Monte Carlo method (embedding) and the other one is deterministic (representation).

Original: Inductive vs Transductive Approaches: Another common misconception our work uncovers, is that of qualifying node embedding methods as transductive learning and graph representation ones as inductive (e.g., Hamilton et al.
Revised: Inductive vs Transductive Approaches: Another common misconception our work uncovers, is that of qualifying node embedding methods as transductive learning and graph representation ones as inductive (e.g., Hamilton et al.

Original: (2017a)
Revised: (2017a)

Original: ; Yang et al.
Revised: ; Yang et al.

Original: In their original deﬁnitions, transductive learning (Gammerman et al., 1998), (Zhu et al., 2003), (Zhou et al., 2004) and inductive learning (Michalski, 1983), (Belkin et al., 2006) are only to be distinguished on the basis of generalizability of the learned model to unobserved instances.
Revised: In their original deﬁnitions, transductive learning (Gammerman et al., 1998), (Zhu et al., 2003), (Zhou et al., 2004) and inductive learning (Michalski, 1983), (Belkin et al., 2006) are only to be distinguished on the basis of generalizability of the learned model to unobserved instances.

Original: However, this has commonly been misinterpreted as node embeddings methods being transductive and structural representations being inductive.
Revised: However, this has commonly been misinterpreted as node embeddings methods being transductive and structural representations being inductive.

Original: Models which depend solely only on the input feature vectors and the immediate neighborhood structure have been classiﬁed as inductive, whereas methods which rely on positional node embeddings to classify relationships in a graph have been incorrectly qualiﬁed as transductive.
Revised: Models which depend solely only on the input feature vectors and the immediate neighborhood structure have been classiﬁed as inductive, whereas methods which rely on positional node embeddings to classify relationships in a graph have been incorrectly qualiﬁed as transductive.

Original: The confusion seems to be rooted in researchers trying to use a single sample of a node embedding method and failing to generalize.
Revised: The confusion seems to be rooted in researchers trying to use a single sample of a node embedding method and failing to generalize.

Original: Corollary 4 resolves this confusion by showing that transductive and inductive learning are fundamentally unrelated to positional node embeddings and graph representations.
Revised: Corollary 4 resolves this confusion by showing that transductive and inductive learning are fundamentally unrelated to positional node embeddings and graph representations.

Original: Both node embeddings and structural representations can be inductive if they can detect interesting conceptual patterns or reveal structure in the data.
Revised: Both node embeddings and structural representations can be inductive if they can detect interesting conceptual patterns or reveal structure in the data.

Original: The theory provided by our work strongly adheres to this deﬁnition.
Revised: The theory provided by our work strongly adheres to this deﬁnition.

Original: Our work additionally provides the theoretical foundation behind the performance gains seen by Epasto & Perozzi (2019) and Goyal et al.
Revised: Our work additionally provides the theoretical foundation behind the performance gains seen by Epasto & Perozzi (2019) and Goyal et al.

Original: (2019), which employ an ensemble of node embeddings for node classiﬁcation tasks.
Revised: (2019), which employ an ensemble of node embeddings for node classiﬁcation tasks.

Original:  C ONCLUSIONS
Revised:  C ONCLUSIONS

Original: This work provided an invaluable unifying theoretical framework for node embeddings and structural graph representations, bridging methods like SVD and graph neural networks.
Revised: This work provided an invaluable unifying theoretical framework for node embeddings and structural graph representations, bridging methods like SVD and graph neural networks.

Original: Using invariant theory, we have shown (both theoretically and empirically) that relationship between structural representations and node embeddings is analogous to that of a distribution and its samples.
Revised: Using invariant theory, we have shown (both theoretically and empirically) that relationship between structural representations and node embeddings is analogous to that of a distribution and its samples.

Original: We proved that all tasks that can be performed by node embeddings can also be performed by structural representations and vice-versa.
Revised: We proved that all tasks that can be performed by node embeddings can also be performed by structural representations and vice-versa.

Original: Our empirical results show that node embeddings can be successfully used as inductive learning methods using our framework, and that non-GNN node embedding methods can be signiﬁcantly more accurate in most tasks than simple GNNs methods.
Revised: Our empirical results show that node embeddings can be successfully used as inductive learning methods using our framework, and that non-GNN node embedding methods can be signiﬁcantly more accurate in most tasks than simple GNNs methods.

Original: Our work introduced new practical guidelines to the use of node embeddings, which we expect will replace today’s na¨ıve direct use of node embeddings in graph tasks.
Revised: Our work introduced new practical guidelines to the use of node embeddings, which we expect will replace today’s na¨ıve direct use of node embeddings in graph tasks.

Original: A CKNOWLEDGMENTS
Revised: A CKNOWLEDGMENTS

Original: This work was sponsored in part by the ARO, under the U.S. Army Research Laboratory contract number W911NF-09-2-0053, the Purdue Integrative Data Science Initiative and the Purdue Research foundation, the DOD through SERC under contract number HQ0034-13-D-0004 RT # 206, and the
Revised: This work was sponsored in part by the ARO, under the U.S. Army Research Laboratory contract number W911NF-09-2-0053, the Purdue Integrative Data Science Initiative and the Purdue Research foundation, the DOD through SERC under contract number HQ0034-13-D-0004 RT # 206, and the

Original: National Science Foundation under contract number CCF-1918483.
Revised: National Science Foundation under contract number CCF-1918483.

Original: A PPENDIX A VISUAL INTERPRETATION OF NODE EMBEDDINGS AND STRUCTURAL REPRESENTATIONS
Revised: A PPENDIX A VISUAL INTERPRETATION OF NODE EMBEDDINGS AND STRUCTURAL REPRESENTATIONS

Original: In what follows we showcase the difference between structural representations (Figure 3) and positional node embeddings (Figure 4) obtained by graph neural networks and SVD, respectively, on the same graph.
Revised: In what follows we showcase the difference between structural representations (Figure 3) and positional node embeddings (Figure 4) obtained by graph neural networks and SVD, respectively, on the same graph.

Original: The graph is the food web of Figure 1 and consists of two disconnected subgraphs.
Revised: The graph is the food web of Figure 1 and consists of two disconnected subgraphs.

Original: Figure 3 shows a 2-dimensional ( R 2 ) structural representation of nodes obtained by a standard
Revised: Figure 3 shows a 2-dimensional ( R 2 ) structural representation of nodes obtained by a standard

Original: GNN (obtained by the 1-WL GNN of Xu et al.
Revised: GNN (obtained by the 1-WL GNN of Xu et al.

Original: (2018)), optimized to try to predict links without the addition of any random edges).
Revised: (2018)), optimized to try to predict links without the addition of any random edges).

Original: Node colors represent the mapping of the learned structural representation in R 2 into the red and blue interval of RGB space [0 , 255] 2 .
Revised: Node colors represent the mapping of the learned structural representation in R 2 into the red and blue interval of RGB space [0 , 255] 2 .

Original: Note that isomorphic nodes are forcibly mapped into the same structural representations (have the same color), even though the representation is trained to predict links.
Revised: Note that isomorphic nodes are forcibly mapped into the same structural representations (have the same color), even though the representation is trained to predict links.

Original: Speciﬁcally, the lynx and the orca in Figure 3 get the same color since they are isomorphic, while the lynx and the coyote have different structural representations (different colors).
Revised: Speciﬁcally, the lynx and the orca in Figure 3 get the same color since they are isomorphic, while the lynx and the coyote have different structural representations (different colors).

Original: The lynx, like the orca, is a top predator in the food web while the coyote is not a top predator.
Revised: The lynx, like the orca, is a top predator in the food web while the coyote is not a top predator.

Original: Hence, it is quite evident why GNN’s are not traditionally used for link prediction: as the structural node representation is a color, using these representations is akin to asking “is a light pink node in Figure 3 connected to a light blue node?”
Revised: Hence, it is quite evident why GNN’s are not traditionally used for link prediction: as the structural node representation is a color, using these representations is akin to asking “is a light pink node in Figure 3 connected to a light blue node?”

Original: We cannot answer this question unless we also specify which light pink (i.e. coyote or seal) and which light blue (i.e. orca or lynx)
Revised: We cannot answer this question unless we also specify which light pink (i.e. coyote or seal) and which light blue (i.e. orca or lynx)

Original: we are talking about.
Revised: we are talking about.

Original: Hence, the failure to predict links.
Revised: Hence, the failure to predict links.

Original: Our Corollary 2 shows that joint 2-node structural representations are required for link prediction.
Revised: Our Corollary 2 shows that joint 2-node structural representations are required for link prediction.

Original: Our Theorem 1 generalizes it to joint k -node structural representations for any k -node joint prediction task.
Revised: Our Theorem 1 generalizes it to joint k -node structural representations for any k -node joint prediction task.

Original: [Figure]
Revised: [Figure]

Original: representation from a 1-WL GNN (GIN - (Xu et al., 2018)) into the [0 , 255] interval of blue and red intensity of RGB colors respectively.
Revised: representation from a 1-WL GNN (GIN - (Xu et al., 2018)) into the [0 , 255] interval of blue and red intensity of RGB colors respectively.

Original: GIN is used a representative of structural representation of nodes.
Revised: GIN is used a representative of structural representation of nodes.

Original: Structurally isomorphic nodes, obtain the same representation and are hence end up being visualised with the same color.
Revised: Structurally isomorphic nodes, obtain the same representation and are hence end up being visualised with the same color.

Original: Consequently, it is clear why structural node representations are used for node and graph classiﬁcation, but not for link prediction.
Revised: Consequently, it is clear why structural node representations are used for node and graph classiﬁcation, but not for link prediction.

Original: Positional node embeddings, on the other hand, are often seen as a lower-dimensional projection of the rows and columns of the adjacency matrix A from R n to R d , d < n , that preserves relative positions of the nodes in a graph.
Revised: Positional node embeddings, on the other hand, are often seen as a lower-dimensional projection of the rows and columns of the adjacency matrix A from R n to R d , d < n , that preserves relative positions of the nodes in a graph.

Original: In Figure 4 we show the same food web graph, where now node colors map the values of the two leading (SVD) eigenvectors (of the undirected food web graph) into the [0 , 255] interval of blue and red intensity of RGB colors, respectively.
Revised: In Figure 4 we show the same food web graph, where now node colors map the values of the two leading (SVD) eigenvectors (of the undirected food web graph) into the [0 , 255] interval of blue and red intensity of RGB colors, respectively.

Original: The graph is made undirected, otherwise the left and right eigenvectors will be different and harder to represent visually.
Revised: The graph is made undirected, otherwise the left and right eigenvectors will be different and harder to represent visually.

Original: SVD is used here as a representative of positional node embedding methods.
Revised: SVD is used here as a representative of positional node embedding methods.

Original: Note the lynx and the coyote now have close positional node embeddings (represented by colors which are closer in the color spectrum), while the positional node embeddings of the lynx and the orca are signiﬁcantly different.
Revised: Note the lynx and the coyote now have close positional node embeddings (represented by colors which are closer in the color spectrum), while the positional node embeddings of the lynx and the orca are signiﬁcantly different.

Original: Node embeddings are often seen as encoding the fact that the lynx and the coyote are part of a tightly-knit community, while the lynx and orca belong to distinct communities.
Revised: Node embeddings are often seen as encoding the fact that the lynx and the coyote are part of a tightly-knit community, while the lynx and orca belong to distinct communities.

Original: It is evident from Figure 4 why positional node embeddings are not traditionally used to predict node classes: predicting that the lynx, like the orca, is a top predator based on their node colors is difﬁcult, since the coyote’s color is very similar to that of the lynx, while the orca obtains a completely different color to the lynx.
Revised: It is evident from Figure 4 why positional node embeddings are not traditionally used to predict node classes: predicting that the lynx, like the orca, is a top predator based on their node colors is difﬁcult, since the coyote’s color is very similar to that of the lynx, while the orca obtains a completely different color to the lynx.

Original: Relying on color shades (light and dark) for node classiﬁcation is unreliable, since nodes with completely different structural positions in the food chain may have similar color shades.
Revised: Relying on color shades (light and dark) for node classiﬁcation is unreliable, since nodes with completely different structural positions in the food chain may have similar color shades.

Original: Our Theorem 2 shows, however, that any positional node embedding (actually, any node embedding) must be a sample of a distribution given by the most-expressive structural representation of the node.
Revised: Our Theorem 2 shows, however, that any positional node embedding (actually, any node embedding) must be a sample of a distribution given by the most-expressive structural representation of the node.

Original: That is, if SVD is ran again over different permutations of the vertices in the adjacency matrix (an isomorphic graph to Figure 1), the set of colors obtained by the lynx and the orca must be the same.
Revised: That is, if SVD is ran again over different permutations of the vertices in the adjacency matrix (an isomorphic graph to Figure 1), the set of colors obtained by the lynx and the orca must be the same.

Original: Hence, node classiﬁcation is possible if we look at the distribution of colors that a node obtains from random permutations of the adjacency matrix.
Revised: Hence, node classiﬁcation is possible if we look at the distribution of colors that a node obtains from random permutations of the adjacency matrix.

Original: On the other hand, link prediction using the colors in Figure 4 is rather trivial, since similar colors mean “closeness” in the graph.
Revised: On the other hand, link prediction using the colors in Figure 4 is rather trivial, since similar colors mean “closeness” in the graph.

Original: For instance, we may easily predict that the baleen whale also eats zooplankton.
Revised: For instance, we may easily predict that the baleen whale also eats zooplankton.

Original: [Figure]
Revised: [Figure]

Original:  P RELIMINARIES
Revised:  P RELIMINARIES

Original: Noise outsourcing, representation learning, and graph representations: The description of our proofs starts with the equivalence between probability models of graphs and graph representations.
Revised: Noise outsourcing, representation learning, and graph representations: The description of our proofs starts with the equivalence between probability models of graphs and graph representations.

Original: We start with the concept of noise outsourcing (Austin, 2008, Lemma 3.1) applied to our task —a weaker version of the more general concept of transfer (Kallenberg, 2006, Theorem 6.10) in pushforward measures.
Revised: We start with the concept of noise outsourcing (Austin, 2008, Lemma 3.1) applied to our task —a weaker version of the more general concept of transfer (Kallenberg, 2006, Theorem 6.10) in pushforward measures.

Original: A probability law Z | ( A , X ) ∼ p ( ·| ( A , X )) , ( A , X ) ∈ Σ , can be described (Kallenberg, 2006,
Revised: A probability law Z | ( A , X ) ∼ p ( ·| ( A , X )) , ( A , X ) ∈ Σ , can be described (Kallenberg, 2006,

Original: Theorem 6.10) by pure random noise (cid:15) ∼ Uniform (0 , 1) independent of ( A , X ) , passing through a deterministic function Z = f
Revised: Theorem 6.10) by pure random noise (cid:15) ∼ Uniform (0 , 1) independent of ( A , X ) , passing through a deterministic function Z = f

Original: (( A , X ) , (cid:15) ) —where f : Σ × [0 , 1] → Ω , where Ω in our task will be a matrix Ω =
Revised: (( A , X ) , (cid:15) ) —where f : Σ × [0 , 1] → Ω , where Ω in our task will be a matrix Ω =

Original: R n × d deﬁning node representations, d ≥ 1 .
Revised: R n × d deﬁning node representations, d ≥ 1 .

Original: That is, the randomness in the conditional p ( z | ( A , X )) is entirely outsourced to (cid:15) , as f is deterministic.
Revised: That is, the randomness in the conditional p ( z | ( A , X )) is entirely outsourced to (cid:15) , as f is deterministic.

Original: Now, consider replacing the graph G =
Revised: Now, consider replacing the graph G =

Original: ( A , X ) by a G -equivariant representation Γ( A , X ) of its nodes, the output of a neural network Γ : Σ
Revised: ( A , X ) by a G -equivariant representation Γ( A , X ) of its nodes, the output of a neural network Γ : Σ

Original: → R n × m , m ≥ 1 , that gives an representation to each node in G .
Revised: → R n × m , m ≥ 1 , that gives an representation to each node in G .

Original: If a representation Γ (cid:63) ( A , X ) is such that ∃ f (cid:48)
Revised: If a representation Γ (cid:63) ( A , X ) is such that ∃ f (cid:48)

Original: where Z = f ( A , X , (cid:15) )
Revised: where Z = f ( A , X , (cid:15) )

Original: =
Revised: =

Original: f (cid:48) (Γ (cid:63) ( A , X ) , (cid:15) ) , ∀ ( A , X ) ∈ Σ and ∀ (cid:15) ∈ [0 , 1] , then Γ (cid:63) ( A , X ) does not lose any information when it comes to predicting Z .
Revised: f (cid:48) (Γ (cid:63) ( A , X ) , (cid:15) ) , ∀ ( A , X ) ∈ Σ and ∀ (cid:15) ∈ [0 , 1] , then Γ (cid:63) ( A , X ) does not lose any information when it comes to predicting Z .

Original: Statistically (Kallenberg, 2006, Proposition 6.13), Z ⊥⊥ Γ (cid:63) ( A , X ) ( A , X ) .
Revised: Statistically (Kallenberg, 2006, Proposition 6.13), Z ⊥⊥ Γ (cid:63) ( A , X ) ( A , X ) .

Original: We call Γ (cid:63) ( A , X ) a most-expressive representation of G with respect to Z .
Revised: We call Γ (cid:63) ( A , X ) a most-expressive representation of G with respect to Z .

Original: A most-expressive representation (without qualiﬁcations) is one that is most-expressive for any target variable.
Revised: A most-expressive representation (without qualiﬁcations) is one that is most-expressive for any target variable.

Original: Representation learning is powerful precisely because it can learn functions Γ that are compact and can encode most of the information available on the input.
Revised: Representation learning is powerful precisely because it can learn functions Γ that are compact and can encode most of the information available on the input.

Original: And because the most-expressive Γ (cid:63) is G -equivariant, it also guarantees that any G -equivariant function over Γ (cid:63) that outputs Z is also
Revised: And because the most-expressive Γ (cid:63) is G -equivariant, it also guarantees that any G -equivariant function over Γ (cid:63) that outputs Z is also

Original: G -equivariant, without loss of information.
Revised: G -equivariant, without loss of information.

Original:  P ROOF OF T HEOREMS , L EMMAS AND C OROLLARIES
Revised:  P ROOF OF T HEOREMS , L EMMAS AND C OROLLARIES

Original: We restate and prove Lemma
Revised: We restate and prove Lemma

Original: Lemma 1.
Revised: Lemma 1.

Original: Two nodes v, u ∈ V , have the same most-expressive structural representation Γ (cid:63) ( v, A , X ) =
Revised: Two nodes v, u ∈ V , have the same most-expressive structural representation Γ (cid:63) ( v, A , X ) =

Original: Γ (cid:63) ( u, A , X ) iff u and v are isomorphic nodes in G =
Revised: Γ (cid:63) ( u, A , X ) iff u and v are isomorphic nodes in G =

Original: ( A , X ) (Deﬁnition 6).
Revised: ( A , X ) (Deﬁnition 6).

Original: Proof.
Revised: Proof.

Original: In this proof, we consider both directions.
Revised: In this proof, we consider both directions.

Original: ( ⇒ ) Consider two nodes v, u ∈ V which satisfy the condition, Γ (cid:63) ( v, A , X )
Revised: ( ⇒ ) Consider two nodes v, u ∈ V which satisfy the condition, Γ (cid:63) ( v, A , X )

Original: =
Revised: =

Original: Γ (cid:63) ( u, A , X ) but are not isomorphic in G =
Revised: Γ (cid:63) ( u, A , X ) but are not isomorphic in G =

Original: ( A , X ) .
Revised: ( A , X ) .

Original: By contradiction, suppose u and v have different node orbits.
Revised: By contradiction, suppose u and v have different node orbits.

Original: This is a contradiction, since the bijective mapping of Deﬁnition 9 would have to take the same input and map them to different outputs.
Revised: This is a contradiction, since the bijective mapping of Deﬁnition 9 would have to take the same input and map them to different outputs.

Original: ( ⇐ ) By contradiction, consider the two nodes u, v ∈ V which are isomorphic in G =
Revised: ( ⇐ ) By contradiction, consider the two nodes u, v ∈ V which are isomorphic in G =

Original: ( A , X ) but with different most expressive structural representations i.e. Γ (cid:63) ( v, A , X ) (cid:54) =
Revised: ( A , X ) but with different most expressive structural representations i.e. Γ (cid:63) ( v, A , X ) (cid:54) =

Original: Γ (cid:63) ( u, A , X ) .
Revised: Γ (cid:63) ( u, A , X ) .

Original: This is a contradiction, because as per Deﬁnition 6 two nodes should have the same structural representation, which would imply the most expressive structural representation is not a structural representation.
Revised: This is a contradiction, because as per Deﬁnition 6 two nodes should have the same structural representation, which would imply the most expressive structural representation is not a structural representation.

Original: Hence, the two nodes should share the same most expressive structural representation.
Revised: Hence, the two nodes should share the same most expressive structural representation.

Original: Next, we restate and prove Lemma
Revised: Next, we restate and prove Lemma

Original: Lemma 2 (Causal modeling through noise outsourcing) .
Revised: Lemma 2 (Causal modeling through noise outsourcing) .

Original: Deﬁnition 1 of Galles & Pearl (1998) gives a causal model as a triplet
Revised: Deﬁnition 1 of Galles & Pearl (1998) gives a causal model as a triplet

Original: [Equation]
Revised: [Equation]

Original: , where U is a set of exogenous variables, V is a set of endogenous variables, and F is a set of functions, such that in the causal model, v i =
Revised: , where U is a set of exogenous variables, V is a set of endogenous variables, and F is a set of functions, such that in the causal model, v i =

Original: f
Revised: f

Original: ( (cid:126)pa i , u ) is the realization of random variable V i ∈ V and a sequence of random variables (cid:126)PA i with PA i ⊆
Revised: ( (cid:126)pa i , u ) is the realization of random variable V i ∈ V and a sequence of random variables (cid:126)PA i with PA i ⊆

Original: V \ V i as the endogenous variable parents of variable V
Revised: V \ V i as the endogenous variable parents of variable V

Original: i as given by a directed acyclic graph.
Revised: i as given by a directed acyclic graph.

Original: Then, there exists a pure random noise (cid:15) and a set of (measurable) functions { g u } u ∈ U such that for V i ∈ V , V i can be equivalently deﬁned as v i a.s. =
Revised: Then, there exists a pure random noise (cid:15) and a set of (measurable) functions { g u } u ∈ U such that for V i ∈ V , V i can be equivalently deﬁned as v i a.s. =

Original: f ( (cid:126)pa i , g u ( (cid:15) u )) , where (cid:15) u has joint distribution ( (cid:15) u ) ∀ u
Revised: f ( (cid:126)pa i , g u ( (cid:15) u )) , where (cid:15) u has joint distribution ( (cid:15) u ) ∀ u

Original: ∈ U a.s. =
Revised: ∈ U a.s. =

Original: g (cid:48) ( (cid:15) ) for some Borel measurable function g (cid:48) and a random variable (cid:15) ∼ Uniform (0 , 1) .
Revised: g (cid:48) ( (cid:15) ) for some Borel measurable function g (cid:48) and a random variable (cid:15) ∼ Uniform (0 , 1) .

Original: The latter deﬁnes M via noise outsourcing (Austin, 2008).
Revised: The latter deﬁnes M via noise outsourcing (Austin, 2008).

Original: Proof.
Revised: Proof.

Original: By Deﬁnition 1 of Galles & Pearl (1998), the set of exogenous variables U are not affected by the set of endogeneous variables V .
Revised: By Deﬁnition 1 of Galles & Pearl (1998), the set of exogenous variables U are not affected by the set of endogeneous variables V .

Original: The noise outsourcing lemma (Lemma 3.1) of Austin (2008) (or its more complete version Theorem 6.10 of Kallenberg (2006)) shows that any samples of a joint distribution over a set of random variables U can be described as ( u ) ∀ u ∈ U a.s.
Revised: The noise outsourcing lemma (Lemma 3.1) of Austin (2008) (or its more complete version Theorem 6.10 of Kallenberg (2006)) shows that any samples of a joint distribution over a set of random variables U can be described as ( u ) ∀ u ∈ U a.s.

Original: = g ( (cid:15) ) , for some Borel measurable function g and a random variable (cid:15) ∼ Uniform (0 , 1) .
Revised: = g ( (cid:15) ) , for some Borel measurable function g and a random variable (cid:15) ∼ Uniform (0 , 1) .

Original: As the composition of two Borel measurable functions is also Borel measurable, it is trivial to show that there exists Borel measurable functions { g u } u ∈ U and g (cid:48) , such that u a.s. =
Revised: As the composition of two Borel measurable functions is also Borel measurable, it is trivial to show that there exists Borel measurable functions { g u } u ∈ U and g (cid:48) , such that u a.s. =

Original: g u ( (cid:15) u ) and ( (cid:15) u ) ∀ u ∈
Revised: g u ( (cid:15) u ) and ( (cid:15) u ) ∀ u ∈

Original: U
Revised: U

Original: a.s. =
Revised: a.s. =

Original: g (cid:48) ( (cid:15) ) .
Revised: g (cid:48) ( (cid:15) ) .

Original: The latter is trivial since g u can just be the identity function, g u ( x ) = x .
Revised: The latter is trivial since g u can just be the identity function, g u ( x ) = x .

Original: Next, we restate and prove Lemma
Revised: Next, we restate and prove Lemma

Original: Lemma 3.
Revised: Lemma 3.

Original: The permutation equivariance of p in Deﬁnition 12 implies that, if two proper subsets of nodes S 1 , S 2 (cid:40) V are isomorphic, then their marginal node embedding distributions must be the same up to a permutation, i.e., p (( Z i ) i ∈
Revised: The permutation equivariance of p in Deﬁnition 12 implies that, if two proper subsets of nodes S 1 , S 2 (cid:40) V are isomorphic, then their marginal node embedding distributions must be the same up to a permutation, i.e., p (( Z i ) i ∈

Original: S 1 | A , X ) =
Revised: S 1 | A , X ) =

Original: π ( p (( Z j ) j ∈ S 2 | A , X )) for some appropriate permutation π .
Revised: π ( p (( Z j ) j ∈ S 2 | A , X )) for some appropriate permutation π .

Original: Proof.
Revised: Proof.

Original: From Deﬁnition 12, it is trivial to observe that two isomorphic nodes u, v ∈ V in graph
Revised: From Deﬁnition 12, it is trivial to observe that two isomorphic nodes u, v ∈ V in graph

Original: G =
Revised: G =

Original: ( A , X ) have the same marginal node embedding distributions.
Revised: ( A , X ) have the same marginal node embedding distributions.

Original: In this proof we extend this to node sets S ⊂ V where | S | > 1 .
Revised: In this proof we extend this to node sets S ⊂ V where | S | > 1 .

Original: We marginalize over ( Z i ) i (cid:54)∈ S 1 to obtain p (( Z i ) i ∈
Revised: We marginalize over ( Z i ) i (cid:54)∈ S 1 to obtain p (( Z i ) i ∈

Original: S 1 | A , X ) and in the other case over ( Z i )
Revised: S 1 | A , X ) and in the other case over ( Z i )

Original: i (cid:54)∈ S 2 to obtain p (( Z i ) i ∈
Revised: i (cid:54)∈ S 2 to obtain p (( Z i ) i ∈

Original: S 2 | A , X ) respectively.
Revised: S 2 | A , X ) respectively.

Original: We consider 2 cases as follows:
Revised: We consider 2 cases as follows:

Original: Case 1: S 1 = S 2 : This is the trivial where S 1 and S 2 are the exact same nodes, hence their marginal distributions are the identical as well by deﬁnition.
Revised: Case 1: S 1 = S 2 : This is the trivial where S 1 and S 2 are the exact same nodes, hence their marginal distributions are the identical as well by deﬁnition.

Original: Case 2: S 1 (cid:54) =
Revised: Case 2: S 1 (cid:54) =

Original: S 2 : Since S 1 and S 2 are also given to be isomorphic, it is clear to see that every node in S 1 has an isomorphic equivalent in S 2 .
Revised: S 2 : Since S 1 and S 2 are also given to be isomorphic, it is clear to see that every node in S 1 has an isomorphic equivalent in S 2 .

Original: In a graph G =
Revised: In a graph G =

Original: ( A , X ) , the above statement conveys that
Revised: ( A , X ) , the above statement conveys that

Original: S 2 can be written as a permutation π on S 1 , i.e S 2 =
Revised: S 2 can be written as a permutation π on S 1 , i.e S 2 =

Original: π ( S 1 ) .
Revised: π ( S 1 ) .

Original: Now, employing Deﬁnition 12, it is clear to see that p (( Z i ) i ∈
Revised: Now, employing Deﬁnition 12, it is clear to see that p (( Z i ) i ∈

Original: S 1 | A , X ) =
Revised: S 1 | A , X ) =

Original: π ( p (( Z j ) j ∈ S 2 | A , X ))
Revised: π ( p (( Z j ) j ∈ S 2 | A , X ))

Original: Next, we restate and prove Theorem
Revised: Next, we restate and prove Theorem

Original: Theorem 1.
Revised: Theorem 1.

Original: Let S ⊆ P ( V ) be a set of subsets of V .
Revised: Let S ⊆ P ( V ) be a set of subsets of V .

Original: Let Y ( S , A , X ) =
Revised: Let Y ( S , A , X ) =

Original: ( Y ( (cid:126)S, A , X ))
Revised: ( Y ( (cid:126)S, A , X ))

Original: S ∈S be a sequence of random variables deﬁned over the sets S ∈ S of a graph G =
Revised: S ∈S be a sequence of random variables deﬁned over the sets S ∈ S of a graph G =

Original: ( A , X ) , such that we deﬁne Y ( (cid:126)S, A , X ) :
Revised: ( A , X ) , such that we deﬁne Y ( (cid:126)S, A , X ) :

Original: =
Revised: =

Original: Y (cid:126)S | A , X and Y ( (cid:126)S 1 , A , X ) d
Revised: Y (cid:126)S | A , X and Y ( (cid:126)S 1 , A , X ) d

Original: = Y ( (cid:126)S 2 , A , X ) for any two jointly isomorphic subsets S 1 , S 2 ∈ S in ( A , X ) (Deﬁnition 7), where d = means equality in their marginal distributions.
Revised: = Y ( (cid:126)S 2 , A , X ) for any two jointly isomorphic subsets S 1 , S 2 ∈ S in ( A , X ) (Deﬁnition 7), where d = means equality in their marginal distributions.

Original: Then, there exists a deterministic function ϕ such that, Y ( S , A , X ) a.s. =
Revised: Then, there exists a deterministic function ϕ such that, Y ( S , A , X ) a.s. =

Original: ( ϕ (Γ (cid:63) ( (cid:126)S, A , X ) , (cid:15) S ))
Revised: ( ϕ (Γ (cid:63) ( (cid:126)S, A , X ) , (cid:15) S ))

Original: S ∈S , where (cid:15) S is a pure source of random noise from a joint distribution p (( (cid:15) S (cid:48) )
Revised: S ∈S , where (cid:15) S is a pure source of random noise from a joint distribution p (( (cid:15) S (cid:48) )

Original: ∀ S (cid:48) ∈S ) independent of A and X .
Revised: ∀ S (cid:48) ∈S ) independent of A and X .

Original: Proof.
Revised: Proof.

Original: The case S =
Revised: The case S =

Original: V is given in Theorem 12 of Bloem-Reddy & Teh (2019).
Revised: V is given in Theorem 12 of Bloem-Reddy & Teh (2019).

Original: The case
Revised: The case

Original: S = ∅ is trivial.
Revised: S = ∅ is trivial.

Original: The case S (cid:40) V , S (cid:54) = ∅ , is described as follows with a constructive proof.
Revised: The case S (cid:40) V , S (cid:54) = ∅ , is described as follows with a constructive proof.

Original: First consider the case of two isomorphic sets of nodes S 1 , S 2 ∈ S .
Revised: First consider the case of two isomorphic sets of nodes S 1 , S 2 ∈ S .

Original: As by deﬁnition Y ( (cid:126)S 1 , A , X ) d = Y ( (cid:126)S 2 , A , X ) , we must assume p ( Y (cid:126)S 1 | A , X ) =
Revised: As by deﬁnition Y ( (cid:126)S 1 , A , X ) d = Y ( (cid:126)S 2 , A , X ) , we must assume p ( Y (cid:126)S 1 | A , X ) =

Original: p ( Y (cid:126)S 2 | A , X ) .
Revised: p ( Y (cid:126)S 2 | A , X ) .

Original: We can now use the transfer theorem (Kallenberg, 2006, Theorem 6.10) to obtain a joint description
Revised: We can now use the transfer theorem (Kallenberg, 2006, Theorem 6.10) to obtain a joint description

Original: Y (cid:126)S 1 | A , X a.s. =
Revised: Y (cid:126)S 1 | A , X a.s. =

Original: φ 1 ( (cid:126)S 1 , A , X , (cid:15) ) and Y (cid:126)S 2 | A , X a.s. =
Revised: φ 1 ( (cid:126)S 1 , A , X , (cid:15) ) and Y (cid:126)S 2 | A , X a.s. =

Original: φ 2 ( (cid:126)S 2 , A , X , (cid:15) ) , where (cid:15) is a common source of independent noise.
Revised: φ 2 ( (cid:126)S 2 , A , X , (cid:15) ) , where (cid:15) is a common source of independent noise.

Original: As S 1 and S 2 are joint isomorphic (Deﬁnition 7), there exists an isomorphism S 1 = iso
Revised: As S 1 and S 2 are joint isomorphic (Deﬁnition 7), there exists an isomorphism S 1 = iso

Original: ( (cid:126)S 2 ) , where φ 1 ( iso ( (cid:126)S 2 ) , A , X , (cid:15) )
Revised: ( (cid:126)S 2 ) , where φ 1 ( iso ( (cid:126)S 2 ) , A , X , (cid:15) )

Original: =
Revised: =

Original: φ 1 ( (cid:126)S 1 , A , X , (cid:15) ) .
Revised: φ 1 ( (cid:126)S 1 , A , X , (cid:15) ) .

Original: Because the distribution given by φ 1 ( · , (cid:15) ) must be isomorphic-invariant in ( A , X ) and S 1 and S 2 are also isomorphic in ( A , X ) then, for all permutation actions π ∈ Π n , there exists a new isomorphism iso (cid:48)
Revised: Because the distribution given by φ 1 ( · , (cid:15) ) must be isomorphic-invariant in ( A , X ) and S 1 and S 2 are also isomorphic in ( A , X ) then, for all permutation actions π ∈ Π n , there exists a new isomorphism iso (cid:48)

Original: such that φ 1 ( π ( iso ( (cid:126)S 2 )) , π ( A ) , π ( X ) , (cid:15) )
Revised: such that φ 1 ( π ( iso ( (cid:126)S 2 )) , π ( A ) , π ( X ) , (cid:15) )

Original: d = φ 1 ( iso (cid:48) ( π ( (cid:126)S 1 )) , π ( A ) , π ( X ) , (cid:15) ) , which allows us to create a function ϕ (cid:48) that incorporates iso (cid:48) into φ 1 .
Revised: d = φ 1 ( iso (cid:48) ( π ( (cid:126)S 1 )) , π ( A ) , π ( X ) , (cid:15) ) , which allows us to create a function ϕ (cid:48) that incorporates iso (cid:48) into φ 1 .

Original: Due to the isomorphism between S 1 and S 2 , we can do the same process for S 2 to arrive at the same function ϕ (cid:48) .
Revised: Due to the isomorphism between S 1 and S 2 , we can do the same process for S 2 to arrive at the same function ϕ (cid:48) .

Original: We can now apply Corollary 6.11 (Kallenberg, 2006) over ( Y (cid:126)S 1 | A , X , Y (cid:126)S 2 | A , X ) along with a measure-preserving mapping f to show that
Revised: We can now apply Corollary 6.11 (Kallenberg, 2006) over ( Y (cid:126)S 1 | A , X , Y (cid:126)S 2 | A , X ) along with a measure-preserving mapping f to show that

Original: Y (cid:126)S 1 | A , X a.s. = ϕ (cid:48)
Revised: Y (cid:126)S 1 | A , X a.s. = ϕ (cid:48)

Original: ( (cid:126)S 1 , A , X , (cid:15) 1 ) and Y (cid:126)S 2 | A , X a.s. = ϕ (cid:48)
Revised: ( (cid:126)S 1 , A , X , (cid:15) 1 ) and Y (cid:126)S 2 | A , X a.s. = ϕ (cid:48)

Original: ( (cid:126)S 2 , A , X , (cid:15) 2 ) , where ( (cid:15) 1 , (cid:15) 2 ) =
Revised: ( (cid:126)S 2 , A , X , (cid:15) 2 ) , where ( (cid:15) 1 , (cid:15) 2 ) =

Original: f ( (cid:15) ) .
Revised: f ( (cid:15) ) .

Original: If S 1 and
Revised: If S 1 and

Original: S 2 are not joint isomorphic, we can simply deﬁne ϕ (cid:48)
Revised: S 2 are not joint isomorphic, we can simply deﬁne ϕ (cid:48)

Original: ( S i , · ) :
Revised: ( S i , · ) :

Original: =
Revised: =

Original: φ i ( S i , · ) .
Revised: φ i ( S i , · ) .

Original: Deﬁnition 11 allows us to deﬁne a function ϕ from which we rewrite ϕ (cid:48)
Revised: Deﬁnition 11 allows us to deﬁne a function ϕ from which we rewrite ϕ (cid:48)

Original: ( (cid:126)S i , A , X , (cid:15) i ) as ϕ (Γ (cid:63) ( (cid:126)S i , A , X ) , (cid:15) i ) .
Revised: ( (cid:126)S i , A , X , (cid:15) i ) as ϕ (Γ (cid:63) ( (cid:126)S i , A , X ) , (cid:15) i ) .

Original: Applying the same procedure to all S ∈ S concludes our proof.
Revised: Applying the same procedure to all S ∈ S concludes our proof.

Original: Next, we restate and prove Theorem
Revised: Next, we restate and prove Theorem

Original: Theorem 2 (The statistical equivalence between node embeddings and structural representations) .
Revised: Theorem 2 (The statistical equivalence between node embeddings and structural representations) .

Original: Let Y ( S , A , X ) =
Revised: Let Y ( S , A , X ) =

Original: ( Y ( (cid:126)S, A , X ))
Revised: ( Y ( (cid:126)S, A , X ))

Original: S ∈S be as in Theorem 1.
Revised: S ∈S be as in Theorem 1.

Original: Consider a graph G =
Revised: Consider a graph G =

Original: ( A , X ) ∈ Σ .
Revised: ( A , X ) ∈ Σ .

Original: Let Γ (cid:63) ( (cid:126)S, A , X ) be a most-expressive structural representation of nodes S ∈ S in ( A , X ) .
Revised: Let Γ (cid:63) ( (cid:126)S, A , X ) be a most-expressive structural representation of nodes S ∈ S in ( A , X ) .

Original: Then,
Revised: Then,

Original: [Equation]
Revised: [Equation]

Original: for any node embedding matrix Z that satisﬁes Deﬁnition 12, where A ⊥⊥ B C means A is independent of C given B .
Revised: for any node embedding matrix Z that satisﬁes Deﬁnition 12, where A ⊥⊥ B C means A is independent of C given B .

Original: Finally, ∀ ( A , X ) ∈ Σ , there exists a most-expressive node embedding Z (cid:63) | A , X such that,
Revised: Finally, ∀ ( A , X ) ∈ Σ , there exists a most-expressive node embedding Z (cid:63) | A , X such that,

Original: [Equation]
Revised: [Equation]

Original: for some appropriate collection of functions { f ( k ) ( · ) } k =1 ,...,n .
Revised: for some appropriate collection of functions { f ( k ) ( · ) } k =1 ,...,n .

Original: Proof.
Revised: Proof.

Original: In the ﬁrst part of the proof, for any embedding distribution p ( Z | A , X ) , we note that by
Revised: In the ﬁrst part of the proof, for any embedding distribution p ( Z | A , X ) , we note that by

Original: Theorem 1, y ( (cid:126)S, A , X ) a.s. = f (cid:48)
Revised: Theorem 1, y ( (cid:126)S, A , X ) a.s. = f (cid:48)

Original: (Γ (cid:63) ( (cid:126)S, A , X ) , (cid:15) S ) .
Revised: (Γ (cid:63) ( (cid:126)S, A , X ) , (cid:15) S ) .

Original: Hence, Y ( (cid:126)S, A , X ) ⊥⊥ Γ (cid:63) ( (cid:126)S, A , X ) Z | A , X , ∀ S ∈
Revised: Hence, Y ( (cid:126)S, A , X ) ⊥⊥ Γ (cid:63) ( (cid:126)S, A , X ) Z | A , X , ∀ S ∈

Original: S , is a direct consequence of Proposition 6.13 in (Kallenberg, 2006).
Revised: S , is a direct consequence of Proposition 6.13 in (Kallenberg, 2006).

Original: In the second part of the proof, we construct an orbit over a most-expressive representation of a graph ( A , X ) of size n , with permutations that act only on unique node ids (node orderings)
Revised: In the second part of the proof, we construct an orbit over a most-expressive representation of a graph ( A , X ) of size n , with permutations that act only on unique node ids (node orderings)

Original: added as node features: Π (cid:48)
Revised: added as node features: Π (cid:48)

Original: ( A , X ) =
Revised: ( A , X ) =

Original: { ((Γ (cid:63) ( v, A , [ X , π (1 , . . . , n ) T ]))
Revised: { ((Γ (cid:63) ( v, A , [ X , π (1 , . . . , n ) T ]))

Original: ∀ v ∈ V } ∀ π ∈ Π n , where [ A, b ] concatenates column vector b as a column of matrix A .
Revised: ∀ v ∈ V } ∀ π ∈ Π n , where [ A, b ] concatenates column vector b as a column of matrix A .

Original: Deﬁne Z (cid:63) | A , X as the random variable with a uniform measure over the set Π (cid:48)
Revised: Deﬁne Z (cid:63) | A , X as the random variable with a uniform measure over the set Π (cid:48)

Original: ( A , X ) .
Revised: ( A , X ) .

Original: We ﬁrst prove that Z (cid:63) | A , X is a most-expressive node embedding.
Revised: We ﬁrst prove that Z (cid:63) | A , X is a most-expressive node embedding.

Original: Clearly, Z (cid:63) | A , X is a node embedding, since the uniform measure over Π (cid:48)
Revised: Clearly, Z (cid:63) | A , X is a node embedding, since the uniform measure over Π (cid:48)

Original: ( A , X ) is
Revised: ( A , X ) is

Original: G -equivariant.
Revised: G -equivariant.

Original: All that is left to show is that we can construct Γ (cid:63) of any-size subset S ∈ S from
Revised: All that is left to show is that we can construct Γ (cid:63) of any-size subset S ∈ S from

Original: Z (cid:63) | A , X via Γ
Revised: Z (cid:63) | A , X via Γ

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: [Equation]
Revised: [Equation]

Original: for some function f ( | S | ) .
Revised: for some function f ( | S | ) .

Original: This part of the proof has a constructive argument and comes in two parts.
Revised: This part of the proof has a constructive argument and comes in two parts.

Original: Assume S ∈ S has no other joint isomorphic set of nodes in S , i.e., (cid:64) S 2 ∈ S such that S and S 2 are joint isomorphic in ( A , X ) .
Revised: Assume S ∈ S has no other joint isomorphic set of nodes in S , i.e., (cid:64) S 2 ∈ S such that S and S 2 are joint isomorphic in ( A , X ) .

Original: For any such subset of nodes S ∈ S , and any element R π ∈ Π (cid:48)
Revised: For any such subset of nodes S ∈ S , and any element R π ∈ Π (cid:48)

Original: ( A , X ) , there is a bijective measurable map between the nodes in S and their positions in the representation vector R π
Revised: ( A , X ) , there is a bijective measurable map between the nodes in S and their positions in the representation vector R π

Original: =
Revised: =

Original: (Γ (cid:63) ( v, A , [ X , π (1 , . . . , n ) T ])) ∀ v ∈ V , since all node features are unique and there are no isomorphic nodes under such conditions.
Revised: (Γ (cid:63) ( v, A , [ X , π (1 , . . . , n ) T ])) ∀ v ∈ V , since all node features are unique and there are no isomorphic nodes under such conditions.

Original: Consider the multiset
Revised: Consider the multiset

Original: [Equation]
Revised: [Equation]

Original: of the representations restricted to the set S .
Revised: of the representations restricted to the set S .

Original: We now show that there exists an surjection between O S ( A , X ) and Γ (cid:63) ( (cid:126)S, A , X ) .
Revised: We now show that there exists an surjection between O S ( A , X ) and Γ (cid:63) ( (cid:126)S, A , X ) .

Original: There is a surjection if for all S 1 , S 2 ∈ P (cid:63) ( V ) that are non-isomorphic, it implies O S 1 ( A , X ) (cid:54) =
Revised: There is a surjection if for all S 1 , S 2 ∈ P (cid:63) ( V ) that are non-isomorphic, it implies O S 1 ( A , X ) (cid:54) =

Original: O S 2 ( A , X ) .
Revised: O S 2 ( A , X ) .

Original: The condition is trivial if | S 1 | (cid:54) =
Revised: The condition is trivial if | S 1 | (cid:54) =

Original: | S 2 | as |O S 1 ( A , X ) | (cid:54) =
Revised: | S 2 | as |O S 1 ( A , X ) | (cid:54) =

Original: |O S 2 ( A , X ) | .
Revised: |O S 2 ( A , X ) | .

Original: If | S 1 | = | S 2 | , we prove by contradiction.
Revised: If | S 1 | = | S 2 | , we prove by contradiction.

Original: Assume O S 1 ( A , X ) =
Revised: Assume O S 1 ( A , X ) =

Original: O S 2 ( A , X ) .
Revised: O S 2 ( A , X ) .

Original: Because of the unique feature ids and because Γ (cid:63) is most-expressive, the representation Γ (cid:63) ( v, A , [ X , π (1 , . . . , n ) T ]) of node v ∈ V and permutation π ∈ Π n is unique.
Revised: Because of the unique feature ids and because Γ (cid:63) is most-expressive, the representation Γ (cid:63) ( v, A , [ X , π (1 , . . . , n ) T ]) of node v ∈ V and permutation π ∈ Π n is unique.

Original: As S 1 is not isomorphic to S 2 , and both sets have the same size, there must be at least one node u ∈ S 1 that has no isomorphic equivalent in S 2 .
Revised: As S 1 is not isomorphic to S 2 , and both sets have the same size, there must be at least one node u ∈ S 1 that has no isomorphic equivalent in S 2 .

Original: Hence, there exists π ∈ Π n that gives a unique representation Γ (cid:63) ( u, A , [ X , π (1 , . .
Revised: Hence, there exists π ∈ Π n that gives a unique representation Γ (cid:63) ( u, A , [ X , π (1 , . .

Original: . , n ) T ]) that does not have a matching Γ (cid:63) ( v, A , [ X , π (1 , . .
Revised: . , n ) T ]) that does not have a matching Γ (cid:63) ( v, A , [ X , π (1 , . .

Original: . , n ) T ]) for any v ∈ S 2 and π (cid:48) ∈ Π n .
Revised: . , n ) T ]) for any v ∈ S 2 and π (cid:48) ∈ Π n .

Original: Therefore, ∃ a ∈ O S 1 ( A , X ) , where a (cid:54)∈ O S 2 ( A , X ) , which is a contradiction since we assumed O S 1 ( A , X ) =
Revised: Therefore, ∃ a ∈ O S 1 ( A , X ) , where a (cid:54)∈ O S 2 ( A , X ) , which is a contradiction since we assumed O S 1 ( A , X ) =

Original: O S 2 ( A , X ) .
Revised: O S 2 ( A , X ) .

Original: Now that we know there is such a surjection, a possible surjective measurable map between
Revised: Now that we know there is such a surjection, a possible surjective measurable map between

Original: O S ( A , X ) and Γ (cid:63) ( (cid:126)S, A , X ) is a multiset function that takes O S ( A , X ) and outputs Γ (cid:63) ( (cid:126)S, A , X ) .
Revised: O S ( A , X ) and Γ (cid:63) ( (cid:126)S, A , X ) is a multiset function that takes O S ( A , X ) and outputs Γ (cid:63) ( (cid:126)S, A , X ) .

Original: For ﬁnite multisets whose elements are real numbers R , Wagstaff et al.
Revised: For ﬁnite multisets whose elements are real numbers R , Wagstaff et al.

Original: (2019) shows that a mostexpressive multiset function can be deﬁned as the average of a function f ( | S | ) over the multiset.
Revised: (2019) shows that a mostexpressive multiset function can be deﬁned as the average of a function f ( | S | ) over the multiset.

Original: The elements of O S ( A , X ) are ﬁnite ordered sequences (ordered according to the permutation) and, thus, can be uniquely (bijectively) mapped to the real line with a measurable map, even when A and
Revised: The elements of O S ( A , X ) are ﬁnite ordered sequences (ordered according to the permutation) and, thus, can be uniquely (bijectively) mapped to the real line with a measurable map, even when A and

Original: X have edge and node attributes deﬁned over the real numbers R .
Revised: X have edge and node attributes deﬁned over the real numbers R .

Original: Thus, by Wagstaff et al.
Revised: Thus, by Wagstaff et al.

Original: (2019), there exists some surjective function f ( | S | ) whose average over O S ( A , X ) give Γ (cid:63) ( (cid:126)S, A , X ) .
Revised: (2019), there exists some surjective function f ( | S | ) whose average over O S ( A , X ) give Γ (cid:63) ( (cid:126)S, A , X ) .

Original: Now assume S 1 , S 2 ⊆
Revised: Now assume S 1 , S 2 ⊆

Original: V are joint isomorphic in ( A , X ) , S 1 , S 2 (cid:54) = ∅ .
Revised: V are joint isomorphic in ( A , X ) , S 1 , S 2 (cid:54) = ∅ .

Original: Then, we have concluded that O S 1 ( A , X ) =
Revised: Then, we have concluded that O S 1 ( A , X ) =

Original: O S 2 ( A , X ) .
Revised: O S 2 ( A , X ) .

Original: Fortunately, by Deﬁnition 10, this non-uniqueness is a required property of the structural representations of Γ (cid:63) ( S 1 , A , X ) and Γ (cid:63) ( S 2 , A , X ) , which must satisfy Γ (cid:63) ( S 1 , A , X ) =
Revised: Fortunately, by Deﬁnition 10, this non-uniqueness is a required property of the structural representations of Γ (cid:63) ( S 1 , A , X ) and Γ (cid:63) ( S 2 , A , X ) , which must satisfy Γ (cid:63) ( S 1 , A , X ) =

Original: Γ (cid:63) ( S 2 , A , X ) if S 1 and S 2 are joint isomorphic, which concludes our proof.
Revised: Γ (cid:63) ( S 2 , A , X ) if S 1 and S 2 are joint isomorphic, which concludes our proof.

Original: Next, we restate and prove Corollary
Revised: Next, we restate and prove Corollary

Original: Corollary 1.
Revised: Corollary 1.

Original: The node embeddings in Deﬁnition 12 encompass embeddings given by matrix and tensor factorization methods —such as Singular Value Decomposition (SVD), Non-negative Matrix
Revised: The node embeddings in Deﬁnition 12 encompass embeddings given by matrix and tensor factorization methods —such as Singular Value Decomposition (SVD), Non-negative Matrix

Original: Factorization (NMF), implicit matrix factorization (a.k.a. word2vec)–, latent embeddings given by
Revised: Factorization (NMF), implicit matrix factorization (a.k.a. word2vec)–, latent embeddings given by

Original: Bayesian graph models —such as Probabilistic Matrix Factorizations (PMFs) and variants—, variational autoencoder methods and graph neural networks that use random lighthouses to extract node embedddings.
Revised: Bayesian graph models —such as Probabilistic Matrix Factorizations (PMFs) and variants—, variational autoencoder methods and graph neural networks that use random lighthouses to extract node embedddings.

Original: Proof.
Revised: Proof.

Original: In Probabilistic Matrix Factorization (Mnih & Salakhutdinov, 2008), we have A uv ∼
Revised: In Probabilistic Matrix Factorization (Mnih & Salakhutdinov, 2008), we have A uv ∼

Original: N ( Z Tu Z v , σ 2 a I ) where Z u ∼ N (0 , σ 2 I ) , Z v ∼ N (0 , σ 2 I ) .
Revised: N ( Z Tu Z v , σ 2 a I ) where Z u ∼ N (0 , σ 2 I ) , Z v ∼ N (0 , σ 2 I ) .

Original: We note that the posterior of P ( Z | A ) is clearly equivariant, satisfying deﬁnition 12, as a permutation action on the nodes requires the same permutation on the σ 2 I matrix as well to obtain Z .
Revised: We note that the posterior of P ( Z | A ) is clearly equivariant, satisfying deﬁnition 12, as a permutation action on the nodes requires the same permutation on the σ 2 I matrix as well to obtain Z .

Original: The proof for Poisson Matrix Factorization (Gopalan et al., 2014a;b) follows a similar construction to the above, wherein the Normal assumption is replaced by the Poisson distribution.
Revised: The proof for Poisson Matrix Factorization (Gopalan et al., 2014a;b) follows a similar construction to the above, wherein the Normal assumption is replaced by the Poisson distribution.

Original: Moreover, any matrix factorization algorithm gives an equivariant distribution of embeddings if the input matrices are randomly permuted upon input.
Revised: Moreover, any matrix factorization algorithm gives an equivariant distribution of embeddings if the input matrices are randomly permuted upon input.

Original: Speciﬁcally, any Singular Value Decomposition (SVD) method satisﬁes Deﬁnition 12 as the distribution of the eigenvector solutions to degenerate singular values —which are invariant to unitary rotations in the corresponding degenerate eigenspace— will trivially be G -equivariant even if the algorithm itself outputs values dependent on the node ids.
Revised: Speciﬁcally, any Singular Value Decomposition (SVD) method satisﬁes Deﬁnition 12 as the distribution of the eigenvector solutions to degenerate singular values —which are invariant to unitary rotations in the corresponding degenerate eigenspace— will trivially be G -equivariant even if the algorithm itself outputs values dependent on the node ids.

Original: Same is true for non-negative matrix factorization (Lee & Seung, 2001) and implicit matrix factorization (Levy & Goldberg, 2014; Mikolov et al., 2013).
Revised: Same is true for non-negative matrix factorization (Lee & Seung, 2001) and implicit matrix factorization (Levy & Goldberg, 2014; Mikolov et al., 2013).

Original: PGNN’s
Revised: PGNN’s

Original: (You et al., 2019) compute the shortest distances between every node of the graph with a predetermined set of ‘anchor’ nodes to encode a distance metric.
Revised: (You et al., 2019) compute the shortest distances between every node of the graph with a predetermined set of ‘anchor’ nodes to encode a distance metric.

Original: By deﬁnition, using such a distance metric would make the node embeddings learned by this technique G -equivariant.
Revised: By deﬁnition, using such a distance metric would make the node embeddings learned by this technique G -equivariant.

Original: The shortest path between all pairs of nodes in a graph can be seen equivalently as a function of a polynomial in A k .
Revised: The shortest path between all pairs of nodes in a graph can be seen equivalently as a function of a polynomial in A k .

Original: Alternatively, this can also be represented using the adjacency matrix and computed using the FloydWarshall algorithm (Cormen et al., 2009).
Revised: Alternatively, this can also be represented using the adjacency matrix and computed using the FloydWarshall algorithm (Cormen et al., 2009).

Original: The shortest distance is thus a function of A ignoring the node features X .
Revised: The shortest distance is thus a function of A ignoring the node features X .

Original: Since the inputs to the GNN comprises of the distance metric, A and X , the node embeddings Z can equivalently seen as a function of A , X and noise.
Revised: Since the inputs to the GNN comprises of the distance metric, A and X , the node embeddings Z can equivalently seen as a function of A , X and noise.

Original: The noise in this case is characterized by the randomized anchor set selection.
Revised: The noise in this case is characterized by the randomized anchor set selection.

Original: In variational auto-encoder models such as CVAE’s, GVAE’s, Graphite (Tang et al., 2019;
Revised: In variational auto-encoder models such as CVAE’s, GVAE’s, Graphite (Tang et al., 2019;

Original: Kipf &
Revised: Kipf &

Original: Welling, 2016b; Grover et al., 2019)
Revised: Welling, 2016b; Grover et al., 2019)

Original: the latent representations Z are learned either via a mean ﬁeld approximation or are sampled independently of each other i.e. Z ∼ P ( ·| A , X ) .
Revised: the latent representations Z are learned either via a mean ﬁeld approximation or are sampled independently of each other i.e. Z ∼ P ( ·| A , X ) .

Original: We note that in the case of the mean ﬁeld approximation, the probability distribution is a Dirac Delta.
Revised: We note that in the case of the mean ﬁeld approximation, the probability distribution is a Dirac Delta.

Original: It is clear to see that the Z learned in this case is G -equivariant with respect to any permutation action of the nodes in the graph.
Revised: It is clear to see that the Z learned in this case is G -equivariant with respect to any permutation action of the nodes in the graph.

Original: Next, we restate and prove Corollary
Revised: Next, we restate and prove Corollary

Original: Corollary 2.
Revised: Corollary 2.

Original: The link prediction task between any two nodes u, v ∈ V depends only on the mostexpressive tuple representation Γ (cid:63) (( u, v ) , A , X ) .
Revised: The link prediction task between any two nodes u, v ∈ V depends only on the mostexpressive tuple representation Γ (cid:63) (( u, v ) , A , X ) .

Original: Moreover, Γ (cid:63) (( u, v ) , A , X ) always exists for any graph ( A , X ) and nodes ( u, v ) .
Revised: Moreover, Γ (cid:63) (( u, v ) , A , X ) always exists for any graph ( A , X ) and nodes ( u, v ) .

Original: Finally, given most-expressive node embeddings Z (cid:63) , there exists a function f such that Γ (cid:63) (( u, v ) , A , X ) =
Revised: Finally, given most-expressive node embeddings Z (cid:63) , there exists a function f such that Γ (cid:63) (( u, v ) , A , X ) =

Original: E Z (cid:63) [ f ( Z (cid:63)u , Z (cid:63)v )] , ∀ u, v .
Revised: E Z (cid:63) [ f ( Z (cid:63)u , Z (cid:63)v )] , ∀ u, v .

Original: Proof.
Revised: Proof.

Original: It is a consequence of Corollary 3 with | S | = 2 .
Revised: It is a consequence of Corollary 3 with | S | = 2 .

Original: Next, we restate and prove Corollary
Revised: Next, we restate and prove Corollary

Original: Corollary 3.
Revised: Corollary 3.

Original: Sample Z according to Deﬁnition 12.
Revised: Sample Z according to Deﬁnition 12.

Original: Then, we can learn a k -node structural representation of a subset of k nodes S ⊆ V , | S |
Revised: Then, we can learn a k -node structural representation of a subset of k nodes S ⊆ V , | S |

Original: = k , simply by learning a function f ( k ) whose average Γ( (cid:126)S, A , X ) =
Revised: = k , simply by learning a function f ( k ) whose average Γ( (cid:126)S, A , X ) =

Original: E [ f ( k ) (( Z v ) v ∈ S )] can be used to accurately predict Y ( (cid:126)S, A , X ) .
Revised: E [ f ( k ) (( Z v ) v ∈ S )] can be used to accurately predict Y ( (cid:126)S, A , X ) .

Original: Proof.
Revised: Proof.

Original: This proof is a direct application of Theorem 2 which shows the statistical equivalence between node embeddings and strucutral representations.
Revised: This proof is a direct application of Theorem 2 which shows the statistical equivalence between node embeddings and strucutral representations.

Original: Note that f ( k ) (( Z v ) v ∈ S ) can equivalently be represented as f ( k ) ( ϕ (Γ( v, A , X ) v ∈ S , (cid:15) S )) using Theorem 2 and that the noise (cid:15) S is marginalized from the noise distribution of Theorem 1, still preserving equivariance.
Revised: Note that f ( k ) (( Z v ) v ∈ S ) can equivalently be represented as f ( k ) ( ϕ (Γ( v, A , X ) v ∈ S , (cid:15) S )) using Theorem 2 and that the noise (cid:15) S is marginalized from the noise distribution of Theorem 1, still preserving equivariance.

Original: With an assumption of the most powerful f (cid:48) ( k ) , which is able to capture dependencies within the node set (Murphy et al., 2018) and noise (cid:15) S , we can replace the above with f (cid:48)
Revised: With an assumption of the most powerful f (cid:48) ( k ) , which is able to capture dependencies within the node set (Murphy et al., 2018) and noise (cid:15) S , we can replace the above with f (cid:48)

Original: ( k ) ( ϕ (Γ( S, A , X ) , (cid:15) S )) and subsequently compute an expectation over this function to eliminate the noise.
Revised: ( k ) ( ϕ (Γ( S, A , X ) , (cid:15) S )) and subsequently compute an expectation over this function to eliminate the noise.

Original: Next, we restate and prove Corollary
Revised: Next, we restate and prove Corollary

Original: Corollary 4.
Revised: Corollary 4.

Original: Transductive and inductive learning are unrelated to the concepts of node embeddings and structural representations.
Revised: Transductive and inductive learning are unrelated to the concepts of node embeddings and structural representations.

Original: Proof.
Revised: Proof.

Original: By Theorem 2, we can build most-expressive any-size joint representations from node embeddings, and we can get node embeddings from any-size most-expressive joint representations.
Revised: By Theorem 2, we can build most-expressive any-size joint representations from node embeddings, and we can get node embeddings from any-size most-expressive joint representations.

Original: X i Z i  ij i ∈ [1, n ] i ∈ [1, n ] i,j ∈ [1, n ]

Original: Hence, given enough computational resources, node embeddings and graph representations can have the same generalization performance over any tasks.
Revised: Hence, given enough computational resources, node embeddings and graph representations can have the same generalization performance over any tasks.

Original: This shows they are unrelated with the concepts of transductive and inductive learning.
Revised: This shows they are unrelated with the concepts of transductive and inductive learning.

Original: Next, we restate and prove Corollary
Revised: Next, we restate and prove Corollary

Original: Corollary 5.
Revised: Corollary 5.

Original: A node embeddings sampling scheme can increase the structural representation power of GNNs.
Revised: A node embeddings sampling scheme can increase the structural representation power of GNNs.

Original: Proof.
Revised: Proof.

Original: The proof follows as a direct consequence of Theorem 2, along with Murphy et al.
Revised: The proof follows as a direct consequence of Theorem 2, along with Murphy et al.

Original: (2019) which demonstrates RP-GNN as a concrete method to do so.
Revised: (2019) which demonstrates RP-GNN as a concrete method to do so.

Original: More speciﬁcally, appending unique node ids to node features uniformly at random, makes the nodes unique, and can be seen as a strategy to obtain node embeddings which satisfy Deﬁnition 12 using GNN’s.
Revised: More speciﬁcally, appending unique node ids to node features uniformly at random, makes the nodes unique, and can be seen as a strategy to obtain node embeddings which satisfy Deﬁnition 12 using GNN’s.

Original: By averaging over multiple such node embeddings gives us structural representations more powerful than that of standalone
Revised: By averaging over multiple such node embeddings gives us structural representations more powerful than that of standalone

Original: GNN’s.
Revised: GNN’s.

Revised:  C OLLIDING G RAPH N EURAL N ETWORKS (CGNN S )

Revised: In this section we propose a new variational auto-encoder procedure to obtain node embeddings using neural networks, denoted Colliding Graph Neural Networks (CGNNs).

Revised: Our sole reason to propose a new auto-encoder method is because we want to test the expressiveness of node embedding auto-encoders —and, unfortunately, existing auto-encoders, such as Grover et al.

Revised: (2019), do not properly account for the dependencies introduced by the colliding variables in the graphical model.

Original:  CGNN A LGORITHMS
Revised: In our experiments, shown later, we aggregate multiple node embedding sampled from CGNN to obtain structural representations of the corresponding nodes and node sets. Node Embedding Auto-encoder. In CGNN’s, we adopt a latent variable approach to learn node embeddings. Corresponding to each evidence feature vector X

Revised: i, · ∈ R k ∀ i ∈ V , we introduce a latent variable Z i, · ∈ R k .

Revised: In addition, our graphical model also consists of observed variables

Revised: A i,j, · ∈ R d ∀ i, j ∈ V .

Revised: These are related through the joint distribution p

Revised: ( A , X | Z )

Revised: = (cid:81) i,j ∈ V × V p

Revised: ( A i,j, · | Z i, · , Z j, · ) (cid:81)

Revised: h ∈ V p

Revised: ( X h, · | Z h, · ) , which is summarized by the Bayesian network in Figure 5 in the Appendix.

Revised: Note that A i,j, · is a collider, since it is observed and inﬂuenced by two hidden variables, Z i, · , Z j, · .

Revised: A neural network is used to learn the joint probability via MCMC, in an unsupervised fashion.

Revised: The model learns the parameters of the MCMC transition kernel via an unrolled Gibbs Sampler, a templated recurrent model (an MLP with shared weights across Gibbs sampling steps), partially inspired by Fan & Huang (2017).

Revised: The unrolled Gibbs Sampler, starts with a normal distribution of the latent variables Z (0) i, · , ∀ i ∈ V, with each Z (0) i, · ∼ N ( 0 , I ) independently, where I is the identity matrix.

Revised: Subsequently at time steps t = 1 , 2 , . .

Revised: . , in accordance with the graphical model, each variable Z ( t ) i, · is sequentially sampled from its true distribution conditioned on all observed edges of its corresponding node i , in addition to the most-up-to-date latent variables

Revised: Z ’s associated with its immediate neighbors.

Revised: The reparametrization trick (Kingma & Welling, 2013) allows us to backpropogate through the unrolled

Revised: Gibbs Sampler.

Revised: Algorithm 1 in the Appendix details our method.

Revised: Consequently, this procedure has an effect on the run-time of this technique, which we alleviate by performing Parallel Gibbs

Revised: Sampling by constructing parallel splashes (Gonzalez et al., 2011).

Revised: Our unsupervised objective is reconstructing the noisy adjacency matrix.

Revised:  CGNN A LGORITHMS

Original: The procedure to generate node embeddings used by the CGNN is given by Algorithm 1.
Revised: The procedure to generate node embeddings used by the CGNN is given by Algorithm 1.

Original: Structural representations are computed as an unbiased estimate of the expected value of a function of the node embedding samples as given by Algorithm 2 via a set of sets function (Meng et al., 2019).
Revised: Structural representations are computed as an unbiased estimate of the expected value of a function of the node embedding samples as given by Algorithm 2 via a set of sets function (Meng et al., 2019).

Original: input : A , X , num-times output: Z initialization: Z u ∼ N (0 , 1) ∀ u ∈ V while num-times > 0 do for u ∈ V do ∀ v ∈ V such that A uv =hidden ← f ( { Z v } ) ; // f is a permutation invariant function visible ← g ( { X v } ) ; // g is a permutation invariant function
Revised: input : A , X , num-times output: Z initialization: Z u ∼ N (0 , 1) ∀ u ∈ V while num-times > 0 do for u ∈ V do ∀ v ∈ V such that A uv =hidden ← f ( { Z v } ) ; // f is a permutation invariant function visible ← g ( { X v } ) ; // g is a permutation invariant function

Original: Z u ← MLP (hidden, visible, X u ) + Noise // With Reparametrization Trick //
Revised: Z u ← MLP (hidden, visible, X u ) + Noise // With Reparametrization Trick //

Original: Equivalently, Z u ∼ P ( ·|{ Z v } , { X v } , { A uv } , X u ) end num-times ← num-times - 1 end 
Revised: Equivalently, Z u ∼ P ( ·|{ Z v } , { X v } , { A uv } , X u ) end num-times ← num-times - 1 end [Figure]

Original:  F URTHER R ESULTS
Revised:  F URTHER R ESULTS

Original: In Table 2 we provide the results on node classiﬁcation, link prediction and triad prediction on the
Revised: In Table 2 we provide the results on node classiﬁcation, link prediction and triad prediction on the

Original: Citeseer dataset.
Revised: Citeseer dataset.

Original: input : { Z ( i ) } mi =1 , k ; // node embedding samples, node set size output: g ( { Z } S );

Original: S =

Original: { S 1 } ∀ S 1 ⊂ V : | S 1 | =

Original: k // structural representations, S is a set of sets initialization: g ( { Z } S )

Original: = { (cid:126) 0 } for i ∈ [1 , m ] do for S ∈ S do g

Original: ( { Z u } u ∈ S )

Original: ← g

Original: ( { Z u } u ∈ S )

Original: + 1 m f

Original: ( { Z ( i ) u } u ∈ S ) //

Original: f is a permutation invariant function end end

Original: [Table]

Original:  D ESCRIPTION OF D ATASETS AND E XPERIMENTAL S ETUP
Revised:  D ESCRIPTION OF D ATASETS AND E XPERIMENTAL S ETUP

Original: A detailed description of the datasets and the splits are given in Table 3.
Revised: A detailed description of the datasets and the splits are given in Table 3.

Original: Our implementation is in
Revised: Our implementation is in

Original: PyTorch using Python 3.6.
Revised: PyTorch using Python 3.6.

Original: The implementations for GIN and RP-GIN are done using the PyTorch
Revised: The implementations for GIN and RP-GIN are done using the PyTorch

Original: Geometric Framework.
Revised: Geometric Framework.

Original: We used two convolutional layers for GIN, RP-GIN since it had the best performance in our tasks (we had tested with 2/3/4/5 convolutional layers).
Revised: We used two convolutional layers for GIN, RP-GIN since it had the best performance in our tasks (we had tested with 2/3/4/5 convolutional layers).

Original: Also since we perform tasks based on node representations rather than graph representations, we ignore the graph wide readout.
Revised: Also since we perform tasks based on node representations rather than graph representations, we ignore the graph wide readout.

Original: For GIN and RP-GIN, the embedding dimension was set to 256 at both convolutional layers.
Revised: For GIN and RP-GIN, the embedding dimension was set to 256 at both convolutional layers.

Original: All MLPS, across all models have 256 neurons.
Revised: All MLPS, across all models have 256 neurons.

Original: Optimization is performed with the Adam
Revised: Optimization is performed with the Adam

Original: Optimizer (Kingma & Ba, 2014).
Revised: Optimizer (Kingma & Ba, 2014).

Original: For the GIN, RP-GIN the learning rate was tuned in { 0.01, 0.001, 0.0001, 0.00001 } whereas for CGNN’s the learning rate was set to 0.001.
Revised: For the GIN, RP-GIN the learning rate was tuned in { 0.01, input : { Z ( i ) } mi =1 , k ; // node embedding samples, node set size output: g ( { Z } S ); S = { S 1 } ∀ S 1 ⊂ V : | S 1 | =

Original: Training was performed on Titan V GPU’s.
Revised: k // structural representations, S is a set of sets initialization: g ( { Z } S ) = { (cid:126) 0 } for i ∈ [1 , m ] do for S ∈ S do g ( { Z u } u ∈ S ) ← g

Original: For more details refer to the code provided.
Revised: ( { Z u } u ∈ S ) + 1 m f ( { Z ( i ) u } u ∈ S ) // f is a permutation invariant function end end

Original: [Table]
Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]