Original: G RAPH S TRUCTURAL A GGREGATION FOR E XPLAINABLE L EARNING
Revised: G RAPH S TRUCTURAL A GGREGATION FOR E XPLAINABLE L EARNING

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Graph neural networks have proven to be very efﬁcient to solve several tasks in graphs such as node classiﬁcation or link prediction.
Revised: Graph neural networks have proven to be very efﬁcient to solve several tasks in graphs such as node classiﬁcation or link prediction.

Original: These algorithms that operate by propagating information from vertices to their neighbors allow one to build node embeddings that contain local information.
Revised: These algorithms that operate by propagating information from vertices to their neighbors allow one to build node embeddings that contain local information.

Original: In order to use graph neural networks for graph classiﬁcation, node embeddings must be aggregated to obtain a graph representation able to discriminate among different graphs (of possibly various sizes).
Revised: In order to use graph neural networks for graph classiﬁcation, node embeddings must be aggregated to obtain a graph representation able to discriminate among different graphs (of possibly various sizes).

Original: Moreover, in analogy to neural networks for image classiﬁcation, there is a need for explainability regarding the features that are selected in the graph classiﬁcation process.
Revised: Moreover, in analogy to neural networks for image classiﬁcation, there is a need for explainability regarding the features that are selected in the graph classiﬁcation process.

Original: To this end, we introduce StructAgg, a simple yet effective aggregation process based on the identiﬁcation of structural roles for nodes in graphs that we use to create an end-to-end model.
Revised: To this end, we introduce StructAgg, a simple yet effective aggregation process based on the identiﬁcation of structural roles for nodes in graphs that we use to create an end-to-end model.

Original: Through extensive experiments we show that this architecture can compete with state-of-the-art methods.
Revised: Through extensive experiments we show that this architecture can compete with state-of-the-art methods.

Original: We show how this aggregation step allows us to cluster together nodes that have comparable structural roles and how these roles provide explainability to this neural network model.
Revised: We show how this aggregation step allows us to cluster together nodes that have comparable structural roles and how these roles provide explainability to this neural network model.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Convolution neural networks (LeCun et al., 1995) have proven to be very efﬁcient at learning meaningful patterns for many articiﬁcial intelligence tasks.
Revised: Convolution neural networks (LeCun et al., 1995) have proven to be very efﬁcient at learning meaningful patterns for many articiﬁcial intelligence tasks.

Original: They convey the ability to learn hierarchical information in data with Euclidean grid-like structures such as images and text.
Revised: They convey the ability to learn hierarchical information in data with Euclidean grid-like structures such as images and text.

Original: Convolutional Neural
Revised: Convolutional Neural

Original: Networks (CNNs) have rapidly become state-of-the art methods in the ﬁelds of computer vision (Russakovsky et al., 2015) and natural language processing (Devlin et al., 2018).
Revised: Networks (CNNs) have rapidly become state-of-the art methods in the ﬁelds of computer vision (Russakovsky et al., 2015) and natural language processing (Devlin et al., 2018).

Original: However in many scientiﬁc ﬁelds, studied data have an underlying graph or manifold structure such as communication networks (whether social or technical) or knowledge graphs.
Revised: However in many scientiﬁc ﬁelds, studied data have an underlying graph or manifold structure such as communication networks (whether social or technical) or knowledge graphs.

Original: Recently there have been many attempts to extend convolutions to those non-Euclidean structured data (Hammond et al., 2011; Kipf & Welling, 2016; Defferrard et al., 2016).
Revised: Recently there have been many attempts to extend convolutions to those non-Euclidean structured data (Hammond et al., 2011; Kipf & Welling, 2016; Defferrard et al., 2016).

Original: In these new approaches, the authors propose to compute node embeddings in a semi-supervised fashion in order to perform node classiﬁcation.
Revised: In these new approaches, the authors propose to compute node embeddings in a semi-supervised fashion in order to perform node classiﬁcation.

Original: Those node embeddings can also be used for link prediction by computing distances between each node of the graph (Hammond et al., 2011; Kipf & Welling, 2016).
Revised: Those node embeddings can also be used for link prediction by computing distances between each node of the graph (Hammond et al., 2011; Kipf & Welling, 2016).

Original: Graph classiﬁcation is studied in many ﬁelds.
Revised: Graph classiﬁcation is studied in many ﬁelds.

Original: Whether for predicting the chemical activity of a molecule or to cluster authors from different scientiﬁc domains based on their ego-networks (Freeman, 1982).
Revised: Whether for predicting the chemical activity of a molecule or to cluster authors from different scientiﬁc domains based on their ego-networks (Freeman, 1982).

Original: However when trying to generalize neural network approaches to the task of graph classiﬁcation there are several aspects that differ widely from image classiﬁcation.
Revised: However when trying to generalize neural network approaches to the task of graph classiﬁcation there are several aspects that differ widely from image classiﬁcation.

Original: When trying to perform graph classiﬁcation, we can deal with graphs of different sizes.
Revised: When trying to perform graph classiﬁcation, we can deal with graphs of different sizes.

Original: To compare them we ﬁrst need to obtain a graph representation that is independant of the size of the graph.
Revised: To compare them we ﬁrst need to obtain a graph representation that is independant of the size of the graph.

Original: Moreover, for a ﬁxed graph, nodes are not ordered.
Revised: Moreover, for a ﬁxed graph, nodes are not ordered.

Original: The graph representation obtained with neural network algorithms must be independant of the order of nodes and thus be invariant by node permutation.
Revised: The graph representation obtained with neural network algorithms must be independant of the order of nodes and thus be invariant by node permutation.

Original: Aggregation functions are functions that operate on node embeddings to produce a graph representation.
Revised: Aggregation functions are functions that operate on node embeddings to produce a graph representation.

Original: When tackling a graph classiﬁcation task, the aggregation function used is usually just a mean or a max of node embeddings as illustrated in ﬁgure 1b.
Revised: When tackling a graph classiﬁcation task, the aggregation function used is usually just a mean or a max of node embeddings as illustrated in ﬁgure 1b.

Original: But when working with graphs of large sizes, the mean over all nodes does not allow us to extract signiﬁcant patterns with a good discriminating power.
Revised: But when working with graphs of large sizes, the mean over all nodes does not allow us to extract signiﬁcant patterns with a good discriminating power.

Original: In order to identify patterns in graphs, some methods try to identify structural roles for nodes.
Revised: In order to identify patterns in graphs, some methods try to identify structural roles for nodes.

Original: Donnat et al.
Revised: Donnat et al.

Original: (2018) deﬁne structural role discovery as the process of identifying nodes which have topologically similar network neighborhoods while residing in potentially distant areas of the network as illustrated in ﬁgure 1a.
Revised: (2018) deﬁne structural role discovery as the process of identifying nodes which have topologically similar network neighborhoods while residing in potentially distant areas of the network as illustrated in ﬁgure 1a.

Original: Those structural roles represent local patterns in graphs.
Revised: Those structural roles represent local patterns in graphs.

Original: Identifying them and comparing them among graphs could improve the discriminative power of graph embeddings obtained with graph neural networks.
Revised: Identifying them and comparing them among graphs could improve the discriminative power of graph embeddings obtained with graph neural networks.

Original: In this work, we build an aggregation process based on the identiﬁcation of structural roles, called StructAgg.
Revised: In this work, we build an aggregation process based on the identiﬁcation of structural roles, called StructAgg.

Original: The main contributions of this work are summarized bellow:
Revised: The main contributions of this work are summarized bellow:

Original: Learned aggregation process.
Revised: Learned aggregation process.

Original: A differentiable aggregation process that learns how to aggregate node embeddings in order to produce a graph representation for a graph classiﬁcation task.
Revised: A differentiable aggregation process that learns how to aggregate node embeddings in order to produce a graph representation for a graph classiﬁcation task.

Original: Identiﬁcation of structural roles.
Revised: Identiﬁcation of structural roles.

Original: Based on the deﬁnition of structural roles from Donnat et al.
Revised: Based on the deﬁnition of structural roles from Donnat et al.

Original: (2018), our algorithm learns structural roles during the aggregation process.
Revised: (2018), our algorithm learns structural roles during the aggregation process.

Original: This is innovative because most algorithms that learn structural roles in graphs are not based on graph neural networks.
Revised: This is innovative because most algorithms that learn structural roles in graphs are not based on graph neural networks.

Original: Explainability of selected features for a graph classiﬁcation task.
Revised: Explainability of selected features for a graph classiﬁcation task.

Original: The identiﬁcation of structural roles enables us to understand and explain what features are selected during training.
Revised: The identiﬁcation of structural roles enables us to understand and explain what features are selected during training.

Original: Graph neural networks often lack explainability and there are only few works that tackle this issue.
Revised: Graph neural networks often lack explainability and there are only few works that tackle this issue.

Original: One contribution of this work is the explainability of the approach.
Revised: One contribution of this work is the explainability of the approach.

Original: We show how our end-to-end model provides interpretability to a graph classiﬁcation task based on graph neural networks.
Revised: We show how our end-to-end model provides interpretability to a graph classiﬁcation task based on graph neural networks.

Original: Experimental results.
Revised: Experimental results.

Original: Our method achieves state-of-the-art results on benchmark datasets.
Revised: Our method achieves state-of-the-art results on benchmark datasets.

Original: We compare it with kernel methods and state-of-the-art message passing algorithms that use pooling layers as aggregation processes.
Revised: We compare it with kernel methods and state-of-the-art message passing algorithms that use pooling layers as aggregation processes.

Original: [Figure]
Revised: [Figure]

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: The identiﬁcation of nodes that have similar structural roles is usually done by an explicit featurization of nodes or by algorithms that rely on random walks to explore nodes’ neighborhoods.
Revised: The identiﬁcation of nodes that have similar structural roles is usually done by an explicit featurization of nodes or by algorithms that rely on random walks to explore nodes’ neighborhoods.

Original: A well known algorithm in this line of research is RolX (Gilpin et al., 2013; Henderson et al., 2012), a matrix factorization that focuses on computing a soft assignment matrix based on a listing of topological properties set as inputs for nodes.
Revised: A well known algorithm in this line of research is RolX (Gilpin et al., 2013; Henderson et al., 2012), a matrix factorization that focuses on computing a soft assignment matrix based on a listing of topological properties set as inputs for nodes.

Original: Similarly struct2vec builds a multilayered graph based on topological metrics on nodes and then generates random walks to capture structural information.
Revised: Similarly struct2vec builds a multilayered graph based on topological metrics on nodes and then generates random walks to capture structural information.

Original: In another line of research, many works rely on graphlets to capture nodes’ topological properties and identify nodes with similar neighborhoods (Rossi et al., 2017; Lee et al., 2018;
Revised: In another line of research, many works rely on graphlets to capture nodes’ topological properties and identify nodes with similar neighborhoods (Rossi et al., 2017; Lee et al., 2018;

Original: Ahmed et al., 2018).
Revised: Ahmed et al., 2018).

Original: In their work, Donnat et al.
Revised: In their work, Donnat et al.

Original: (2018) compute node embeddings from wavelets in graphs to caracterize nodes’ neighborhood at different scales.
Revised: (2018) compute node embeddings from wavelets in graphs to caracterize nodes’ neighborhood at different scales.

Original: In this work, we introduce an aggregation process based on the identiﬁcation of structural roles in graphs that is computed in an end-to-end trainable fashion.
Revised: In this work, we introduce an aggregation process based on the identiﬁcation of structural roles in graphs that is computed in an end-to-end trainable fashion.

Original: We build a hierarchical representation of nodes by using neural network models in graphs to propagate nodes’ features at different hops.
Revised: We build a hierarchical representation of nodes by using neural network models in graphs to propagate nodes’ features at different hops.

Original: Recently there has been a rich line of research, inspired by deep models in images, that aims at redeﬁning neural networks in graphs and in particular convolutional neural networks (Defferrard et al., 2016; Kipf & Welling, 2016; Veliˇckovi´c et al., 2017; Hamilton et al., 2017; Bronstein et al., 2017; Bruna et al., 2013; Scarselli et al., 2009).
Revised: Recently there has been a rich line of research, inspired by deep models in images, that aims at redeﬁning neural networks in graphs and in particular convolutional neural networks (Defferrard et al., 2016; Kipf & Welling, 2016; Veliˇckovi´c et al., 2017; Hamilton et al., 2017; Bronstein et al., 2017; Bruna et al., 2013; Scarselli et al., 2009).

Original: Those convolutions can be viewed as message passing algorithms that are composed of two phases.
Revised: Those convolutions can be viewed as message passing algorithms that are composed of two phases.

Original: A message passing phase that runs for T steps is ﬁrst deﬁned in terms of message functions and vertex update functions.
Revised: A message passing phase that runs for T steps is ﬁrst deﬁned in terms of message functions and vertex update functions.

Original: A readout phase then computes a feature vector for the whole graph using some readout function.
Revised: A readout phase then computes a feature vector for the whole graph using some readout function.

Original: In this work we will see how to deﬁne a readout phase that is learnable and that is representative of meaningful patterns in graphs.
Revised: In this work we will see how to deﬁne a readout phase that is learnable and that is representative of meaningful patterns in graphs.

Original:  P ROPOSED M ETHOD
Revised:  P ROPOSED M ETHOD

Original: In this section we introduce the structural aggregation layer (StructAgg).
Revised: In this section we introduce the structural aggregation layer (StructAgg).

Original: We show how we identify structural classes for nodes in graphs; how those classes are used in order to develop an aggregation layer; and how this layer allows us to compare signiﬁcant structural patterns in graphs for a supervised classiﬁcation task.
Revised: We show how we identify structural classes for nodes in graphs; how those classes are used in order to develop an aggregation layer; and how this layer allows us to compare signiﬁcant structural patterns in graphs for a supervised classiﬁcation task.

Original: 3. N OTATIONS
Revised: 3. N OTATIONS

Original: Let G =
Revised: Let G =

Original: ( V, E, X ) be a graph, where V is the set of nodes of G , E the set of edges and X ∈ R n × f the feature matrix of G ’s nodes where f is the dimensionality of node features.
Revised: ( V, E, X ) be a graph, where V is the set of nodes of G , E the set of edges and X ∈ R n × f the feature matrix of G ’s nodes where f is the dimensionality of node features.

Original: Let n = | V | be the number of nodes of G and e =
Revised: Let n = | V | be the number of nodes of G and e =

Original: | E | the number of edges of G .
Revised: | E | the number of edges of G .

Original: Let A be the adjacency matrix of graph G and D be its degree diagonal matrix.
Revised: Let A be the adjacency matrix of graph G and D be its degree diagonal matrix.

Original: Let v i and v j be the i th and j th nodes of G , we have:
Revised: Let v i and v j be the i th and j th nodes of G , we have:

Original: [Equation]
Revised: [Equation]

Original: Let S
Revised: Let S

Original: =
Revised: =

Original: { G 1 , ..., G d } be a set of d graphs and { y 1 , ..., y d } be the labels associated with these graphs.
Revised: { G 1 , ..., G d } be a set of d graphs and { y 1 , ..., y d } be the labels associated with these graphs.

Original: 3. H IERARCHICAL S TRUCTURAL E MBEDDING
Revised: 3. H IERARCHICAL S TRUCTURAL E MBEDDING

Original: Graph neural networks.
Revised: Graph neural networks.

Original: We build our work upon graph neural networks (GNNs).
Revised: We build our work upon graph neural networks (GNNs).

Original: Several architectures of graph neural networks have been proposed by Defferrard et al.
Revised: Several architectures of graph neural networks have been proposed by Defferrard et al.

Original: ; Kipf & Welling (2016); Veliˇckovi´c et al.
Revised: ; Kipf & Welling (2016); Veliˇckovi´c et al.

Original: (2017) or Bruna & Li (2017).
Revised: (2017) or Bruna & Li (2017).

Original: Those graph neural network models are all based on propagation mechanisms of node features that follow a general neural message passing architecture (Ying et al., 2018;
Revised: Those graph neural network models are all based on propagation mechanisms of node features that follow a general neural message passing architecture (Ying et al., 2018;

Original: Gilmer et al., 2017):
Revised: Gilmer et al., 2017):

Original: [Equation]
Revised: [Equation]

Original: where X ( l ) ∈ R n × f l are the node embeddings computed after l steps of the GNN, X (0)
Revised: where X ( l ) ∈ R n × f l are the node embeddings computed after l steps of the GNN, X (0)

Original: =
Revised: =

Original: X , and
Revised: X , and

Original: MP is the message propagation function, which depends on the adjacency matrix.
Revised: MP is the message propagation function, which depends on the adjacency matrix.

Original: W ( l ) is a trainable weight matrix that depends on layer l .
Revised: W ( l ) is a trainable weight matrix that depends on layer l .

Original: Let f l be the dimension of the node vectors after l steps of the
Revised: Let f l be the dimension of the node vectors after l steps of the

Original: GNN, f 0 = f .
Revised: GNN, f 0 = f .

Original: The aggregation process that we introduce next can be used with any neural message passing algorithm that follows the propagation rule 1.
Revised: The aggregation process that we introduce next can be used with any neural message passing algorithm that follows the propagation rule 1.

Original: In all the following of our work we denote by MP the algorithm.
Revised: In all the following of our work we denote by MP the algorithm.

Original: For the experiments, we consider Graph Convolutional Network (GCN) deﬁned by (Kipf & Welling, 2016).
Revised: For the experiments, we consider Graph Convolutional Network (GCN) deﬁned by (Kipf & Welling, 2016).

Original: This model is based on an approximation of convolutions on graphs deﬁned by (Defferrard et al., 2016) and that use spectral decompositions of the Laplacian.
Revised: This model is based on an approximation of convolutions on graphs deﬁned by (Defferrard et al., 2016) and that use spectral decompositions of the Laplacian.

Original: The popularity of this model comes from its computational efﬁciency and the state-of-the-art results obtained on benchmark datasets.
Revised: The popularity of this model comes from its computational efﬁciency and the state-of-the-art results obtained on benchmark datasets.

Original: This layer propagates node features to 1 -hop neighbors.
Revised: This layer propagates node features to 1 -hop neighbors.

Original: Its propagation rule is the following:
Revised: Its propagation rule is the following:

Original: [Equation]
Revised: [Equation]

Original: Where ρ is a non-linear function (a ReLU in our case),
Revised: Where ρ is a non-linear function (a ReLU in our case),

Original: ˜ A =
Revised: ˜ A =

Original: A + I n is the adjacency matrix with added self-loops and ˜ D ii
Revised: A + I n is the adjacency matrix with added self-loops and ˜ D ii

Original: =
Revised: =

Original: (cid:80)
Revised: (cid:80)

Original: j ˜
Revised: j ˜

Original: A ij .
Revised: A ij .

Original: This propagation process allows us to obtain a node representation representing its l -hop neighborhood after l layers of GCN.
Revised: This propagation process allows us to obtain a node representation representing its l -hop neighborhood after l layers of GCN.

Original: We build a hierarchical representation for nodes by concatenating their embeddings after each step of GCN.
Revised: We build a hierarchical representation for nodes by concatenating their embeddings after each step of GCN.

Original: The ﬁnal representation X structi of a node i is given by:
Revised: The ﬁnal representation X structi of a node i is given by:

Original: [Equation]
Revised: [Equation]

Original: Where L is the total number of GCN layers applied.
Revised: Where L is the total number of GCN layers applied.

Original: Identifying structural classes.
Revised: Identifying structural classes.

Original: Embedding nodes with MP creates embeddings that are close for nodes that are structurally equivalent.
Revised: Embedding nodes with MP creates embeddings that are close for nodes that are structurally equivalent.

Original: Some use a handcrafted node embedding based on propagation processes with wavelets in graphs to identify structural clusters based on hierarchical representation of nodes (Donnat et al., 2018).
Revised: Some use a handcrafted node embedding based on propagation processes with wavelets in graphs to identify structural clusters based on hierarchical representation of nodes (Donnat et al., 2018).

Original: By analogy, we learn hierarchical node embeddings and an aggregation layer that identiﬁes structural roles for node in graphs.
Revised: By analogy, we learn hierarchical node embeddings and an aggregation layer that identiﬁes structural roles for node in graphs.

Original: Those structural roles are consistent along graphs of a dataset which allows us to bring interpretability to our graph classiﬁcation task.
Revised: Those structural roles are consistent along graphs of a dataset which allows us to bring interpretability to our graph classiﬁcation task.

Original: Node features X struct contain the information of their L -hop neighborhood decomposed into L vectors each representing their l -hop neighborhood for l varying between 1 and L .
Revised: Node features X struct contain the information of their L -hop neighborhood decomposed into L vectors each representing their l -hop neighborhood for l varying between 1 and L .

Original: We will show next that nodes that have the same L -hop neighborhood are embedded into the same vector.
Revised: We will show next that nodes that have the same L -hop neighborhood are embedded into the same vector.

Original: To identify structural roles, we thus project each node embedding on a matrix p ∈ R f struct × c where c is the number of structural classes and f struct =
Revised: To identify structural roles, we thus project each node embedding on a matrix p ∈ R f struct × c where c is the number of structural classes and f struct =

Original: L (cid:80) l =1 f l is the dimensionality of X structi for each node i .
Revised: L (cid:80) l =1 f l is the dimensionality of X structi for each node i .

Original: We obtain a soft assignment matrix:
Revised: We obtain a soft assignment matrix:

Original: [Equation]
Revised: [Equation]

Original: Where the softmax function is applied in a row-wise fashion.
Revised: Where the softmax function is applied in a row-wise fashion.

Original: This way, C ij represents the probability that node i belongs to cluster j .
Revised: This way, C ij represents the probability that node i belongs to cluster j .

Original: Deﬁnition 1.
Revised: Deﬁnition 1.

Original: Let i and j be two nodes of a graph G
Revised: Let i and j be two nodes of a graph G

Original: =
Revised: =

Original: ( V, E, X ) .
Revised: ( V, E, X ) .

Original: Let N l ( i )
Revised: Let N l ( i )

Original: =
Revised: =

Original: { i (cid:48) ∈ N | d ( i, i (cid:48) ) ≤ l } be the l -hop neighborhood of i , which means all the nodes that are at distance lower of equal to l of i , d being the shortest-path distance.
Revised: { i (cid:48) ∈ N | d ( i, i (cid:48) ) ≤ l } be the l -hop neighborhood of i , which means all the nodes that are at distance lower of equal to l of i , d being the shortest-path distance.

Original: Let X N l ( i ) be the feature matrix of the l -hop neighborhood of u.
Revised: Let X N l ( i ) be the feature matrix of the l -hop neighborhood of u.

Original: Let G i,l be the subgraph of G composed of the l -hop neighborhood of i .
Revised: Let G i,l be the subgraph of G composed of the l -hop neighborhood of i .

Original: We say that u and v are l -structurally equivalent if there exists an isomorphism ψ from N l ( j ) to N l ( i ) such that the two following conditions are veriﬁed:
Revised: We say that i and j are l -structurally equivalent if there exists an isomorphism ψ from N l ( j ) to N l ( i ) such that the two following conditions are veriﬁed:

Original: • G i,l = ψ ( G j,l ) • ∀ j (
Revised: • G i,l = ψ ( G j,l ) • ∀ j (

Original: cid:48) ∈ N l ( j ) , X ψ ( j (cid:48) )
Revised: cid:48) ∈ N l ( j ) , X ψ ( j (cid:48) )

Original: =
Revised: =

Original: X j (cid:48)
Revised: X j (cid:48)

Original: Theorem 1.
Revised: Theorem 1.

Original: Two nodes i and j that are L -structurally equivalent have the same ﬁnal embedding,
Revised: Two nodes i and j that are L -structurally equivalent have the same ﬁnal embedding,

Original: X structi =
Revised: X structi =

Original: X structj .
Revised: X structj .

Original: 3. S TRUCTURAL A GGREGATION
Revised: 3. S TRUCTURAL A GGREGATION

Original: After having identiﬁed structural classes, we aggregate node embeddings over those classes, as illustrated in ﬁgure 2.
Revised: After having identiﬁed structural classes, we aggregate node embeddings over those classes, as illustrated in ﬁgure 2.

Original: The goal is to obtain an embedding that discriminates graphs that do not belong to the same class and that selects similar patterns in graphs of the same class.
Revised: The goal is to obtain an embedding that discriminates graphs that do not belong to the same class and that selects similar patterns in graphs of the same class.

Original: Graphs that have similar properties and thus similar node patterns should have nodes with similar roles and similar embeddings.
Revised: Graphs that have similar properties and thus similar node patterns should have nodes with similar roles and similar embeddings.

Original: Performing a structural aggregation.
Revised: Performing a structural aggregation.

Original: The structural aggregation aims at comparing embeddings of nodes that have similar roles in the graph.
Revised: The structural aggregation aims at comparing embeddings of nodes that have similar roles in the graph.

Original: When computing the distance between two graphs, if those two graphs have the same distribution of structural roles, they will have embeddings that are close.
Revised: When computing the distance between two graphs, if those two graphs have the same distribution of structural roles, they will have embeddings that are close.

Original: Nodes that are leaves or nodes that are central in the graph should be compared separately.
Revised: Nodes that are leaves or nodes that are central in the graph should be compared separately.

Original: Mathematically, the variance over all nodes if the graph may be high and decomposing nodes per structural roles aims at decreasing the variance per cluster and thus at bringing more information to the ﬁnal graph embedding.
Revised: Mathematically, the variance over all nodes if the graph may be high and decomposing nodes per structural roles aims at decreasing the variance per cluster and thus at bringing more information to the ﬁnal graph embedding.

Original: The structural aggregation layer performs an aggregation per structural role.
Revised: The structural aggregation layer performs an aggregation per structural role.

Original: This aggregation is a mean of embeddings of nodes that belong to the same cluster.
Revised: This aggregation is a mean of embeddings of nodes that belong to the same cluster.

Original: The ﬁnal embedding is a concatenation of the mean embeddings of the nodes per cluster of structural role.
Revised: The ﬁnal embedding is a concatenation of the mean embeddings of the nodes per cluster of structural role.

Original: [Equation]
Revised: [Equation]

Original: Proposition 1.
Revised: Proposition 1.

Original: The embedding Z graph is invariant by node permutation.
Revised: The embedding Z graph is invariant by node permutation.

Original: Moreover when performing soft classiﬁcation, the output cluster assignment for each node should generally be close to a one-hot vector so that each node has a clear structural role identiﬁed.
Revised: Moreover when performing soft classiﬁcation, the output cluster assignment for each node should generally be close to a one-hot vector so that each node has a clear structural role identiﬁed.

Original: We therefore regularize the entropy of the cluster assigmnent matrix by minimizing L reg = 1 n n (cid:80)
Revised: We therefore regularize the entropy of the cluster assigmnent matrix by minimizing L reg = 1 n n (cid:80)

Original: [Equation]
Revised: [Equation]

Original: ) is the entropy of the assignment vector C i. of node i .
Revised: ) is the entropy of the assignment vector C i. of node i .

Original: This is often done when identifying classes for nodes.
Revised: This is often done when identifying classes for nodes.

Original: The same regularization is applied in (Ying et al., 2018) to identify communities in graphs to develop a pooling layer.
Revised: The same regularization is applied in (Ying et al., 2018) to identify communities in graphs to develop a pooling layer.

Original: Remarks.
Revised: Remarks.

Original: • The structural classes identiﬁed by our algorithm are local.
Revised: • The structural classes identiﬁed by our algorithm are local.

Original: They contain the information of the L -hop neighborhood of each node.
Revised: They contain the information of the L -hop neighborhood of each node.

Original: • Compared to the structural embedding presented in Donnat et al.
Revised: • Compared to the structural embedding presented in Donnat et al.

Original: (2018), our algorithm needs multiple graphs and is trained in a supervised fashion.
Revised: (2018), our algorithm needs multiple graphs and is trained in a supervised fashion.

Original: Donnat et al.
Revised: Donnat et al.

Original: (2018) introduced a node embedding that allows us to identify structural classes in a single graph.
Revised: (2018) introduced a node embedding that allows us to identify structural classes in a single graph.

Original: It is not straightforward how to generalize this procedure to the case of a set of graphs.
Revised: It is not straightforward how to generalize this procedure to the case of a set of graphs.

Original: Indeed, when dealing with a single graph, this procedure is very efﬁcient at identifying structural roles.
Revised: Indeed, when dealing with a single graph, this procedure is very efﬁcient at identifying structural roles.

Original: However, those structural roles are deﬁned per graph and thus two nodes that have the same structural role but that lie in two different graphs can have embeddings that differ and can thus be classiﬁed
Revised: However, those structural roles are deﬁned per graph and thus two nodes that have the same structural role but that lie in two different graphs can have embeddings that differ and can thus be classiﬁed

Original: in two different classes.
Revised: in two different classes.

Original: 3. I NTERPRETABLE G RAPH N EURAL N ETWORK
Revised: 3. I NTERPRETABLE G RAPH N EURAL N ETWORK

Original: Graph neural networks lack interpretability.
Revised: Graph neural networks lack interpretability.

Original: In the case of convolutional neural networks on images, it is possible to visualize the activation of certain layers to have an idea of the patterns that are selected during the process of image classiﬁcation and of which pattern is usefull to discriminate images of a certain class from the rest of the images of a dataset.
Revised: In the case of convolutional neural networks on images, it is possible to visualize the activation of certain layers to have an idea of the patterns that are selected during the process of image classiﬁcation and of which pattern is usefull to discriminate images of a certain class from the rest of the images of a dataset.

Original: In the case of graphs, most algorithms based on neural network models bring no interpretability regarding the features that are selected and that make the classiﬁcation accurate.
Revised: In the case of graphs, most algorithms based on neural network models bring no interpretability regarding the features that are selected and that make the classiﬁcation accurate.

Original: In this work, we bring some interpretability to this class of models and illustrate it in the next sections in experiments.
Revised: In this work, we bring some interpretability to this class of models and illustrate it in the next sections in experiments.

Original: By propagating information from node to node, we are able to identify structural roles in graphs.
Revised: By propagating information from node to node, we are able to identify structural roles in graphs.

Original: Those structural roles contain the information of local neighborhoods of nodes and of their local topological structure.
Revised: Those structural roles contain the information of local neighborhoods of nodes and of their local topological structure.

Original: By identifying roles in graphs, we compare between graphs the embeddings per structural roles.
Revised: By identifying roles in graphs, we compare between graphs the embeddings per structural roles.

Original: This way, we can identify roles that are speciﬁc to a certain class and caracterize each class with the combination of roles they are made of.
Revised: This way, we can identify roles that are speciﬁc to a certain class and caracterize each class with the combination of roles they are made of.

Original: We show through extensive experiments that the information contained in the identiﬁcation of roles allows us to discriminate graphs of different classes and lead the way to other works that could bring interpretability to this ﬁeld of research.
Revised: We show through extensive experiments that the information contained in the identiﬁcation of roles allows us to discriminate graphs of different classes and lead the way to other works that could bring interpretability to this ﬁeld of research.

Original:  E XPERIMENTS 4. G RAPH CLASSIFICATION
Revised:  E XPERIMENTS 4. G RAPH CLASSIFICATION

Original: Datasets: We choose a wide variety of benchmark datasets for graph classiﬁcation to evaluate our model.
Revised: Datasets: We choose a wide variety of benchmark datasets for graph classiﬁcation to evaluate our model.

Original: The datasets can be separated in two types.
Revised: The datasets can be separated in two types.

Original: 2 bioinformatics datasets: PROTEINS and
Revised: 2 bioinformatics datasets: PROTEINS and

Original: D&D; and a social network dataset: COLLAB.
Revised: D&D; and a social network dataset: COLLAB.

Original: In the bioinformatics datasets, graphs represent chemical compounds.
Revised: In the bioinformatics datasets, graphs represent chemical compounds.

Original: Nodes are atoms and edges represent connections between two atoms.
Revised: Nodes are atoms and edges represent connections between two atoms.

Original: D&D and PROTEINS contain two classes of molecules that represent the fact that a molecule can be either active or inactive against a certain type of cancer.
Revised: D&D and PROTEINS contain two classes of molecules that represent the fact that a molecule can be either active or inactive against a certain type of cancer.

Original: The aim is to classify molecules according to their anti-cancer activity.
Revised: The aim is to classify molecules according to their anti-cancer activity.

Original: COLLAB is composed of ego-networks.
Revised: COLLAB is composed of ego-networks.

Original: Graphs’ labels are the nature of the entity from which we have generated the ego-network.
Revised: Graphs’ labels are the nature of the entity from which we have generated the ego-network.

Original: In Table 1 we report some information on these datasets such as the maximum number of nodes in graphs, the average number of nodes per graph, the number of graphs, the size of node features f (if available) and the number of classes.
Revised: In Table 1 we report some information on these datasets such as the maximum number of nodes in graphs, the average number of nodes per graph, the number of graphs, the size of node features f (if available) and the number of classes.

Original: More details can be found in (Yanardag & Vishwanathan, 2015).
Revised: More details can be found in (Yanardag & Vishwanathan, 2015).

Original: We also use the new database
Revised: We also use the new database

Original: Open Graph Benchmark (OBG) to test our method on larger datasets (Hu et al., 2020).
Revised: Open Graph Benchmark (OBG) to test our method on larger datasets (Hu et al., 2020).

Original: We use the ogb-molhiv dataset whose properties are listed bellow.
Revised: We use the ogb-molhiv dataset whose properties are listed bellow.

Original: Each graph represents a molecule, where nodes are atoms, and edges are chemical bonds.
Revised: Each graph represents a molecule, where nodes are atoms, and edges are chemical bonds.

Original: Input node features are 9 -dimensional, containing atomic number and chirality, as well as other additional atom features such as formal charge and whether the atom is in the ring or not.
Revised: Input node features are 9 -dimensional, containing atomic number and chirality, as well as other additional atom features such as formal charge and whether the atom is in the ring or not.

Original: Experimental setup: We perform a 10 -fold cross validation split which gives 10 sets of train, validation and test data indices in the ratio 8:1:1.
Revised: Experimental setup: We perform a 10 -fold cross validation split which gives 10 sets of train, validation and test data indices in the ratio 8:1:1.

Original: We use stratiﬁed sampling to ensure that the class distribution remains the same across splits.
Revised: We use stratiﬁed sampling to ensure that the class distribution remains the same across splits.

Original: We ﬁne tune hyperparameters n roles the number of structural roles, lr the learning rate and ﬁnally the dimensions of the successive layers respectively chosen from the sets { 2 , 5 , 10 , 20 } , { 0 . 01 , 0 . 001 , 0 . 0001 } , { 8 , 16 , 32 , 64 , 128 , 256 } .We ﬁx l 1 =and l 2 = 2 .
Revised: We ﬁne tune hyperparameters n roles the number of structural roles, lr the learning rate and ﬁnally the dimensions of the successive layers respectively chosen from the sets { 2 , 5 , 10 , 20 } , { 0 . 01 , 0 . 001 , 0 . 0001 } , { 8 , 16 , 32 , 64 , 128 , 256 } .

Original: The sets from which our hyperparameters are selected vary according to the sizes of graphs in each dataset.
Revised: The sets from which our hyperparameters are selected vary according to the sizes of graphs in each dataset.

Original: We do not set a maximum number of epochs but we perform early stopping to stop the training which means that we stop the training when the validation loss has not improved for 20 epochs.
Revised: We do not set a maximum number of epochs but we perform early stopping to stop the training which means that we stop the training when the validation loss has not improved for 20 epochs.

Original: We report the mean accuracy and the standard deviation over the 10 folds on the validation set.
Revised: We report the mean accuracy and the standard deviation over the 10 folds on the validation set.

Original: We compare our method with kernel methods and with a graph neural network that uses pooling layers (Ying et al., 2018).
Revised: We compare our method with kernel methods and with a graph neural network that uses pooling layers (Ying et al., 2018).

Original: We should note that kernel methods do not use node features that are available on bioinformatics datasets.
Revised: We should note that kernel methods do not use node features that are available on bioinformatics datasets.

Original: For COLLAB, we don’t have any features available on nodes.
Revised: For COLLAB, we don’t have any features available on nodes.

Original: We compute the one-hot encoding of node degrees that we use as node features for our algorithm.
Revised: We compute the one-hot encoding of node degrees that we use as node features for our algorithm.

Original: Results:
Revised: Results:

Original: From the results of Table 1 we can observe that StructAgg is competing with state-of-the-art methods.
Revised: From the results of Table 1 we can observe that StructAgg is competing with state-of-the-art methods.

Original: Indeed, on most datasets, StructAgg has a score very close to those obtained by Ying et al.
Revised: Indeed, on most datasets, StructAgg has a score very close to those obtained by Ying et al.

Original: and Gao & Ji (2019).
Revised: and Gao & Ji (2019).

Original: From Table 1, StructAgg outperforms all algorithms on COLLAB.
Revised: From Table 1, StructAgg outperforms all algorithms on COLLAB.

Original: Moreover, StructAgg allows us to improve classiﬁcation results from GCN on ogb-molhiv dataset as illustrated in ﬁgure
Revised: Moreover, StructAgg allows us to improve classiﬁcation results from GCN on ogb-molhiv dataset as illustrated in ﬁgure

Original: Dataset
Revised: Dataset

Original: D&D
Revised: D&D

Original: PROTEINS
Revised: PROTEINS

Original: COLLAB
Revised: COLLAB

Original: Max #Nodes Avg #Nodes 284.
Revised: [Table] [Table]

Original: #Graphs fclasses
Revised: [Table] fclasses

Original: Graphlet (Shervashidze et al., 2009) 74.
Revised: Graphlet (Shervashidze et al., 2009) 74.

Original: Shortest-Path (Borgwardt & Kriegel, 2005) 78.
Revised: Shortest-Path (Borgwardt & Kriegel, 2005) 78.

Original: 1-WL (Shervashidze et al., 2011) 74.
Revised: 1-WL (Shervashidze et al., 2011) 74.

Original: WL-OA
Revised: WL-OA

Original: (Kriege et al., 2016) 79.
Revised: (Kriege et al., 2016) 79.

Original: GraphSage (Hamilton et al., 2017) 75.
Revised: GraphSage (Hamilton et al., 2017) 75.

Original: DGCNN (Zhang et al., 2018) 79.
Revised: DGCNN (Zhang et al., 2018) 79.

Original: DIFFPOOL (Ying et al., 2018) 80.
Revised: DIFFPOOL (Ying et al., 2018) 80.

Original: g-U-Nets (Gao & Ji, 2019) 82.
Revised: g-U-Nets (Gao & Ji, 2019) 82.

Revised: StructAgg

Original: [Table] [Table]
Revised: [Table] [Table]

Original: 4. I DENTIFICATION OF STRUCTURAL ROLES
Revised: 4. I DENTIFICATION OF STRUCTURAL ROLES

Original: [Figure]
Revised: [Figure]

Original: We show some examples of molecules of a molecular dataset and the roles identiﬁed by our algorithms.
Revised: We show some examples of molecules of a molecular dataset and the roles identiﬁed by our algorithms.

Original: From ﬁgure 3 we can see that there is some consistence in the assignment of nodes to structural roles.
Revised: From ﬁgure 3 we can see that there is some consistence in the assignment of nodes to structural roles.

Original: Moreover, a great advantage of our method is that it takes into account node features that are in our case a one-hot encoding of the atom type.
Revised: Moreover, a great advantage of our method is that it takes into account node features that are in our case a one-hot encoding of the atom type.

Original: Compared to Donnat et al.
Revised: Compared to Donnat et al.

Original: (2018) whose embeddings do not include node features, our method identiﬁes roles in a macroscopic fashion by identifying similar roles in the whole dataset and not in a single graph.
Revised: (2018) whose embeddings do not include node features, our method identiﬁes roles in a macroscopic fashion by identifying similar roles in the whole dataset and not in a single graph.

Original: Most methods rely on computed features that correspond to a single graph and do not generalize to multiple graphs because of scales issues.
Revised: Most methods rely on computed features that correspond to a single graph and do not generalize to multiple graphs because of scales issues.

Original: It is the case for GraphWave for which features computed in order to cluster nodes per roles depend on the size of the graph.
Revised: It is the case for GraphWave for which features computed in order to cluster nodes per roles depend on the size of the graph.

Original: To compare our role assignment with GraphWave, we computed assignments per graph and not along the whole dataset.
Revised: To compare our role assignment with GraphWave, we computed assignments per graph and not along the whole dataset.

Original: We can note that compared to GraphWave, our algorithm uses node features to select structal roles.
Revised: We can note that compared to GraphWave, our algorithm uses node features to select structal roles.

Original: 4. I MPORTANCE OF STRUCTURAL PATTERNS
Revised: 4. I MPORTANCE OF STRUCTURAL PATTERNS

Original: Identifying roles in graphs boils down to identifying signiﬁcant patterns.
Revised: Identifying roles in graphs boils down to identifying signiﬁcant patterns.

Original: As shown in subsection 4.2, the roles that we were able to identify represent coherent patterns in graphs.
Revised: As shown in subsection 4.2, the roles that we were able to identify represent coherent patterns in graphs.

Original: Graphs’ ﬁnal embeddings are made of two parts.
Revised: Graphs’ ﬁnal embeddings are made of two parts.

Original: Nodes’ embeddings that are the result of successive GCNs and the allocation of all nodes to different structural classes.
Revised: Nodes’ embeddings that are the result of successive GCNs and the allocation of all nodes to different structural classes.

Original: We would like to quantify how much information is contained in the structural roles and if the decomposition of the graph in c classes improves our classiﬁcation accuracy.
Revised: We would like to quantify how much information is contained in the structural roles and if the decomposition of the graph in c classes improves our classiﬁcation accuracy.

Original: To this end, from our trained algorithm, we compute all allocation matrices of all graphs of a dataset.
Revised: To this end, from our trained algorithm, we compute all allocation matrices of all graphs of a dataset.

Original: In order to validate the fact that roles themselves have a high discriminative power, we need to identify combinations of roles that are speciﬁc to different classes.
Revised: In order to validate the fact that roles themselves have a high discriminative power, we need to identify combinations of roles that are speciﬁc to different classes.

Original: We thus want to identify the structural roles that compose each graph.
Revised: We thus want to identify the structural roles that compose each graph.

Original: Let’s consider a graph G and an allocation matrix C ∈ R n × c .
Revised: Let’s consider a graph G and an allocation matrix C ∈ R n × c .

Original: We obtain a soft assignment matrix.
Revised: We obtain a soft assignment matrix.

Original: We compute on it a histogram per class over each node of the graph.
Revised: We compute on it a histogram per class over each node of the graph.

Original: The feature vector is now a concatenation of all histograms per class.
Revised: The feature vector is now a concatenation of all histograms per class.

Original: We have:
Revised: We have:

Original: [Equation]
Revised: [Equation]

Original: Where bins is the number of bins for the histogram.
Revised: Where bins is the number of bins for the histogram.

Original: The histogram upper and lower bounds areand 1 because the values of C are the output of a softmax function.
Revised: The histogram upper and lower bounds areand 1 because the values of C are the output of a softmax function.

Original: Let ( G 1 , ..., G d ) be d graphs of labels ( y 1 , ..., y d ) .
Revised: Let ( G 1 , ..., G d ) be d graphs of labels ( y 1 , ..., y d ) .

Original: After having computed all graphs’ histograms we obtain d vectors ( Feat G 1 , ..., Feat G d ) that we use as imputs of a classiﬁer.
Revised: After having computed all graphs’ histograms we obtain d vectors ( Feat G 1 , ..., Feat G d ) that we use as imputs of a classiﬁer.

Original: We use a SVM and we display the results in table 3 in order to compare the information that comes from the strutural roles identiﬁcation and the information that comes from node embeddings.
Revised: We use a SVM and we display the results in table 3 in order to compare the information that comes from the strutural roles identiﬁcation and the information that comes from node embeddings.

Original: We can see that the accuracy is lower when we don’t consider node embeddings which is consistent with the fact that a lot of information is contained in the embeddings of nodes that are the outputs of GCNs.
Revised: We can see that the accuracy is lower when we don’t consider node embeddings which is consistent with the fact that a lot of information is contained in the embeddings of nodes that are the outputs of GCNs.

Original: But the accuracy is signiﬁcantly higher than a random model which proves that we can identify some patterns in the distribution of structural roles among graphs and that those patterns are a good ﬁrst approximation to separate classes in a dataset.
Revised: But the accuracy is signiﬁcantly higher than a random model which proves that we can identify some patterns in the distribution of structural roles among graphs and that those patterns are a good ﬁrst approximation to separate classes in a dataset.

Original: [Table]
Revised: [Table]

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: Graph neural networks are very effective to build node embeddings by propagating node features in graphs.
Revised: Graph neural networks are very effective to build node embeddings by propagating node features in graphs.

Original: But in order to build a graph representation from these embeddings there are several ways to procede.
Revised: But in order to build a graph representation from these embeddings there are several ways to procede.

Original: In this work, we proposed a novel technique based on the identiﬁcation of structural roles for nodes in graphs.
Revised: In this work, we proposed a novel technique based on the identiﬁcation of structural roles for nodes in graphs.

Original: We showed that the identiﬁcation of roles allows us to compare patterns between graphs that are signiﬁcant for graph classiﬁcation.
Revised: We showed that the identiﬁcation of roles allows us to compare patterns between graphs that are signiﬁcant for graph classiﬁcation.

Original: Moreover, this work opens new perspectives in the ﬁeld of explainability of graph neural networks.
Revised: Moreover, this work opens new perspectives in the ﬁeld of explainability of graph neural networks.

Original: We showed that these patterns brought explainability to this task and to which kind of structures are selected by our algorithm during the process of graph classiﬁcation.
Revised: We showed that these patterns brought explainability to this task and to which kind of structures are selected by our algorithm during the process of graph classiﬁcation.

Original: A better understanding of which features are selected during training can enable us to develop new methods based on drawn conclusion and maybe open new perspective in applied mathematics for drug discovery.
Revised: A better understanding of which features are selected during training can enable us to develop new methods based on drawn conclusion and maybe open new perspective in applied mathematics for drug discovery.

Original: A A PPENDIX
Revised: A A PPENDIX

Original: You may include other additional sections here.
Revised: You may include other additional sections here.

Original: B P ROOFS B. P ROOF OF THEOREM
Revised: B P ROOFS B. P ROOF OF THEOREM

Original: Theorem 2.
Revised: Theorem 2.

Original: Two nodes i and j that are L -structurally equivalent have the same ﬁnal embedding,
Revised: Two nodes i and j that are L -structurally equivalent have the same ﬁnal embedding,

Original: X structi =
Revised: X structi =

Original: X structj .
Revised: X structj .

Original: Proof.
Revised: Proof.

Original: Let i and j be two nodes of a graph G
Revised: Let i and j be two nodes of a graph G

Original: =
Revised: =

Original: ( V, E, X )
Revised: ( V, E, X )

Original: that are L -structurally equivalent.
Revised: that are L -structurally equivalent.

Original: Let P ( l ) be the following proposition:
Revised: Let P ( l ) be the following proposition:

Original: Two nodes that are l -structurally equivalent for some l , have the same embedding after l steps of GCN.
Revised: Two nodes that are l -structurally equivalent for some l , have the same embedding after l steps of GCN.

Original: Let’sprove this proposition by induction.
Revised: Let’sprove this proposition by induction.

Original: P (0) is true, two nodes that have the same embedding are 0 -structurally equivalent after 0 step of GCN.
Revised: P (0) is true, two nodes that have the same embedding are 0 -structurally equivalent after 0 step of GCN.

Original: Let P ( l ) be true and let’s prove P ( l + 1) .
Revised: Let P ( l ) be true and let’s prove P ( l + 1) .

Original: Let i and j be two nodes that are l -structurally equivalent.
Revised: Let i and j be two nodes that are l -structurally equivalent.

Original: After a step of GCN we have:
Revised: After a step of GCN we have:

Original: [Equation]
Revised: [Equation]

Original: So we have:
Revised: So we have:

Original: [Equation]
Revised: [Equation]

Original: Since i and j are l -structurally equivalent, there exists an isomorphism ψ such that:
Revised: Since i and j are l -structurally equivalent [Equation] there exists an isomorphism ψ such that:

Original: [Equation]
Revised: [Equation]

Original: that
Revised: that

Original: [Equation]
Revised: [Equation]

Original: So two nodes i and j that are L structurally equivalent, are l structurally equivalent for all l between 0 and L and thus, X structi =
Revised: So two nodes i and j that are L structurally equivalent, are l structurally equivalent for all l between 0 and L and thus, X structi =

Original: X structj because X struct is the concatenation of embeddings after each layer.
Revised: X structj because X struct is the concatenation of embeddings after each layer.

Original: B. P ROOF OF PROPOSITION
Revised: B. P ROOF OF PROPOSITION

Original: Proposition 2.
Revised: Proposition 2.

Original: The embedding Z graph is invariant by node permutation.
Revised: The embedding Z graph is invariant by node permutation.

Original: Proof.
Revised: Proof.

Original: Let P ∈ 0 , 1 n × n be any permutation matrix.
Revised: Let P ∈ 0 , 1 n × n be any permutation matrix.

Original: Since P is a permutation matrix we have
Revised: Since P is a permutation matrix we have

Original: PP T
Revised: PP T

Original: =
Revised: =

Original: I .
Revised: I .

Original: We have PX ( l +1) =
Revised: We have PX ( l +1) =

Original: GCN ( PAP T , PX ( l ) ) .
Revised: GCN ( PAP T , PX ( l ) ) .

Original: [Equation]
Revised: [Equation]

Original: Since the softmax is applied in a row-wise fashion, we have:
Revised: Since the softmax is applied in a row-wise fashion, we have:

Original: [Equation]
Revised: [Equation]

Original: Where Z Pgraph is the embedding of our graph to which we have applied the permutation P
Revised: Where Z Pgraph is the embedding of our graph to which we have applied the permutation P