Original: F LOW X: T OWARDS E XPLAINABLE G RAPH N EURAL N ETWORKS VIA M ESSAGE F LOWS
Revised: F LOW X: T OWARDS E XPLAINABLE G RAPH N EURAL N ETWORKS VIA M ESSAGE F LOWS

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: We investigate the explainability of graph neural networks (GNNs) as a step towards elucidating their working mechanisms.
Revised: We investigate the explainability of graph neural networks (GNNs) as a step towards elucidating their working mechanisms.

Original: While most current methods focus on explaining graph nodes, edges, or features, we argue that, as the inherent functional mechanism of GNNs, message ﬂows are more natural for performing explainability.
Revised: While most current methods focus on explaining graph nodes, edges, or features, we argue that, as the inherent functional mechanism of GNNs, message ﬂows are more natural for performing explainability.

Original: To this end, we propose a novel method here, known as FlowX, to explain GNNs by identifying important message ﬂows.
Revised: To this end, we propose a novel method here, known as FlowX, to explain GNNs by identifying important message ﬂows.

Original: To quantify the importance of ﬂows, we propose to employ the concept of Shapley values from cooperative game theory.
Revised: To quantify the importance of ﬂows, we propose to follow the philosophy of Shapley values from cooperative game theory.

Original: To tackle the complexity of computing Shapley values, we propose an approximation scheme to compute Shapley values as initial assessments of ﬂow importance.
Revised: To tackle the complexity of computing all coalitions’ marginal contributions, we propose an approximation scheme to compute Shapley-like values as initial assessments of further redistribution training.

Original: We then propose a learning algorithm to reﬁne scores and improve explainability.
Revised: We then propose a learning algorithm to train ﬂow scores and improve explainability.

Original: Experimental studies on both synthetic and real-world datasets demonstrate that our proposed FlowX leads to improved explainability of GNNs
Revised: Experimental studies on both synthetic and real-world datasets demonstrate that our proposed FlowX leads to improved explainability of GNNs

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: With the advances of deep learning, graph neural networks (GNNs) are achieving promising performance on many graph tasks, including graph classiﬁcation (Xu et al., 2019; Gao & Ji, 2019;
Revised: With the advances of deep learning, graph neural networks (GNNs) are achieving promising performance on many graph tasks, including graph classiﬁcation (Xu et al., 2019; Gao & Ji, 2019;

Original: Chen et al., 2020), node classiﬁcation (Kipf & Welling, 2017; Veliˇckovi´c et al., 2018; Wu et al., 2019), and graph generation (Luo et al., 2021; You et al., 2018).
Revised: Chen et al., 2020), node classiﬁcation (Kipf & Welling, 2017; Veliˇckovi´c et al., 2018; Wu et al., 2019), and graph generation (Luo et al., 2021; You et al., 2018).

Original: Many research efforts have been made to develop advanced graph operations, such as graph message passing (Kipf & Welling, 2017;
Revised: Many research efforts have been made to develop advanced graph operations, such as graph message passing (Kipf & Welling, 2017;

Original: Veli ˇ ckovi ´ c et al., 2018; Li et al., 2019), graph pooling (Yuan & Ji, 2020; Zhang et al., 2018;
Revised: Veli ˇ ckovi ´ c et al., 2018; Li et al., 2019), graph pooling (Yuan & Ji, 2020; Zhang et al., 2018;

Original: Ying et al., 2018), and 3D graph operations (Schütt et al., 2017; Klicpera et al., 2020).
Revised: Ying et al., 2018), and 3D graph operations (Schütt et al., 2017; Klicpera et al., 2020).

Original: Deep graph models usually consist of many layers of these operations stacked on top of each other interspersed with nonlinear functions.
Revised: Deep graph models usually consist of many layers of these operations stacked on top of each other interspersed with nonlinear functions.

Original: The resulting deep models are usually deep and highly nonlinear and complex.
Revised: The resulting deep models are usually deep and highly nonlinear and complex.

Original: While these complex systems allow for accurate modeling, their decision mechanisms are highly elusive and not human-intelligible.
Revised: While these complex systems allow for accurate modeling, their decision mechanisms are highly elusive and not human-intelligible.

Original: Given the increasing importance and demand for trustworthy and fair artiﬁcial intelligence, it is imperative to develop methods to open the black-box and explain these highly complex deep models.
Revised: Given the increasing importance and demand for trustworthy and fair artiﬁcial intelligence, it is imperative to develop methods to open the black-box and explain these highly complex deep models.

Original: Driven by these needs, signiﬁcant efforts have been made to investigate the explainability of deep models on images and texts.
Revised: Driven by these needs, signiﬁcant efforts have been made to investigate the explainability of deep models on images and texts.

Original: These methods are developed from different perspectives, including studying the gradients of models (Simonyan et al., 2013; Smilkov et al., 2017; Yang et al., 2019), mapping hidden features to input space (Zhou et al., 2016; Selvaraju et al., 2017), occluding different input features (Yuan et al., 2020a; Dabkowski & Gal, 2017; Chen et al., 2018), and studying the meaning of hidden layers (Yuan et al., 2019; Olah et al., 2018; Du et al., 2018), etc.
Revised: These methods are developed from different perspectives, including studying the gradients of models (Simonyan et al., 2013; Smilkov et al., 2017; Yang et al., 2019), mapping hidden features to input space (Zhou et al., 2016; Selvaraju et al., 2017), occluding different input features (Yuan et al., 2020a; Dabkowski & Gal, 2017; Chen et al., 2018), and studying the meaning of hidden layers (Yuan et al., 2019; Olah et al., 2018; Du et al., 2018), etc.

Original: In contrast, the explainability of deep graph models is still less explored.
Revised: In contrast, the explainability of deep graph models is still less explored.

Original: Since graph data contain limited locality information but have important structural information, it is usually not natural to directly extend image or text based methods to graphs.
Revised: Since graph data contain limited locality information but have important structural information, it is usually not natural to directly extend image or text based methods to graphs.

Original: Recently, several techniques have been proposed to explain GNNs, such as XGNN (Yuan et al., 2020b), GNNExplainer
Revised: Recently, several techniques have been proposed to explain GNNs, such as XGNN (Yuan et al., 2020b), GNNExplainer

Original: (Ying et al., 2019), PGExplainer (Luo et al., 2020), and SubgraphX (Yuan et al., 2021), etc.
Revised: (Ying et al., 2019), PGExplainer (Luo et al., 2020), and SubgraphX (Yuan et al., 2021), etc.

Original: These methods mainly focus on explaining graph nodes, edges, features, or subgraphs.
Revised: These methods mainly focus on explaining graph nodes, edges, features, or subgraphs.

Original: In this work, we observe and argue that message ﬂows are the inherent functional mechanism of
Revised: In this work, we observe and argue that message ﬂows are the inherent functional mechanism of

Original: GNNs and thus are more natural and intuitive for studying the explainability of GNNs.
Revised: GNNs and thus are more natural and intuitive for studying the explainability of GNNs.

Original: To this end, we propose a message ﬂow based explanation method, known as FlowX, to explain GNNs.
Revised: To this end, we propose a message ﬂow based explanation method, known as FlowX, to explain GNNs.

Original: FlowX attributes GNN predictions to message ﬂows and studies the importance of different message ﬂows.
Revised: FlowX attributes GNN predictions to message ﬂows and studies the importance of different message ﬂows.

Original: We ﬁrst develop a systematic framework that lays the foundation on message ﬂows for naturally explaining the message passing in GNNs.
Revised: We ﬁrst develop a systematic framework that lays the foundation on message ﬂows for naturally explaining the message passing in GNNs.

Original: With our framework, the FlowX ﬁrst quantiﬁes the importance of ﬂows by employing the concept of Shapley values.
Revised: With our framework, the FlowX ﬁrst quantiﬁes the importance of ﬂows by following the phylosophy of Shapley values.

Original: Since message ﬂows cannot be directly quantiﬁed using Shapley values, we propose an approximation scheme as the initial assessments of different ﬂows.
Revised: Since message ﬂows cannot be directly quantiﬁed to calculate marginal contributions, we propose an approximation scheme as the initial assessments of different ﬂows.

Original: We then propose a learning-based algorithm to reﬁne the initial assessments to capture important ﬂows in predictions.
Revised: We then propose a learning-based algorithm taking advantages from the initial assessments to capture important ﬂows in predictions.

Original: We conduct extensive experiments on both synthetic and real-world datasets.
Revised: We conduct extensive experiments on both synthetic and real-world datasets.

Original: Experimental results show that our proposed FlowX outperforms existing methods signiﬁcantly and consistently.
Revised: Experimental results show that our proposed FlowX outperforms existing methods signiﬁcantly and consistently.

Original: Both quantitative and qualitative studies demonstrate that our proposed FlowX leads to improved explainability of GNNs.
Revised: Both quantitative and qualitative studies demonstrate that our proposed FlowX leads to improved explainability of GNNs.

Original:  R ELATED W ORK 2. G RAPH N EURAL N ETWORKS
Revised:  R ELATED W ORK 2. G RAPH N EURAL N ETWORKS

Original: With the advances of deep learning, several graph neural network approaches have been proposed to solve graph tasks, including graph convolutional networks (GCNs) (Kipf & Welling, 2017), graph attention networks (GATs) (Veliˇckovi´c et al., 2018), and graph isomorphism networks (GINs)
Revised: With the advances of deep learning, several graph neural network approaches have been proposed to solve graph tasks, including graph convolutional networks (GCNs) (Kipf & Welling, 2017), graph attention networks (GATs) (Veliˇckovi´c et al., 2018), and graph isomorphism networks (GINs)

Original: (Xu et al., 2019), etc.
Revised: (Xu et al., 2019), etc.

Original: They generally follow a message-passing framework to learn graph node features.
Revised: They generally follow a message-passing framework to learn graph node features.

Original: Speciﬁcally, the new features of a target node are learned by aggregating messages ﬂows passed from its neighboring nodes.
Revised: Speciﬁcally, the new features of a target node are learned by aggregating messages ﬂows passed from its neighboring nodes.

Original: Without loss of generality, we consider the input graph as a directed graph with n nodes and m edges.
Revised: Without loss of generality, we consider the input graph as a directed graph with n nodes and m edges.

Original: The graph is denoted as G =
Revised: The graph is denoted as G =

Original: ( V, E ) , where V = { v 1 , . .
Revised: ( V, E ) , where V = { v 1 , . .

Original: . , v n } denotes nodes, and E = { e ij } represents edges in which e ij is the directed edge v i → v j .
Revised: . , v n } denotes nodes, and E = { e ij } represents edges in which e ij is the directed edge v i → v j .

Original: Then it can be represented by a feature matrix X ∈ R d × n and an adjacency matrix A ∈ R n × n .
Revised: Then it can be represented by a feature matrix X ∈ R d × n and an adjacency matrix A ∈ R n × n .

Original: Each node v i is associated with a d -dimensional feature vector x i corresponding to the i -th column of X .
Revised: Each node v i is associated with a d -dimensional feature vector x i corresponding to the i -th column of X .

Original: The element a ij in A represents the weight of e ij , and a ij = 0 indicates e ij does not exist.
Revised: The element a ij in A represents the weight of e ij , and a ij = 0 indicates e ij does not exist.

Original: For the t -th layer in GNNs, the message aggregation procedures can be mathematically written as a two-step computation as 
Revised: For the t -th layer in GNNs, the message aggregation procedures can be mathematically written as a two-step computation as [Equation] Combine:

Original: [Equation]
Revised: [Equation]

Original: where X t ∈ R d t × n denotes the node feature matrix computed by the t -th GNN layer and X 

Revised: where X t ∈ R d t × n denotes the node feature matrix computed by the t -th GNN layer and X

Original: = X .
Revised: = X .

Original: Here M t ( · ) denotes the node feature transformation function at layer t and ˆ A t is the connectivity matrix at layer t .
Revised: Here M t ( · ) denotes the node feature transformation function at layer t and ˆ A t is the connectivity matrix at layer t .

Original: Note that we name the elements in ˆ A t as layer edges and ˆ a tjk indicates the layer edge connecting node j and k in layer t .
Revised: Note that we name the elements in ˆ A t as layer edges and ˆ a tjk indicates the layer edge connecting node j and k in layer t .

Original: For example, in GCNs, the transformations are deﬁned as σ ( W t S t ) and ˆ A t =
Revised: For example, in GCNs, the transformations are deﬁned as σ ( W t S t ) and ˆ A t =

Original: D − 12 ( A + I ) D − 12 where W t ∈ R d t × d t − 1 is a trainable weight matrix, σ ( · ) denotes the activation function, I is an identity matrix to add self-loops to the adjacency matrix, and
Revised: D − 12 ( A + I ) D − 12 where W t ∈ R d t × d t − 1 is a trainable weight matrix, σ ( · ) denotes the activation function, I is an identity matrix to add self-loops to the adjacency matrix, and

Original: D denotes the diagonal node degree matrix.
Revised: D denotes the diagonal node degree matrix.

Original: We can stack T GNN layers on top of each other to form a T -layer network, and the network function can be expressed as
Revised: We can stack T GNN layers on top of each other to form a T -layer network, and the network function can be expressed as

Original: [Equation]
Revised: [Equation]

Original: When f ( G ) is a graph classiﬁcation model, g ( · ) generally consists of a readout function, such as global mean pooling, and a multi-layer perceptron (MLP) graph classiﬁer.
Revised: When f ( G ) is a graph classiﬁcation model, g ( · ) generally consists of a readout function, such as global mean pooling, and a multi-layer perceptron (MLP) graph classiﬁer.

Original: Meanwhile, when f ( G ) is a node classiﬁcation model, g ( · ) represents a MLP node classiﬁer.
Revised: Meanwhile, when f ( G ) is a node classiﬁcation model, g ( · ) represents a MLP node classiﬁer.

Original: 2. E XPLAINABILITY OF G RAPH N EURAL N ETWORKS
Revised: 2. E XPLAINABILITY OF G RAPH N EURAL N ETWORKS

Original: A major limitation of GNNs is their lack of explainability.
Revised: A major limitation of GNNs is their lack of explainability.

Original: Thus, different methods have been proposed to explain the predictions of GNNs, such as GraphLime (Huang et al., 2020), GNNExplainer
Revised: Thus, different methods have been proposed to explain the predictions of GNNs, such as GraphLime (Huang et al., 2020), GNNExplainer

Original: (Ying et al., 2019), PGExplainer (Luo et al., 2020), PGMExplainer (Vu & Thai, 2020), SubgraphX (Yuan et al., 2021), XGNN (Yuan et al., 2020b), and GraphSVX (Duval & Malliaros, 2021).
Revised: (Ying et al., 2019), PGExplainer (Luo et al., 2020), PGMExplainer (Vu & Thai, 2020), SubgraphX (Yuan et al., 2021), XGNN (Yuan et al., 2020b), and GraphSVX (Duval & Malliaros, 2021).

Original: These methods can be mainly grouped into four categories based on the views of their explanations.
Revised: These methods can be mainly grouped into four categories based on the views of their explanations.

Original: First, several techniques provide explanations by identifying important nodes in the input graph.
Revised: First, several techniques provide explanations by identifying important nodes in the input graph.

Original: For example,
Revised: For example,

Original: GradCAM (Pope et al., 2019) measures node importance by combining the hidden features and gradients; LRP (Baldassarre & Azizpour, 2019) and Excitation BP (Pope et al., 2019) decompose the predictions into several terms and assign these terms to different nodes; PGM-Explainer (Vu & Thai, 2020) builds a probabilistic graphical model by randomly perturbing the node features and employs an interpretable Bayesian network to generate explanations.
Revised: GradCAM (Pope et al., 2019) measures node importance by combining the hidden features and gradients; LRP (Baldassarre & Azizpour, 2019) and Excitation BP (Pope et al., 2019) decompose the predictions into several terms and assign these terms to different nodes; PGM-Explainer (Vu & Thai, 2020) builds a probabilistic graphical model by randomly perturbing the node features and employs an interpretable Bayesian network to generate explanations.

Original: Second, several existing methods, such as GNNExplainer (Ying et al., 2019), PGExplainer (Luo et al., 2020), and GraphMask (Schlichtkrull et al., 2021), explain GNNs by studying the importance of different graph edges.
Revised: Second, several existing methods, such as GNNExplainer (Ying et al., 2019), PGExplainer (Luo et al., 2020), and GraphMask (Schlichtkrull et al., 2021), explain GNNs by studying the importance of different graph edges.

Original: These methods follow a similar high-level idea that learns masks to identify important edges while maximizing the mutual information.
Revised: These methods follow a similar high-level idea that learns masks to identify important edges while maximizing the mutual information.

Original: Next, the recent study SubgraphX (Yuan et al., 2021) proposes to explain
Revised: Next, the recent study SubgraphX (Yuan et al., 2021) proposes to explain

Original: GNNs via subgraphs.
Revised: GNNs via subgraphs.

Original: It incorporates the Monte Carlo tree search algorithm to explore subgraphs and employs Shapley values to measure the importance.
Revised: It incorporates the Monte Carlo tree search algorithm to explore subgraphs and employs Shapley values to measure the importance.

Original: Finally, XGNN (Yuan et al., 2020b) focuses on model-level explanations, which can provide high-level insights and general understanding.
Revised: Finally, XGNN (Yuan et al., 2020b) focuses on model-level explanations, which can provide high-level insights and general understanding.

Original: It proposes to generate graph patterns that can maximize a certain model prediction.
Revised: It proposes to generate graph patterns that can maximize a certain model prediction.

Original: While those methods explain GNNs from different views, none of them can provide explanations in terms of message ﬂows and we believe message ﬂows are more natural for performing explainability.
Revised: While those methods explain GNNs from different views, none of them can provide explanations in terms of message ﬂows and we believe message ﬂows are more natural for performing explainability.

Original: To the best of our knowledge, GNN-LRP (Schnake et al., 2020) is the only algorithm that explains
Revised: To the best of our knowledge, GNN-LRP (Schnake et al., 2020) is the only algorithm that explains

Original: GNNs by relevant walks.
Revised: GNNs by relevant walks.

Original: The relevant walk is deﬁned as a T -length ordered edge sequence that corresponds to a T -step directed path on the input graph.
Revised: The relevant walk is deﬁned as a T -length ordered edge sequence that corresponds to a T -step directed path on the input graph.

Original: To study walk explainability, GNN-LRP considers the GNN prediction as a function and decomposes it using higher-order Taylor expansions to distribute prediction scores to relevant walks.
Revised: To study walk explainability, GNN-LRP considers the GNN prediction as a function and decomposes it using higher-order Taylor expansions to distribute prediction scores to relevant walks.

Original: Speciﬁcally, by using T -order Taylor expansion with a proper root, each term in the Taylor expansion corresponds to a relevant walk and is regarded as the importance score.
Revised: Speciﬁcally, by using T -order Taylor expansion with a proper root, each term in the Taylor expansion corresponds to a relevant walk and is regarded as the importance score.

Original: While our proposed FlowX shares a similar explanation target, i.e., ﬂow/walk, with GNN-LRP, our method is fundamentally different.
Revised: While our proposed FlowX shares a similar explanation target, i.e., ﬂow/walk, with GNN-LRP, our method is fundamentally different.

Original: The GNN-LRP is developed based on score decomposition while our method employs the concept of Shapley values from cooperative game theory as initial assessments and proposes a learning-based algorithm for reﬁnement.
Revised: The GNN-LRP is developed based on score decomposition while our method follows the phylosopy of Shapley values from cooperative game theory as initial assessments and proposes a learning-based algorithm for the score generation.

Original: In addition,

Original:  GNN-LRP has several constraints on the activation function and bias term used in GNNs while our method can be applied to general GNN models.
Revised: In addition, GNN-LRP has several constraints on the activation function and bias term used in GNNs while our method can be applied to general GNN models.

Original: Furthermore, as the GNN-LRP follows the
Revised: Furthermore, as the GNN-LRP follows the

Original: Gradient × Input scheme, it may not pass the model parameter randomization test and maybe not  sensitive to model parameters (Adebayo et al., 2018).
Revised: Gradient × Input scheme, it may not pass the model parameter randomization test and may not be sensitive to model parameters (Adebayo et al., 2018).

Original: Differences with Other Methods Using Shapley Values: Shapley values are commonly used in explaining machine learning methods.
Revised: Differences with Other Methods Using Shapley Values: Shapley values are commonly used in explaining machine learning methods.

Original: In particular, a recent study proposes a surrogate method, known as GraphSVX (Duval & Malliaros, 2021), to explain GNNs with both node and feature masks.
Revised: In particular, a recent study proposes a surrogate method, known as GraphSVX (Duval & Malliaros, 2021), to explain GNNs with both node and feature masks.

Original: Another recent study proposes SubgraphX
Revised: Another recent study proposes SubgraphX

Original: (Yuan et al., 2021), which employs a search algorithm to explore and identify subgraphs with high Shapley scores.
Revised: (Yuan et al., 2021), which employs a search algorithm to explore and identify subgraphs with high Shapley scores.

Original: While both methods use Shapley values, there are several fundamental differences.
Revised: While these methods use Shapley values, there are several fundamental differences.

Original: First, our proposed method focuses on explaining message ﬂows, which are the most basic and natural units for explanations as GNNs are based on message passing schemes.
Revised: First, our proposed method focuses on explaining message ﬂows, which are the most basic and natural units for explanations as GNNs are based on message passing schemes.

Original: Second, we only use Shapley values as initial approximations, which are subsequently reﬁned by our proposed learning based methods.
Revised: Second, we only use Shapley-like values as initial approximations to facilitating further training.

Original: Experiments show that the reﬁnement step is very important.
Revised: Experiments show that the learning step is very important.

Original: Furthermore, the fundamental difference with GraphSVX is reﬂected in the fact that, while our method is a perturbation-based method, GraphSVX is a surrogate method (Yuan et al., 2020c).
Revised: Furthermore, the fundamental difference with GraphSVX is reﬂected in the fact that, while our method is a perturbation-based method, GraphSVX is a surrogate method (Yuan et al., 2020c).

Original: Due to these differences, we show in experiments that our FlowX achieves more natural and improved performance as compared with other methods.
Revised: Due to these differences, we show in experiments that our FlowX achieves more natural and improved performance as compared with other methods.

Original:  T HE P ROPOSED F LOW X
Revised:  T HE P ROPOSED F LOW X

Original: While existing methods mainly focus on explaining GNNs with graph nodes, edges, or subgraphs, we propose to study the explainability of GNNs from the view of message ﬂows.
Revised: While existing methods mainly focus on explaining GNNs with graph nodes, edges, or subgraphs, we propose to study the explainability of GNNs from the view of message ﬂows.

Original: We argue that message ﬂows are the fundamental building blocks of GNNs and it is natural to study their contributions towards GNN predictions.
Revised: We argue that message ﬂows are the fundamental building blocks of GNNs and it is natural to study their contributions towards GNN predictions.

Original: With our message ﬂow framework, we propose a novel method, known as FlowX, to investigate the importance of different message ﬂows.
Revised: With our message ﬂow framework, we propose a novel method, known as FlowX, to investigate the importance of different message ﬂows.

Original: Speciﬁcally, we adopt the concept of Shapley values (Kuhn & Tucker, 1953) from game theory and propose an  approximation scheme for them.
Revised: Speciﬁcally, we follow the phylosophy of Shapley values (Kuhn & Tucker, 1953) from game theory and propose an marginal contribution approximation scheme for them.

Original: In addition, a learning-based algorithm is proposed to improve the explainability of message ﬂows.
Revised: In addition, a learning-based algorithm is proposed to improve the explainability of message ﬂows.

Original: 3. A M ESSAGE F LOW V IEW OF GNN S
Revised: 3. A M ESSAGE F LOW V IEW OF GNN S

Original: We consider a deep graph model with T GNN layers.
Revised: We consider a deep graph model with T GNN layers.

Original: Each GNN layer aggregates 1-hop neighboring information to learn new node embeddings.
Revised: Each GNN layer aggregates 1-hop neighboring information to learn new node embeddings.

Original: Hence, for any node, the outgoing messages are transmitted within its T -hop neighbors.
Revised: Hence, for any node, the outgoing messages are transmitted within its T -hop neighbors.

Original: Then the outputs of GNNs can be regarded as a function of such transmitted T -step messages, which are named as message ﬂows in this work.
Revised: Then the outputs of GNNs can be regarded as a function of such transmitted T -step messages, which are named as message ﬂows in this work.

Original: Formally, we introduce the concept of message ﬂows and message carriers as follow:
Revised: Formally, we introduce the concept of message ﬂows and message carriers as follow:

Original: Deﬁnition 1: Message Carrier.
Revised: Deﬁnition 1: Message Carrier.

Original: We use the connectivity matrix to represent the carriers for message ﬂows.
Revised: We use the connectivity matrix to represent the carriers for message ﬂows.

Original: Given the connectivity matrix ˆ A t at layer t , the layer edge ˆ a tij represents the message carrier with which the message passes from node v i to v j at layer t .
Revised: Given the connectivity matrix ˆ A t at layer t , the layer edge ˆ a tij represents the message carrier with which the message passes from node v i to v j at layer t .

Original: Note that we use the superscript t to distinguish the message carriers in different layers since their corresponding message ﬂows are different.
Revised: Note that we use the superscript t to distinguish the message carriers in different layers since their corresponding message ﬂows are different.

Original: Then the set of all message carriers, i.e., all layer edges, is deﬁned as A = {· · · , ˆ a 1 uv , · · · , ˆ a tuv , · · · , ˆ a Tuv , · · · } and | A | = | E | × T .
Revised: Then the set of all message carriers, i.e., all layer edges, is deﬁned as A = {· · · , ˆ a 1 uv , · · · , ˆ a tuv , · · · , ˆ a Tuv , · · · } and | A | = | E | × T .

Original: Deﬁnition 2: Message Flow.
Revised: Deﬁnition 2: Message Flow.

Original: In a T -layer GNN model, we use F ijk...(cid:96)m to denote the message ﬂow that starts from node v i in the input layer, and sequentially passes the message to node v j , v k , . .
Revised: In a T -layer GNN model, we use F ijk...(cid:96)m to denote the message ﬂow that starts from node v i in the input layer, and sequentially passes the message to node v j , v k , . .

Original: , v (cid:96) until to node v m in the ﬁnal layer T .
Revised: , v (cid:96) until to node v m in the ﬁnal layer T .

Original: The corresponding message carriers can be represented as { ˆ a 1 ij , ˆ a 2 jk , . . .
Revised: The corresponding message carriers can be represented as { ˆ a 1 ij , ˆ a 2 jk , . . .

Original: , ˆ a T(cid:96)m } .
Revised: , ˆ a T(cid:96)m } .

Original: In a T -layer GNN model, all message ﬂows start from the input layer and end with the ﬁnal layer so that their lengths equal to T .
Revised: In a T -layer GNN model, all message ﬂows start from the input layer and end with the ﬁnal layer so that their lengths equal to T .

Original: For the ease of notations, we introduce the wildcard ∗ to represent any valid node sequence and F to denote the message ﬂow set.
Revised: For the ease of notations, we introduce the wildcard ∗ to represent any valid node sequence and F to denote the message ﬂow set.

Original: For example, we can use F ij ∗ , F ∗ (cid:96)m , and F ij ∗ (cid:96)m to denote the message ﬂow sets that share the same message carrier ˆ a 1 ij , ˆ a T(cid:96)m , or both of them, respectively.
Revised: For example, we can use F ij ∗ , F ∗ (cid:96)m , and F ij ∗ (cid:96)m to denote the message ﬂow sets that share the same message carrier ˆ a 1 ij , ˆ a T(cid:96)m , or both of them, respectively.

Original: In addition, we employ another wildcard ?
Revised: In addition, we employ another wildcard ?

Original: to denote any single node and ? { t } to represent any valid node sequence with t nodes.
Revised: to denote any single node and ? { t } to represent any valid node sequence with t nodes.

Original: For example, F ? { 3 } means the set of valid 2-step message ﬂow with 3 nodes.
Revised: For example, F ? { 3 } means the set of valid 2-step message ﬂow with 3 nodes.

Original: Note that the following property of message ﬂow sets also holds:
Revised: Note that the following property of message ﬂow sets also holds:

Original: [Equation]
Revised: [Equation]

Original: The ﬁnal embeddings of node v m are determined by all incoming message ﬂows to node v m , which can be denoted as F ∗ m .
Revised: The ﬁnal embeddings of node v m are determined by all incoming message ﬂows to node v m , which can be denoted as F ∗ m .

Original: Since the output of the GNN model is obtained based on the ﬁnal node embeddings, then it is reasonable to treat the GNN output as the combinations of different message ﬂows.
Revised: Since the output of the GNN model is obtained based on the ﬁnal node embeddings, then it is reasonable to treat the GNN output as the combinations of different message ﬂows.

Original: Hence, it is natural to demystify GNN models by studying the importance of different message ﬂows towards GNN predictions.
Revised: Hence, it is natural to demystify GNN models by studying the importance of different message ﬂows towards GNN predictions.

Original: 3. S HAPLEY V ALUES AS I NITIAL A SSESSMENTS
Revised: 3. S AMPLING MARGINAL CONTRIBUTIONS AS I NITIAL A SSESSMENTS

Original: While explaining GNNs with message ﬂows seems to be promising, it is still crucial to properly measure the importance of those message ﬂows.
Revised: While explaining GNNs with message ﬂows seems to be promising, it is still crucial to properly measure the importance of those message ﬂows.

Original: Hence, in this work, our FlowX proposes to adopt the concept of Shapley value (Kuhn & Tucker, 1953)  as the initial assessments of ﬂow importance.
Revised: Hence, in this work, our FlowX proposes to follow the philosophy of Shapley value (Kuhn & Tucker, 1953) to use the marginal contributions in different ﬂow sets as the initial assessments of ﬂow importance.

Revised: It is noteworthy that our initial assessments are not computing real Shapley values.

Original: Note that Shapley value is a solution concept in cooperative game theory and used to fairly assign the game gain to different players.
Revised: Shapley value is a solution concept in cooperative game theory and used to fairly assign the game gain to different players.

Original: When applying it to GNN explanation tasks, we treat the message ﬂows as different players and the GNN prediction score as the total game gain.
Revised: When considering marginal contributions in GNN explanation tasks, we treat the message ﬂows as different players and the GNN prediction score as the total game gain.

Original: Formally, given the trained GNN model f ( · ) and the input graph G , we use F ∗ to denote the set of all valid ﬂows, i.e. , all players in the game.
Revised: Formally, given the trained GNN model f ( · ) and the input graph G , we use F ∗ to denote the set of all valid ﬂows, i.e. , all players in the game.

Original: Then the Shapley value of ﬂow F k can be mathematically written as
Revised: Then given any ﬂow F k , we mathematically deﬁne the contribution of it as

Original: [Equation]
Revised: [Equation]

Original: where  P denotes the possible coalition group of players,| · | is the set size, and φ ( · ) denotes the Shapley score.
Revised: where W ( | P | ) is a weight function assigned to each term in the summation; P denotes the possible coalition group of players, and φ ( · ) denotes the ﬂow score.

Original: In addition, f (cid:0) P ∪ {F k } (cid:1) − f ( P ) is the marginal contribution of ﬂow F k for a particular coalition group, which can be computed by the prediction difference between combining 
Revised: Here f (cid:0) P ∪ {F k } (cid:1) − f ( P ) is the marginal contribution of ﬂow F k for a particular coalition group, which can be computed by the prediction difference between combining F k with the coalition group P and only using P . Note that

Original: F k with the coalition group P and only using P .
Revised: Eq. 4 is equivalent to the classic Shapley value when W ( | P | ) = | P | ! ( | F ∗ |−| P |− 1)! | F ∗ | !

Original:  To compute the Shapley score φ ( F k ) , we need to enumerate all possible coalition groups and considers different interactions among players.
Revised: where | · | is the set size. To compute the ﬂow score φ ( F k ) , we need to enumerate all possible coalition groups and considers different interactions among players.

Original: Note that Shapley value is a fair solution since it is  the only solution satisfying four desirable properties, including efﬁciency, symmetry, linearity, and dummy (Lundberg & Lee, 2017).
Revised: However, it is time-consuming to consider all possible coalition; thus, we sample several marginal contributions to approximate the ﬁnal ﬂow score.

Original: However, it is not possible to compute Shapley values with Eq.
Revised: Note that it is not possible to compute importance score with Eq.

Original: (4) for message ﬂows since we cannot remove individual message ﬂows from the GNN model and the ﬁnest component we can directly remove is the layer edge.
Revised: (4) for message ﬂows since we cannot remove individual message ﬂows from the GNN model and the ﬁnest component we can directly remove is the layer edge.

Original: For example, when removing the layer edge ˆ a 1 ij , the whole ﬂow set F ij ∗ is removed from the model.
Revised: For example, when removing the layer edge ˆ a 1 ij , the whole ﬂow set F ij ∗ is removed from the model.

Original: In addition, enumerating all possible coalition groups is time-consuming when the input graph is large-scale and the GNN model is deep.
Revised: In addition, enumerating all possible coalition groups is time-consuming when the input graph is large-scale and the GNN model is deep.

Original: Hence, in our FlowX, we propose an approximation scheme to compute Shapley values based on Monte Carlo (MC) sampling (Štrumbelj & Kononenko, 2014).
Revised: Hence, in our FlowX, we propose an approximation scheme to compute Eq.

Original: Formally, let M denote the total number of MC sampling steps. For the i -th sampling step, we use π ( i ) to represent a random permutation of | E | × T elements. Then the set of all layer edge A is permuted based on π ( i ) , denoted as A π ( i ) that A π ( i ) [ j ] =
Revised: 4 based on Monte Carlo (MC) sampling (Štrumbelj &

Original: A [ π ( i ) [ j ]] .
Revised: Kononenko, 2014).

Original: In each step i , we iteratively

Original: Iterations 1 v 2 v 3 v 4 v 1 v 2 v 3 v 4 v 1 st iteration 2 nd iteration
Revised: Iterations 1 v 2 v 3 v 4 v 1 v 2 v 3 v 4 v 1 st iteration 2 nd iteration

Original: [Figure]
Revised: Flows

Original: 1: Given a trained GNN model f ( · ) and an input graph, the set of all layer edges is represented as A .
Revised: 1: Given a trained GNN model f ( · ) and an input graph, the set of all layer edges is represented as A .

Original: For each message ﬂow F k , two | A | -dimensional vectors S ( F k ) and C ( F k ) denote its importance scores and removing index counts respectively.
Revised: For each message ﬂow F k , two | A | -dimensional vectors S ( F k ) and C ( F k ) denote its importance scores and removing index counts respectively.

Original: In addition, (cid:98) A denotes the set of removed layer edges and (cid:98) F is the set of removed message ﬂows
Revised: In addition, (cid:98) A denotes the set of removed layer edges and (cid:98) F is the set of removed message ﬂows.

Original: Initialize
Revised: Initialize

Original:  all message ﬂows
Revised: ) and all message Monte

Original: sampling
Revised: sampling

Original: do
Revised: do

Original: Initialize the removed sets
Revised: Initialize the removed sets

Original:  Randomly shufﬂe and permute 
Revised: empty Randomly shufﬂe and permute the layer edge set A , denoted

Original: the
Revised: as A π ( i

Original: layer edge j
Revised: for j

Original:  1 to |  |
Revised: from 1 to | A |

Original: do 7:
Revised: do 7:

Original: Select the j -th layer edge in A π ( i ) , denoted as ˆ a t(cid:96)m .
Revised: Select the j -th layer edge in A π ( i ) , denoted as ˆ a t(cid:96)m .

Original: Block the layer edge ˆ a t(cid:96)m in GNN model f ( · ) , then the removed ﬂows are (cid:98) F j =
Revised: Block the layer edge ˆ a t(cid:96)m in GNN model f ( · ) , then the removed ﬂows are (cid:98) F j =

Original: F ? { t − 1 } (cid:96)m ? { T − t } \ ( (cid:98) F ∩ F ? { t − 1 } (cid:96)m ?
Revised: F ? { t − 1 } (cid:96)m ? { T − t } \ ( (cid:98) F ∩ F ? { t − 1 } (cid:96)m ?

Original: { T − t } ) .
Revised: { T − t } ) .

Original: Compute the prediction difference that s j = f
Revised: Compute the prediction difference that s j = f

Original: ( A \ (cid:98) A )
Revised: ( A \ (cid:98) A )

Original: − f ( A \ (
Revised: − f ( A \ (

Original: (cid:98) A ∪ ˆ a t(cid:96)m )) .
Revised: (cid:98) A ∪ ˆ a t(cid:96)m )) .

Original: Update (cid:98)
Revised: Update (cid:98)

Original: A = (cid:98)
Revised: A = (cid:98)

Original: A ∪ ˆ a t(cid:96)m and (cid:98) F = (cid:98) F ∪ F ? { t − 1 } (cid:96)m ?
Revised: A ∪ ˆ a t(cid:96)m and (cid:98) F = (cid:98) F ∪ F ? { t − 1 } (cid:96)m ?

Original: { T − t } .
Revised: { T − t } .

Original: Compute averaged score that ¯ s j
Revised: Compute averaged score that ¯ s j

Original: = s j / | (cid:98) F j | .
Revised: = s j / | (cid:98) F j | .

Original: 12: for each ﬂow F k in (cid:98) F j do 13:
Revised: 12: for each ﬂow F k in (cid:98) F j do 13:

Original: Update S ( F k )[ j ] = S ( F k )
Revised: Update S ( F k )[ j ] = S ( F k )

Original: [ j ] + ¯ s j .
Revised: [ j ] + ¯ s j .

Original: Update C ( F k ) 
Revised: Update C ( F k ) =

Original: [ j ] = C ( F k )[ j ] +15: end for 16: end for 17: end for 18: For each message ﬂow F k , compute our Shapley approximations S ( F k )
Revised: C ( F k ) +15: end for 16: end for 17: end for 18: For each message ﬂow F k , compute our marginal contribution vector S ( F k )

Original: =
Revised: =

Original: S ( F k ) /C 
Revised: S ( F k ) /C ( F k )

Revised: Formally, let M denote the total number of MC sampling steps.

Revised: For the i -th sampling step, we use π ( i ) to represent a random permutation of | A | = | E | × T elements.

Revised: Then the set of all layer edge A is permuted based on π ( i ) , denoted as A π ( i ) that A π ( i )

Revised: [ j ] =

Original:  ( F k )  remove one layer edge from A π ( i ) following the order, and compute the marginal contribution.
Revised: A [ π ( i ) [ j ]] . In each step i , we iteratively remove one layer edge from A π ( i ) following the order, and compute the marginal contribution.

Original: An illustration of our proposed Shapley value approximations is shown in Figure 1.
Revised: An illustration of our proposed sampling algorithm is shown in Figure 1.

Original: We use (cid:98) A to denote the set of removed layer edges and (cid:98) F to denote the set of removed message ﬂows, which are initialized as (cid:98) A = ∅ and (cid:98) F = ∅ in the beginning of each MC sampling step.
Revised: We use (cid:98) A to denote the set of removed layer edges and (cid:98) F to denote the set of removed message ﬂows, which are initialized as (cid:98) A = ∅ and (cid:98) F = ∅ in the beginning of each MC sampling step.

Original: For the iteration j in sampling step i , the j -th element of A π ( i ) is removed and we assume the removed layer edge is ˆ a t(cid:96)m .
Revised: For the iteration j in sampling step i , the j -th element of A π ( i ) is removed and we assume the removed layer edge is ˆ a t(cid:96)m .

Original: Then the computation operations can be mathematically written as
Revised: Then the computation operations can be mathematically written as

Original: [Equation]
Revised: [Equation]

Revised: [Equation]

Revised: [Equation]

Revised: [Equation]

Revised: [Equation]

Revised: −

Revised: [Equation]

Revised: { T

Revised: [Equation]

Revised: t }

Revised: [Equation]

Revised: (7) (cid:98)

Revised: [Equation]

Revised: (cid:98) F

Revised: [Equation]

Revised: F ? { t −

Original:  where (cid:98) F j denotes the removed message ﬂows in iteration j by removing ˆ a t(cid:96)m .
Revised: [Equation] { T [Equation] t } , (8) where (cid:98) F j denotes the removed message ﬂows in iteration j by removing ˆ a t(cid:96)m .

Original: Note that it is not equivalent to F ? {
Revised: Note that it is not equivalent to F ? {

Original: t − 1 } (cid:96)m ?
Revised: t − 1 } (cid:96)m ?

Original: { T − t } since the ﬂows in F ? { t − 1 } (cid:96)m ?
Revised: { T − t } since the ﬂows in F ? { t − 1 } (cid:96)m ?

Original: { T − t } may be already removed in the previous iterations.
Revised: { T − t } may be already removed in the previous iterations.

Original: Then the score is averaged that ¯ s j = s j / | (cid:98) F j | and we assign ¯ s j to each ﬂow in (cid:98) F j .
Revised: Then the score is averaged that ¯ s j = s j / | (cid:98) F j | and we assign ¯ s j to each ﬂow in (cid:98) F j .

Original: By repeating such operations until all layer edges are removed, we can obtain | A | marginal scores and assign them to the corresponding ﬂows.
Revised: By repeating such operations until all layer edges are removed, we can obtain | A | marginal scores and assign them to the corresponding ﬂows.

Original: Note that the order information in A π ( i ) is important since in the earlier iterations, the removed ﬂows (cid:98) F j are interacting with a larger coalition group
Revised: Note that the order information in A π ( i ) is important since in the earlier iterations, the removed ﬂows (cid:98) F j are interacting with a larger coalition group

Original: A \ ( (cid:98) A ∪ ˆ a t(cid:96)m ) and in the later iterations the coalition groups are small.
Revised: A \ ( (cid:98) A ∪ ˆ a t(cid:96)m ) and in the later iterations the coalition groups are small.

Original: Altogether, the steps of our Shapley approximations are shown in Algorithm 1.
Revised: Altogether, the steps of our marginal contribution sampling are shown in Algorithm 1.

Original: After M MC sampling steps, our method explores M permutations of layer edge set and each ﬂow is assigned with M importance scores.
Revised: After M MC sampling steps, our method explores M permutations of layer edge set and each ﬂow is sampled to obtain M marginal contributions.

Original: For each ﬂow F k , we use a | A | -dimensional vector to store its importance scores, denoted as S ( F k ) , where S ( F k )
Revised: For each ﬂow F k , we use a | A | -dimensional vector to store its marginal contributions, denoted as S ( F k ) , where S ( F k )

Original: [ j ] is the importance score obtained when F k is removed in the iteration j .
Revised: [ j ] is the importance score obtained when F k is removed in the iteration j .

Original: If the ﬂow is removed mutiple times in the iteration j for different MC sampling steps, the scores are averaged.
Revised: If the ﬂow is removed mutiple times in the iteration j for different MC sampling steps, the scores are averaged, which also means scores obtained from coalitions with same sizes are weighed equally.

Original: Then it raises the question that how to convert S ( F k ) to the ﬁnal importance score of F k .
Revised: Then it raises the question that how to convert S ( F k ) to the ﬁnal importance score of F k .

Revised: The ﬁnal importance score will be a Shapley value approximation if we simply compute the summation of the elements in S ( F k ) .

Original: While Shapley values are promising, our approximations may not be accurate since M is always set as M << | A | !
Revised: While Shapley values are promising, the approximations may not be accurate since M is always set as M << | A | !

Original: for the sake of computation efﬁciency.
Revised: for the sake of computation efﬁciency.

Revised: Moreover, the importance of the early iterations when more layer edges exist in the model should be different to the importance of later iterations when only few layer edges exist in diverse situations.

Original: Hence, directly averaging the scores in S ( F k ) may not  suitable.
Revised: Hence, directly computing the summation of the scores in S ( F k ) may not be suitable.

Original: 3. L EARNINGTO R EFINE I MPORTANCE S CORES
Revised: 3. L EARNING I MPORTANCE S CORES

Original: With our obtained Shapley value approximations S ( F k ) , we propose to regard these values as initial assessments of ﬂow importance and learn to reﬁne such importance scores.
Revised: With our obtained score vectors S ( F k ) , we propose to regard these values as initial assessments of ﬂow importance and learn an associated importance scores with designed redistribution trick.

Original: We propose to learn weights for combining the elements in S ( F k ) to obtain the ﬁnal importance score for each ﬂow.
Revised: We target to learn weights for combining the elements in S ( F k ) to obtain the ﬁnal importance score for each ﬂow, which means to learn W ( P ) in Eq. 4.

Original: Inspired by the mask-based explanation methods (Ying et al., 2019; Luo et al., 2020), we propose to obtain masks for indicating important layer edges.

Original: Note that since message ﬂows cannot be masked out individually, we can only directly focus on the masks of layer edges and indirectly study the message ﬂows.

Original: Formally, given the input graph G and GNN model f ( · ) , we obtain Shapley value approximations
Revised: Formally, given the input graph G and GNN model f ( · ) , we obtain marginal contribution vector

Original: S ( F k ) for each message ﬂow.
Revised: S ( F k ) for each message ﬂow.

Original: Then a weight vector w is randomly initialized, which shares the same dimension of | S ( F k ) | .
Revised: Then the only trainable weight vector w is initialized to all equal to 0.5 with random noise, which shares the same dimension of | S ( F k ) | .

Original: Note that the weights w are shared among all message ﬂows in the same graph.
Revised: Note that the weights w are shared among all message ﬂows in the same graph.

Original: Thenthe score vectors S ( F k ) can be combined via element-wise multiplication with w .
Revised: Then S ( F k ) can be combined via a dot product with w .

Original: Next, the ﬂow importance can be converted to layer edge importance by simply summing up the scores of all ﬂows sharing a particular layer edge as the message carrier.
Revised: Next, the ﬂow importance is converted to layer edge importance by simply summing up the scores of all ﬂows sharing a particular layer edge as the message carrier.

Original: Mathematically, it can be written as
Revised: Mathematically, it can be written as

Original: [Equation]
Revised: [Equation]

Original: with carrier ˆ
Revised: where s ( · ) denotes the importance score for layer edges and ﬂows; ˜ F = F ? { t − 1 } uv ?

Original: [Equation]
Revised: { T − t } .

Revised: Then based on s (ˆ a tuv ) , we can obtain a mask indicating the importance of different layer edges, denoted as

Revised: M :

Original: [Equation]
Revised: [Equation]

Original:  where  s( · ) denotes the importance score for layer edges and ﬂows.
Revised: (11) where ˆ s denotes all layer edges’ importance scores.

Original: Then based on s (ˆ a tuv ) , we can obtain a mask indicating the importance of different layer edges, denoted as M .
Revised: g ( · ) is deﬁned as an exponential redistribution operation that including an input normalization, an element-wise exponential scaling g e ( x ) = x r (r is a hyper-parameter), and an output normalization.

Original:  Note that the mask is normalized so that each element is in the range [0 , 1] .
Revised: We provide more intuitions in Appendix A.3. Note that the mask is normalized so that each element is in the range [0 , 1] .

Original: By applying the mask to layer edges, important layer edges are blocked and the model prediction becomes
Revised: By applying the mask to layer edges, important layer edges are restricted and the model prediction becomes

Original: [Equation]
Revised: [Equation]

Original: (11) where ˆ y is the prediction vector and Combine ( G , 1 − M ) means layer edges are masked out based on the values of M .
Revised: where ˆ y is the prediction vector and Combine ( G , 1 − M ) means layer edges are masked out based on the values of M .

Original: Intuitively, if important layer edges are removed, then the prediction should change signiﬁcantly.
Revised: Intuitively, if important layer edges are restricted, then the prediction should change signiﬁcantly.

Original: Hence, ˆ y is encouraged to be different from the original prediction by learning proper weights w .
Revised: Hence, ˆ y is encouraged to be different from the original prediction by learning proper weights w .

Original: Note that, the weights w are the only learnable parameters in our reﬁning stage while all other variables are ﬁxed.

Original: Speciﬁcally, we employ the negative cross-entropy loss to learn the weights and it can be formally expressed as
Revised: Speciﬁcally, we employ the negative cross-entropy loss that can be formally expressed as

Original: [Equation]
Revised: [Equation]

Original:  where y is a scalar representing the predicted class of the original graph G , m is the number of classes, 1 {·} denotes the indicator function, and (cid:98) y c is the new predicted probability for class c .
Revised: [Equation] where y is a scalar representing the predicted class of the original graph G , m is the number of classes, 1 {·} denotes the indicator function, and (cid:98) y c is the new predicted probability for class c .

Original: After training, the ﬁnal importance scores for different ﬂows can be obtained via Eq.
Revised: After training, the ﬁnal importance scores for different ﬂows can be obtained via Eq.

Original: Furthermore, we apply exponential scaling on the layer edge mask M to facilitate training and encourage discreteness as we observe that the training is not stable without such a scaling term.

Original: [Table]

Original:  E XPERIMENTAL S TUDIES 4. D ATASETS AND E XPERIMENTAL S ETTINGS
Revised:  E XPERIMENTAL S TUDIES 4. D ATASETS AND E XPERIMENTAL S ETTINGS

Original: Datasets.
Revised: Datasets.

Original: We employ seven different datasets to demonstrate the effectiveness of our proposed
Revised: We employ seven different datasets to demonstrate the effectiveness of our proposed

Original: FlowX with both quantitative studies and qualitative visualization results.
Revised: FlowX with both quantitative studies and qualitative visualization results.

Original: These datasets are BAShapes (Ying et al., 2019), BA-LRP (Schnake et al., 2020), ClinTox (Wu et al., 2018), Tox21 (Wu et al., 2018), BBBP (Wu et al., 2018), BACE (Wu et al., 2018), BA-INFE [Table] and Graph-SST2 (Yuan et al., 2020c), which include both synthetic and real-world data.
Revised: These datasets are BAShapes (Ying et al., 2019), BA-LRP (Schnake et al., 2020), ClinTox (Wu et al., 2018), Tox21 (Wu et al., 2018), BBBP (Wu et al., 2018), BACE (Wu et al., 2018), BA-INFE, and Graph-SST2 (Yuan et al., 2020c), which include both synthetic and real-world data.

Original: First, BA-Shapes is a node-classiﬁcation synthetic dataset that is built by attaching house-like motifs to the base Barab ´ asi-Albert graph where the node labels are determined by their own identiﬁes and localizations in motifs.
Revised: First, BA-Shapes is a node-classiﬁcation synthetic dataset that is built by attaching house-like motifs to the base Barab ´ asi-Albert graph where the node labels are determined by their own identiﬁes and localizations in motifs.

Original: Then, BA-LRP is a graph-classiﬁcation synthetic dataset that includes Barab ´ asi-Albert graphs and the two classes are node-degree concentrated graph and evenly graph.
Revised: Then, BA-LRP is a graph-classiﬁcation synthetic dataset that includes Barab ´ asi-Albert graphs and the two classes are node-degree concentrated graph and evenly graph.

Original: Next, ClinTox, Tox21, BBBP [Table] and BACE are real-world molecule datasets for graph classiﬁcation.
Revised: Next, ClinTox, Tox21, BBBP, and BACE are real-world molecule datasets for graph classiﬁcation.

Original: The chemical molecule graphs in these datasets are labeled according to their chemical properties, such as whether the molecule can penetrate a blood-brain barrier.
Revised: The chemical molecule graphs in these datasets are labeled according to their chemical properties, such as whether the molecule can penetrate a blood-brain barrier.

Original: Finally, Graph-SST2 is a natural language sentimental analysis dataset that converts text data to graphs.
Revised: Finally, Graph-SST2 is a natural language sentimental analysis dataset that converts text data to graphs.

Original: These graphs are labeled by their sentiment meanings.
Revised: These graphs are labeled by their sentiment meanings.

Original: Note that the
Revised: Note that the

Original: BA-INFE dataset is a synthetic dataset for Accuracy comparisons and the results are reported in the appendix.
Revised: BA-INFE dataset is a synthetic dataset for Accuracy comparisons and the results are reported in the appendix.

Original: We also report the properties and statistics of these datasets in  Table 1.
Revised: We also report the properties and statistics of these datasets in Appendix B Table 2.

Original: GNN Models.
Revised: GNN Models.

Original: In our experiments, we consider GCNs (Kipf & Welling, 2017) and GINs (Xu et al., 2019) as our graph models for all datasets.
Revised: In our experiments, we consider GCNs (Kipf & Welling, 2017) and GINs (Xu et al., 2019) as our graph models for all datasets.

Original: We adopt 2-layer GNNs for node classiﬁcation and 3-layer
Revised: We adopt 2-layer GNNs for node classiﬁcation and 3-layer

Original: GNNs for graph classiﬁcation.
Revised: GNNs for graph classiﬁcation.

Original: The graph models are trained to achieve competitive performance and the details are reported in Table 1.
Revised: The graph models are trained to achieve competitive performance and the details are reported in Table 2.

Original: Baselines.
Revised: Baselines.

Original: With the trained graph models, we quantitatively and qualitatively compare our FlowX with eight baselines, including GradCAM (Pope et al., 2019), DeepLIFT (Shrikumar et al., 2017),
Revised: With the trained graph models, we quantitatively and qualitatively compare our FlowX with eight baselines, including GradCAM (Pope et al., 2019), DeepLIFT (Shrikumar et al., 2017),

Original: GNNExplainer (Ying et al., 2019), PGExplainer (Luo et al., 2020), PGMExplainer (Vu & Thai, 2020),
Revised: GNNExplainer (Ying et al., 2019), PGExplainer (Luo et al., 2020), PGMExplainer (Vu & Thai, 2020),

Original: SubgraphX (Yuan et al., 2021), GNN-GI (Schnake et al., 2020), GNN-LRP (Schnake et al., 2020).
Revised: SubgraphX (Yuan et al., 2021), GNN-GI (Schnake et al., 2020), GNN-LRP (Schnake et al., 2020).

Original: Note that since these methods cannot be directly compared, we set the explanation target to graph edges for fair comparisons that the explanations are converted to edge importance scores if needed.
Revised: Note that since these methods cannot be directly compared, we set the explanation target to graph edges for fair comparisons that the explanations are converted to edge importance scores if needed.

Original: More implementation details about explanation methods setting and GNN models can be found in the supplementary material.
Revised: More implementation details about explanation methods setting and GNN models can be found in the supplementary material.

Original: We used the datasets and implementations of the comparing algorithms in the DIG library (Liu et al., 2021).
Revised: We used the datasets and implementations of the comparing algorithms in the DIG library (Liu et al., 2021).

Original: We will release our code after the anonymous review period.
Revised: We will release our code after the anonymous review period.

Original: 4. Q UANTITATIVE S TUDIES
Revised: 4. Q UANTITATIVE S TUDIES

Original: We ﬁrst quantitatively compare different explanation methods.
Revised: We ﬁrst quantitatively compare different explanation methods.

Original: We follow the existing studies (Yuan et al., 2020c; Pope et al., 2019) and employ two metrics to evaluate the explanations: Fidelity and
Revised: We follow the existing studies (Yuan et al., 2020c; Pope et al., 2019) and employ two metrics to evaluate the explanations: Fidelity and

Original: Sparsity.
Revised: Sparsity.

Original: Good explanations should be faithful to the model and capture the discriminative features for the predictions.
Revised: Good explanations should be faithful to the model and capture the discriminative features for the predictions.

Original: When such input features are removed, the original predictions should change signiﬁcantly.
Revised: When such input features are removed, the original predictions should change signiﬁcantly.

Original: The Fidelity score measures the change of predicted probabilities when removing important input features identiﬁed by different explanation methods.
Revised: The Fidelity score measures the change of predicted probabilities when removing important input features identiﬁed by different explanation methods.

Original: Higher Fidelity scores indicate the removed features are more important to the predictions and hence the explanations are more faithful to the model.
Revised: Higher Fidelity scores indicate the removed features are more important to the predictions and hence the explanations are more faithful to the model.

Original: In addition, the other desired property of explanations is sparsity.
Revised: In addition, the other desired property of explanations is sparsity.

Original: To encourage the explanations to be more human-intelligible, they should contain fewer but more important features.
Revised: To encourage the explanations to be more human-intelligible, they should contain fewer but more important features.

Original: Hence, we also employ the Sparsity metric which measures the percentage of input features that are [Figure] [Figure] Tox
Revised: Hence, we also employ the Sparsity metric which measures the percentage of input features that are identiﬁed as important.

Original: BBBP

Original: [Figure]

Original: [Table] [Table] identiﬁed as important. Higher Sparsity scores indicate that fewer features are identiﬁed as important in the explanations.
Revised: Higher Sparsity scores indicate that fewer features are identiﬁed as important in the explanations.

Original: The formulations and details of these metrics are discussed in the appendix B.
Revised: The formulations and details of these metrics are discussed in the appendix C. [Figure]

Original:  It is noteworthy that these two metrics are highly correlated since the predictions tend to change more when more input features are removed.
Revised: [Figure] [Table] It is noteworthy that these two metrics are highly correlated since the predictions tend to change more when more input features are removed.

Original: Then explanations with higher Sparsity scores tend to have lower Fidelity scores.
Revised: Then explanations with higher Sparsity scores tend to have lower Fidelity scores.

Original: Hence, we argue that the Fidelity scores need to be compared under a similar Sparsity level.
Revised: Hence, we argue that the Fidelity scores need to be compared under a similar Sparsity level.

Original: Speciﬁcally, we control the Sparsity scores of explanations and compare the corresponding Fidelity scores.
Revised: Speciﬁcally, we control the Sparsity scores of explanations and compare the corresponding Fidelity scores.

Original: For each dataset, we randomly select 100 samples and conduct such quantitative evaluations.
Revised: For each dataset, we randomly select 100 samples and conduct such quantitative evaluations.

Original: The results are reported in Figure 2 where we show the plots of
Revised: The results are reported in Figure 2 where we show the plots of

Original: Fidelity scores with respect to different Sparsity levels.
Revised: Fidelity scores with respect to different Sparsity levels.

Original: Obviously, FlowX performs better on all of the real-world datasets that FlowX consistently achieves higher Fidelity across all Sparsity levels.
Revised: Obviously, FlowX performs better on all of the real-world datasets that FlowX consistently achieves higher Fidelity across all Sparsity levels.

Original: Meanwhile, the results on synthetic datasets are competitive when the Sparsity is low and our FlowX still performs signiﬁcantly better with high Sparsity levels.
Revised: Meanwhile, the results on synthetic datasets are competitive when the Sparsity is low and our FlowX still performs signiﬁcantly better with high Sparsity levels.

Original: Note that our method is shown to achieve stable performance on different datasets and GNN models, which is promising for the applications and generalizations of our FlowX.
Revised: Note that our method is shown to achieve stable performance on different datasets and GNN models, which is promising for the applications and generalizations of our FlowX.

Original: Meanwhile, our proposed method, GNN-LRP, and GNN-GI perform better than the other methods, which indicates the superiority of the methods based on ﬂows and walks.
Revised: Meanwhile, our proposed method, GNN-LRP, and GNN-GI perform better than the other methods, which indicates the superiority of the methods based on ﬂows and walks.

Original: Note that GNN-LRP achieves good results on synthetic datasets but surprisingly, it is not better than GNN-GI on real-world datasets.
Revised: Note that GNN-LRP achieves good results on synthetic datasets but surprisingly, it is not better than GNN-GI on real-world datasets.

Original: In addition, PGExplainer only obtains competitive results on simple synthetic datasets but not on complex real-world datasets.
Revised: In addition, PGExplainer only obtains competitive results on simple synthetic datasets but not on complex real-world datasets.

Original: Note that, for SubgraphX, its subgraph explanations are not directly comparable so we convert its subgraph explanations to edge explanations.
Revised: Note that, for SubgraphX, its subgraph explanations are not directly comparable so we convert its subgraph explanations to edge explanations.

Original: It performs well on the BA-LRP and BBBP datasets but not as expected on the other datasets.
Revised: It performs well on the BA-LRP and BBBP datasets but not as expected on the other datasets.

Original: We believe the reason may be that such converting destroys the continuousness and completeness of its subgraph explanations.
Revised: We believe the reason may be that such converting destroys the continuousness and completeness of its subgraph explanations.

Original: Last but not least, we wish to mention that the comparisons using edge actually limit the performance of the ﬂow-based methods due to explanation granularity changes.
Revised: Last but not least, we wish to mention that the comparisons using edge actually limit the performance of the ﬂow-based methods due to explanation granularity changes.

Original: More results and time comparisons are also reported in the appendix.
Revised: More results and time comparisons are also reported in the appendix.

Original: 4. A BLATION S TUDIES
Revised: 4. A BLATION S TUDIES

Original: To demonstrate the effectiveness of our proposed learning-based algorithm in Section 3.3, we study the performance when only using the average of our Shapley initial assessments as the importance scores without further learning, denoted as FlowX ∗ .
Revised: To demonstrate the effectiveness of our proposed learning-based algorithm in Section 3.3, we study the performance when only using the average of our Shapley initial assessments as the importance scores without further learning, denoted as FlowX ∗ .

Original: In addition, to show the Shapley initial assessments arenecessary, we study the method only using the learning algorithm with randomly initialized scores
Revised: In addition, to show the Shapley initial assessments are

Original: DeepLIFT
Revised: DeepLIFT

Original: Fidelity: 0.
Revised: Fidelity: 0.

Original: DeepLIFT
Revised: DeepLIFT

Original: Fidelity: 0.
Revised: Fidelity: 0.

Original: GNN-GI
Revised: GNN-GI

Original: Fidelity: 0.
Revised: Fidelity: 0.

Original: GNN-GI
Revised: GNN-GI

Original: Fidelity: 0.
Revised: Fidelity: 0.

Original: GNNExplainer
Revised: GNNExplainer

Original: Fidelity: 0.
Revised: Fidelity: 0.

Original: GNNExplainer
Revised: GNNExplainer

Original: Fidelity: 0.
Revised: Fidelity: 0.

Original: GradCAM
Revised: GradCAM

Original: Fidelity: 0.
Revised: Fidelity: 0.

Original: GradCAM
Revised: GradCAM

Original: Fidelity: 0.
Revised: Fidelity: 0.

Original: GNN-LRP
Revised: GNN-LRP

Original: Fidelity: 0.
Revised: Fidelity: 0.

Original: GNN-LRP
Revised: GNN-LRP

Original: Fidelity: 0.
Revised: Fidelity: 0.

Original: FlowX
Revised: FlowX

Original: Fidelity: 0.
Revised: Fidelity: 0.

Original: FlowX
Revised: FlowX

Original: Fidelity: 0.
Revised: Fidelity: 0.

Original:  for different ﬂows, denoted as FlowX † .
Revised: necessary, we study the method only using the learning algorithm with randomly initialized scores for different ﬂows, denoted as FlowX † .

Original: We compare our FlowX with these two baselines on GCN models, and the results are reported in Table 2.
Revised: We compare our FlowX with these two baselines on GCN models, and the results are reported in Table 1.

Original: Obviously, for all datasets, the FlowX and FlowX ∗ outperform FlowX † signiﬁcantly, showing that the Shapley initial assessments are appropriate and necessary.
Revised: Obviously, for all datasets, the FlowX and FlowX ∗ outperform FlowX † signiﬁcantly, showing that the Shapley initial assessments are appropriate and necessary.

Original: In addition, for synthetic datasets, the FlowX and FlowX ∗ show competitive performance, which indicates our initial assessments can obtain reasonable results for simple models and datasets.
Revised: In addition, for synthetic datasets, the FlowX and FlowX ∗ show competitive performance, which indicates our initial assessments can obtain reasonable results for simple models and datasets.

Original: However, FlowX obtains signiﬁcantly better performance on complex real-world datasets than
Revised: However, FlowX obtains signiﬁcantly better performance on complex real-world datasets than

Original: FlowX ∗ , showing the effectiveness of our proposed learning-based algorithm.
Revised: FlowX ∗ , showing the effectiveness of our proposed learning-based algorithm.

Original: More experimental details will be attached in the supplementary material.
Revised: More experimental details will be attached in the supplementary material.

Original: 4. V ISUALIZATION OF E XPLANATION R ESULTS
Revised: 4. V ISUALIZATION OF E XPLANATION R ESULTS

Original: Finally, we report the visualization results of different explanation methods in Figure 3.
Revised: Finally, we report the visualization results of different explanation methods in Figure 3.

Original: Speciﬁcally, we show the explanations of a molecule graph from ClinTox dataset.
Revised: Speciﬁcally, we show the explanations of a molecule graph from ClinTox dataset.

Original: The smiles string of this input graph is C(CC(C(F)F)(C(=O)[O-])[NH3+])C[NH3+] and the model to be explained is a 3-layer GCN model.
Revised: The smiles string of this input graph is C(CC(C(F)F)(C(=O)[O-])[NH3+])C[NH3+] and the model to be explained is a 3-layer GCN model.

Original: In this input graph, motifs 1 and 2 form a Carboxylate Anion; motif 4 and 5 both contain an Ammoniumyl while motif 3 corresponds to a CF2 group.
Revised: In this input graph, motifs 1 and 2 form a Carboxylate Anion; motif 4 and 5 both contain an Ammoniumyl while motif 3 corresponds to a CF2 group.

Original: It is clear that our FlowX ﬁnds all motifs in the explanation and obtains the highest Fidelity score.
Revised: It is clear that our FlowX ﬁnds all motifs in the explanation and obtains the highest Fidelity score.

Original: It indicates that those motifs are indeed important for the model to make predictions.
Revised: It indicates that those motifs are indeed important for the model to make predictions.

Original: In addition, we can conclude that the model combines these motifs for the predictions since we generally observe that explanations with more motifs chosen tend to have higher Fidelity scores.
Revised: In addition, we can conclude that the model combines these motifs for the predictions since we generally observe that explanations with more motifs chosen tend to have higher Fidelity scores.

Original: Furthermore, we ﬁnd that DeepLIFT identiﬁes all motifs but focuses more on the self-loops of atoms.
Revised: Furthermore, we ﬁnd that DeepLIFT identiﬁes all motifs but focuses more on the self-loops of atoms.

Original: However, its Fidelity score is much lower than GNNExplainer who only identiﬁes four motifs but focuses on the information transmissions between atoms.
Revised: However, its Fidelity score is much lower than GNNExplainer who only identiﬁes four motifs but focuses on the information transmissions between atoms.

Original: Hence, we believe the interactions among different atoms contribute more to the model predictions, which further indicates the Shapley value is a promising solution for studying the explainability of GNN models.
Revised: Hence, we believe the interactions among different atoms contribute more to the model predictions, which further indicates the Shapley value is a promising solution for studying the explainability of GNN models.

Original: More visualization results are reported in the supplementary material.
Revised: More visualization results are reported in the supplementary material.

Original:  C ONCLUSIONS
Revised:  C ONCLUSIONS

Original: We study the explainability of deep graph models, which are generally treated as black-boxes.
Revised: We study the explainability of deep graph models, which are generally treated as black-boxes.

Original: From the inherent functional mechanism of GNNs, we propose FlowX to explain GNNs by studying message ﬂows.
Revised: From the inherent functional mechanism of GNNs, we propose FlowX to explain GNNs by studying message ﬂows.

Original: Our FlowX ﬁrst computes Shapley values approximations as initial assessment and then incorporates a learning reﬁnement.
Revised: Our FlowX ﬁrst computes Shapley values approximations as initial assessment and then incorporates a learning reﬁnement.

Original: Extensive experiments demonstrate our FlowX achieves signiﬁcantly improved explanations.
Revised: Extensive experiments demonstrate our FlowX achieves signiﬁcantly improved explanations.

Original: REPRODUCIBILITY STATEMENT
Revised: REPRODUCIBILITY STATEMENT

Original: Several efforts are made to ensure the reproducibility of our work.
Revised: Several efforts are made to ensure the reproducibility of our work.

Original: To demonstrate the details of proposed algorithm, we provide the rigorous description of our approach’s ﬁrst stage in Algorithm 1, the second stage learning process in Subsection 3.3 and the concrete setting in appendix A.4.
Revised: To demonstrate the details of proposed algorithm, we provide the rigorous description of our approach’s ﬁrst stage in Algorithm 1, the second stage learning process in Subsection 3.3 and the concrete setting in appendix B.4.

Original: For experiments, besides the detailed description of datasets and evaluation metrics in Section 4, we provide model conﬁgurations, training details, target mapping, and metrics details in appendix A and appendix B. Our implementations will be publicly available once the paper is published 
Revised: For experiments, besides the detailed description of datasets and evaluation metrics in Section 4, we provide model conﬁgurations, training details, target mapping, and metrics details in appendix B and appendix C. Our implementations will be publicly available once the paper is published A A LGORITHM ANALYSIS & INTUITIONS A. M ESSAGE F LOW I NTUITION

Revised: Message ﬂows can provide multi-hop dependence association, i.e., the reason of a given node why it is labeled as susceptible will be explained by a message ﬂow beginning from an affected node to the given node in a virus infection dataset.

Revised: Therefore, the explanation is that the message ﬂow transmits the affection information from the affected node to the given node making it a susceptible one.

Revised: The upper ﬁgures of Figure.

Revised: 4 show that how a node’s message can be transmitted from the node to which node across which path.

Original:  A E XPERIMENTAL S ETTINGS
Revised: A. C OMPLEXITY A NALYSIS

Revised: Message ﬂows. Considering the total number of message ﬂows | F | , its loose upper bound is | F | = O ( | E | T ) .

Revised: With the consideration of connectivity, given the largest outgoing degree of nodes in the graph d + , the tighter upper bound of the number of message ﬂows is | F | =

Revised: O ( | E | ( d + ) T − 1 ) .

Revised: Marginal contribution sampling.

Revised: It is noticeable that the deep model’s forward operations are the most time-consuming operations, denoted as O ( T f ) .

Revised: Without any parallel consideration, the time complexity of marginal contribution sampling is T mcs

Revised: =

Revised: O ( M | A |T f ) .

Revised: When we consider parallel implementations, the most simple improvement is to use the GPUs acceleration to move line 9-15 out of the two loops and to make line 8 executed in O (1) , which leads to the time complexity

Revised: T mcs

Revised: =

Revised: O ( M | A | + T f ) .

Revised: The space complexity should be considered because there is a big ﬂow score matrix S cpu

Revised: =

Revised: O ( M | A || F | ) .

Revised: As for the graphic memory requirement, this parallel implementation consumes S gpu

Revised: = O

Revised: ( M | A |S f ) , where O ( S f )

Revised: =

Revised: O (( | V | + | E | ) dT + S p ) is the deep model memory complexity; here, | V | = n denotes the number of nodes; S p is the model’s parameter memory complexity.

Revised: In our implementation, there is a trade-off that we only parallelly calculate the model outside the innermost loop, so that our time complexity is T mcs

Revised: =

Revised: O ( M ( | A | + T f )) and the space complexity of GPUs is S gpu

Revised: = O

Revised: ( | A |S f ) .

Revised: A. T RAINING DILEMMA & E XPONENTIAL REDISTRIBUTION

Revised: We observe that the training is challenging as it often even leads to a worse result after training.

Revised: The difﬁculty comes from that the training target is unusual.

Revised: We not only require the important layer edges’ scores to be trained to be their high-value local optimums with correct rankings but also require that all of the unimportant ones, not including those negative contribution elements, are trained to be low values because of the ranking requirement.

Revised: The problem is that because the unimportant layer edges exert nearly no impact on the model results, their scores are trained to be the values that are largely random, which causes an unimportant layer edge’s score may be larger than an important one because it is random (it may be arbitrary high or low).

Revised: This is not what we expected.

Revised: In brief, these random scores severely interfere with the ranking of scores of the important.

Revised: Therefore, exponential redistribution is designed to solve this problem as shown in Eq.

Revised: 11, where the gradients of the exponential scaling g e ( x ) = x r decide that the high scores can be sensitively trained while the low scores are likely to be trapped in a low-value zone with gradients closed to zeros.

Revised: Intuitively, considering the simplest 2-order exponential scaling g e ( x ) = x 2 = x · x , this function can be regarded as using x as an attention map to mask itself so that high scores become easier to train while it is contrary to the low scores.

Revised: It is noticeable that the use of exponential redistribution requires a relatively reasonable initialization, in order to give important elements a lower possibility to be initialized into the low-value zone, which leads to a longer training time.

Revised: Hence, we initialize scores according to the marginal contribution results we obtain from the last stage so that unimportant factors are set into the low-value zone and will be likely to be trapped there having no interference to the high score ranking, while important layer edges are normally trained.

Revised: Note that the ﬁnal scores are not reﬁned Shapley values, because the Shapley-like initialization only serves as heuristic values for the starting of exponential redistribution.

Revised: Worth mentioning that the use of regularization terms like the size of masks and the information entropy does not work, and the use of Gumbel softmax for encouraging discreteness is useless either because both of them cannot solve the real problem.

Original: [Figure]
Revised: [Table]

Revised: B E XPERIMENTAL S ETTINGS B. D EEP G RAPH M ODELS

Original: We ﬁrst introduce the details of the GNN models we try to explain.
Revised: We ﬁrst introduce the details of the GNN models we try to explain.

Original: For all GNN models, we employ two message passing layers for node classiﬁcation (
Revised: For all GNN models, we employ two message passing layers for node classiﬁcation (

Original: BA-Shapes
Revised: BA-Shapes

Original: (Ying et al., 2019)) and three for graph classiﬁcation tasks (ClinTox, Tox21, BBBP, BACE (Wu et al., 2018), BA-LRP (Schnake et al., 2020), Graph-SST2 (Yuan et al., 2020c)).
Revised: (Ying et al., 2019)) and three for graph classiﬁcation tasks (ClinTox, Tox21, BBBP, BACE (Wu et al., 2018), BA-LRP (Schnake et al., 2020), Graph-SST2 (Yuan et al., 2020c)).

Original: In addition, the ﬁnal classiﬁer consists of two fully-connected layers.
Revised: In addition [Table] the ﬁnal classiﬁer consists of two fully-connected layers.

Original: For graph classiﬁcation tasks, the average pooling is used to convert node embeddings to graph embeddings.
Revised: For graph classiﬁcation tasks, the average pooling is used to convert node embeddings to graph embeddings.

Original: In addition, the feature dimensions of message passing layers and fully-connected layers are set to 300.
Revised: In addition, the feature dimensions of message passing layers and fully-connected layers are set to 300.

Original: We apply the ReLU function as the activation function after each message passing layer and fully-connected layer.
Revised: We apply the ReLU function as the activation function after each message passing layer and fully-connected layer.

Original: In addition, the dropout is applied between two fully-connected layers.
Revised: In addition, the dropout is applied between two fully-connected layers.

Original: Note that we consider both GCNs and GINs for all datasets.
Revised: Note that we consider both GCNs and GINs for all datasets.

Original: For the GCN layer, we employ the original normalized Laplacian matrix ˆ A t =
Revised: For the GCN layer, we employ the original normalized Laplacian matrix ˆ A t =

Original: D − 12 ( A + I ) D − 12 .
Revised: D − 12 ( A + I ) D − 12 .

Original: For the GIN layer, two fully-connected layers with the ReLU function are employed as the multilayer perceptron (MLP).
Revised: For the GIN layer, two fully-connected layers with the ReLU function are employed as the multilayer perceptron (MLP).

Original: A. T RAINING S ETTINGS
Revised: B. T RAINING S ETTINGS

Original: The models on different datasets use different learning rates and decay settings.
Revised: The models on different datasets use different learning rates and decay settings.

Original: Generally, we set the learning rate to 1 × 10 − 3 and the learning rate decay equal to 0 .
Revised: Generally, we set the learning rate to 1 × 10 − 3 and the learning rate decay equal to 0 .

Original: 5 after 500 epochs.
Revised: 5 after 500 epochs.

Original: For datasets
Revised: For datasets

Original: BA-LRP and Graph-SST2, the total number of epochs is 100 while we train models for 1000 epochs on the other datasets.
Revised: BA-LRP and Graph-SST2, the total number of epochs is 100 while we train models for 1000 epochs on the other datasets.

Original: All datasets are split into the training set (80%), validation set (10%), and testing set (10%).
Revised: All datasets are split into the training set (80%), validation set (10%), and testing set (10%).

Original: All experiments are conducted using one NVIDIA 2080Ti
Revised: All experiments are conducted using one NVIDIA 2080Ti

Original: GPU.
Revised: GPU.

Original: A. T ARGETS OF E XPLANATIONS
Revised: B. T ARGETS OF E XPLANATIONS

Original: Since different techniques focus on different explainable targets, such as nodes, edges, walks, ﬂows, etc., these methods cannot be directly compared.
Revised: Since different techniques focus on different explainable targets, such as nodes, edges, walks, ﬂows, etc., these methods cannot be directly compared.

Original: For fair comparisons, we convert all explainable targets to graph edges.
Revised: For fair comparisons, we convert all explainable targets to graph edges.

Original: First, GNNExplainer and PGExplainer provide edge-level explanations so their results are directly used.
Revised: First, GNNExplainer and PGExplainer provide edge-level explanations so their results are directly used.

Original: Second, for ﬂow-based methods and walk-based methods, including
Revised: Second, for ﬂow-based methods and walk-based methods, including

Original: GNN-GI, GNN-LRP, and our FlowX, we convert the ﬂow (walk) importance to edge importance by summing the total contribution of message ﬂows (walks) that go through a particular edge.
Revised: GNN-GI, GNN-LRP, and our FlowX, we convert the ﬂow (walk) importance to edge importance by summing the total contribution of message ﬂows (walks) that go through a particular edge.

Original: In addition, for node-based methods such as GradCAM (Pope et al., 2019) and DeepLIFT (Shrikumar et al., 2017), the node importance is mapped to edge importance that the contribution of an edge is the averaged contribution of its connected nodes.
Revised: In addition, for node-based methods such as GradCAM (Pope et al., 2019) and DeepLIFT (Shrikumar et al., 2017), the node importance is mapped to edge importance that the contribution of an edge is the averaged contribution of its connected nodes.

Original: Specially, for subgraph-based method SubgraphX (Yuan et al., 2021), we pick the explainable subgraph out, then assign the edges in this subgraph instead of nodes as the explanation.
Revised: Specially, for subgraph-based method SubgraphX (Yuan et al., 2021), we pick the explainable subgraph out, then assign the edges in this subgraph instead of nodes as the explanation.

Original: Note that the absolute values of contribution are not important since the metrics focus on their relative rankings.
Revised: Note that the absolute values of contribution are not important since the metrics focus on their relative rankings.

Original: The Figure.
Revised: The Figure.

Original: 4 shows how to map from message ﬂows to edges.
Revised: 4 shows how to map from message ﬂows to edges.

Original: A. F LOW X S ETTINGS
Revised: B. F LOW X S ETTINGS

Original: We randomly initialize the weight vector of our learning reﬁnement from the uniform distribution [0 , 0 . 1] .During the training, we apply a learning rate equal to 0 .
Revised: At the ﬁrst stage, the MC sampling time M is set as 30 . At the second stage, we randomly initialize the weight vector of our learning reﬁnement from the uniform distribution [0 , 0 . 1] .

Original: 3 without learning rate decay and train the weight vector for 500 iterations. Exponential scaling. Exponential scaling is employed to facilitate training and encourage discreteness. Formally, given a normalized vector x , the scaled result x (cid:48) can be expressed as:
Revised: During the training,

Original: [Equation]
Revised: [Figure]

Original: where Nor ( · ) denotes the normalization operation; r is the exponent factor.
Revised: we apply a learning rate equal to 0 . 3 without learning rate decay and train the weight vector foriterations.

Original: Generally, we ﬁnd that r = 8 is a proper choice.
Revised: Meanwhile, we adopt r = 8 for exponential redistribution.

Original: FlowX ∗ & FlowX † .
Revised: FlowX ∗ & FlowX † .

Original: FlowX ∗ can be implemented by replacing the weight vector w in section 3.
Revised: FlowX ∗ can be implemented by replacing the weight vector w in section 3.

Original: with a constant vector w (cid:48) that w (cid:48)
Revised: with a constant vector w (cid:48) that w (cid:48)

Original: = 1 / | w | where | w | is the length of w .
Revised: = 1 / | w | where | w | is the length of w .

Original: FlowX † is implemented with a similar pipeline as GNNExplainer (Ying et al., 2019).
Revised: FlowX † is implemented with a similar pipeline as GNNExplainer (Ying et al., 2019).

Original: Instead of training the weights to obtain the edge mask, we use a ﬂow mask to indicate the ﬂow importance scores.
Revised: Instead of training the weights to obtain the edge mask, we use a ﬂow mask to indicate the ﬂow importance scores.

Original: The ﬂow mask can be considered as the weighted ﬂow scores in our FlowX.
Revised: The ﬂow mask can be considered as the weighted ﬂow scores in our FlowX.

Original: In addition, we do not apply exponential scaling to FlowX † .
Revised: In addition, we do not apply exponential scaling to FlowX † .

Original: The rest of FlowX † is the same as the learning reﬁnement part in our FlowX.
Revised: The rest of FlowX † is the same as the learning reﬁnement part in our FlowX.

Original: B E VALUATION M ETRICS
Revised: C E VALUATION M ETRICS

Original: Mathematically, a K -class classiﬁcation dataset with N samples can be represented as { (( X i , Λ i ) , y i ) | i = 1 , 2 , . . .
Revised: Mathematically, a K -class classiﬁcation dataset with N samples can be represented as { (( X i , Λ i ) , y i ) | i = 1 , 2 , . . .

Original: , N } , where ( X i , Λ i ) is the input graph and y i is the ground-truth label of the i -th sample.
Revised: , N } , where ( X i , Λ i ) is the input graph and y i is the ground-truth label of the i -th sample.

Original: The predicted class of a GNN model is ˆ y i =
Revised: The predicted class of a GNN model is ˆ y i =

Original: argmax f
Revised: argmax f

Original: ( X i , Λ i ) .
Revised: ( X i , Λ i ) .

Original: Here the output of f ( X i , Λ i ) is a K -length vector, in which each element denote the probability of the corresponding class.
Revised: Here the output of f ( X i , Λ i ) is a K -length vector, in which each element denote the probability of the corresponding class.

Original: Formally, we deﬁne the algorithm’s explanation as a column vector ˙
Revised: Formally, we deﬁne the algorithm’s explanation as a column vector ˙

Original: m i that represents a mask on explainable targets where the number of targets is | ˙
Revised: m i that represents a mask on explainable targets where the number of targets is | ˙

Original: m i | .
Revised: m i | .

Original: In this mask, the explanation method marks each selected target as a value 1 , otherwise a 0 .
Revised: In this mask, the explanation method marks each selected target as a value 1 , otherwise a 0 .

Original: The source of the mask is a vector ˙
Revised: The source of the mask is a vector ˙

Original: c i with the same shape that stores the contribution score of each explainable target.
Revised: c i with the same shape that stores the contribution score of each explainable target.

Original: Intuitively, we prefer to assign targets with 1 s in the ˙ m i if the corresponding scores in ˙
Revised: Intuitively, we prefer to assign targets with 1 s in the ˙ m i if the corresponding scores in ˙

Original: c i are relatively high.
Revised: c i are relatively high.

Original: With these notations, we introduce the metrics employed for quantitative comparison.
Revised: With these notations, we introduce the metrics employed for quantitative comparison.

Original: Fidelity.
Revised: Fidelity.

Original: Following the metrics mentioned in the graph explainability survey (Yuan et al., 2020c), we choose the metric Fidelity to measure the faithfulness of the explainability methods to the model.
Revised: Following the metrics mentioned in the graph explainability survey (Yuan et al., 2020c), we choose the metric Fidelity to measure the faithfulness of the explainability methods to the model.

Original: Mathematically, Fidelity is deﬁned as:
Revised: Mathematically, Fidelity is deﬁned as:

Original: [Equation]
Revised: [Equation]

Original: where f ˙ m i
Revised: where f ˙ m i

Original: ( X i , Λ i )
Revised: ( X i , Λ i )

Original: ˆ y i denotes the output class
Revised: ˆ y i denotes the output class

Original: ˆ y i ’s
Revised: ˆ y i ’s

Original: probability when masking out the edges based on ˙
Revised: probability when masking out the edges based on ˙

Original: m i .
Revised: m i .

Original: Speciﬁcally, we keep the value unchanged for the explainable targets that are marked as 1 in ˙ m , while setting the other explainable targets as 0.
Revised: Speciﬁcally, we keep the value unchanged for the explainable targets that are marked as 1 in ˙ m , while setting the other explainable targets as 0.

Original: Note that ˙
Revised: Note that ˙

Original: m i = 1 − ˙ m is complementary mask whereis an all-one vector that has the same shape as ˙ m .
Revised: m i = 1 − ˙ m is complementary mask whereis an all-one vector that has the same shape as ˙ m .

Original: In the evaluation period, the higher Fidelity indicates the better performance of the explainability algorithm.
Revised: In the evaluation period, the higher Fidelity indicates the better performance of the explainability algorithm.

Original: Intuitively, Fidelity represents the predicted probability dropping after occluding the important features.
Revised: Intuitively, Fidelity represents the predicted probability dropping after occluding the important features.

Original: Therefore, Fidelity can measure whether the chosen features are important to the predictions from the model’s perspective. 
Revised: Therefore, Fidelity can measure whether the chosen features are important to the predictions from the model’s perspective. [Table] [Table]

Original: Sparsity.
Revised: Sparsity.

Original: In order to fairly compare different techniques, we argue that controlling the Sparsity (Pope et al., 2019; Yuan et al., 2020c) of explanations is necessary.
Revised: In order to fairly compare different techniques, we argue that controlling the Sparsity (Pope et al., 2019; Yuan et al., 2020c) of explanations is necessary.

Original: Mathematically, Sparsity is deﬁned as:
Revised: Mathematically, Sparsity is deﬁned as:

Original: [Equation]
Revised: [Equation]

Original: A higher Sparsity score means fewer explainable targets are selected.
Revised: A higher Sparsity score means fewer explainable targets are selected.

Original: Note that the selections of targets are determined by the ranking of the contribution score in ˙
Revised: Note that the selections of targets are determined by the ranking of the contribution score in ˙

Original: c i .
Revised: c i .

Original: It is noteworthy that these two[Table] [Table] metrics are highly correlated since the predictions tend to change more when more input features are removed.
Revised: It is noteworthy that these two metrics are highly correlated since the predictions tend to change more when more input features are removed.

Original: Then explanations with higher Sparsity scores tend to have lower Fidelity scores.
Revised: Then explanations with higher Sparsity scores tend to have lower Fidelity scores.

Original: For fair comparisons, we compare different methods with similar sparsity levels.
Revised: For fair comparisons, we compare different methods with similar sparsity levels.

Original: C E XPERIMENTAL R ESULTS
Revised: D E XPERIMENTAL R ESULTS

Original: In this section, we provide additional quantitative comparisons and visualization results.
Revised: In this section, we provide additional quantitative comparisons and visualization results.

Original: C. Q UANTITATIVE R ESULTS
Revised: D. Q UANTITATIVE R ESULTS

Original: We report the quantitative results of GCN and GIN models in Table 4 and Table 5.
Revised: We report the quantitative results of GCN and GIN models in Table 4 and Table 5.

Original: Obviously, our method obtains the best averaged Fidelity score for 5 out of 7 GCN models.
Revised: Obviously, our method obtains the best averaged Fidelity score for 5 out of 7 GCN models.

Original: Note that the performance is competitive for synthetic dataset BA-Shapes and BA-LRP and our method performs signiﬁcantly better on the complex real-world datasets.
Revised: Note that the performance is competitive for synthetic dataset BA-Shapes and BA-LRP and our method performs signiﬁcantly better on the complex real-world datasets.

Original: In addition, we show the plots of Fidelity scores with respect to different Sparsity levels for all datasets and GCN models in Figure 5.
Revised: In addition, we show the plots of Fidelity scores with respect to different Sparsity levels for all datasets and GCN models in Figure 5.

Original: Clearly, the performance of our method is stable and better.
Revised: Clearly, the performance of our method is stable and better.

Original: Accuracy comparisons: To further demonstrate the effectiveness of our proposed method, we conduct experiments to compare the explanatory accuracy of three methods that usually evaluated in terms of accuracy.
Revised: Accuracy comparisons: To further demonstrate the effectiveness of our proposed method, we conduct experiments to compare the explanatory accuracy of three methods that usually evaluated in terms of accuracy.

Original: The results are reported in Table 6.
Revised: The results are reported in Table 6.

Original: These results are obtained from a synthetic graph classiﬁcation dataset BA-INFE.
Revised: These results are obtained from a synthetic graph classiﬁcation dataset BA-INFE.

Original: Each graph includes a base BA-graph and four types of motifs.
Revised: Each graph includes a base BA-graph and four types of motifs.

Original: Two classes of motifs are attached (2 motifs denote the one property, 2 for another; let’s denote them as class/property A and B).
Revised: Two classes of motifs are attached (2 motifs denote the one property, 2 for another; let’s denote them as class/property A and B).

Original: We ﬁrst connect the same number (1 3) of each class’s motifs to the base graph.
Revised: We ﬁrst connect the same number (1 3) of each class’s motifs to the base graph.

Original: Then we attach 1 to 3 extra motifs from one of the classes (for example, class A) to the base graph; thus, the number motifs in class A is more than class B, representing the graph tends to have the speciﬁc property (property A).
Revised: Then we attach 1 to 3 extra motifs from one of the classes (for example, class A) to the base graph; thus, the number motifs in class A is more than class B, representing the graph tends to have the speciﬁc property (property A).

Original: The ground-truths of a graph are the elements in those motifs including edges (undirected) and self-loop connections.
Revised: The ground-truths of a graph are the elements in those motifs including edges (undirected) and self-loop connections.

Original: The number of ground-truth elements that are covered by explanation edges is denoted as the hit number.
Revised: The number of ground-truth elements that are covered by explanation edges is denoted as the hit number.

Original: The accuracy is deﬁned as (hit number) /
Revised: The accuracy is deﬁned as (hit number) /

Original: (total number of ground-truth elements).
Revised: (total number of ground-truth elements).

Original:  We also show the time cost of different methods in Table 3.
Revised: [Table] [Table] We also show the time cost of different methods in Table 3.

Original: Speciﬁcally, we report the averaged computation time for 40 graphs in the ClinTox (Wu et al., 2018) dataset.
Revised: Speciﬁcally, we report the averaged computation time for 40 graphs in the ClinTox (Wu et al., 2018) dataset.

Original: It shows that our FlowX performs the best with a reasonable time cost.[Table] [Table]
Revised: It shows that our FlowX performs the best with a reasonable time cost.

Original: C. E XPLANATION V ISUALIZATIONS
Revised: D. E XPLANATION V ISUALIZATIONS

Original: We provide additional examples to visually compare the explanations by six methods.
Revised: We provide additional examples to visually compare the explanations by six methods.

Original: The results are reported in Figure.
Revised: The results are reported in Figure.

Original: 6, and 7.
Revised: 6, and 7.

Original: We can conclude that the message interactions between nodes are more important than self-loops.
Revised: We can conclude that the message interactions between nodes are more important than self-loops.

Original: For example, in Figure.
Revised: For example, in Figure.

Original: 6, all of DeepLIFT, GNNExplainer, and FlowX can identify the motifs; however, FlowX attributes more on the node interactions and obtain a higher
Revised: 6, all of DeepLIFT, GNNExplainer, and FlowX can identify the motifs; however, FlowX attributes more on the node interactions and obtain a higher

Original: Fidelity score.
Revised: Fidelity score.

Original: [Figure]
Revised: [Figure]