Original: O N
Revised: O N

Original: R EGULARIZATION
Revised: R EGULARIZATION

Original: FOR
Revised: FOR

Original: E XPLAINING
Revised: E XPLAINING

Original: G RAPH
Revised: G RAPH

Original: N EURAL
Revised: N EURAL

Original: N ETWORKS :
Revised: N ETWORKS :

Original: A N
Revised: A N

Original: I NFORMATION
Revised: I NFORMATION

Original: T HEORY
Revised: T HEORY

Original: P ERSPECTIVE
Revised: P ERSPECTIVE

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: This work studies the explainability of graph neural networks (GNNs), which is important for the credibility of GNNs in practical usage.
Revised: This work studies the explainability of graph neural networks (GNNs), which is important for the credibility of GNNs in practical usage.

Original: Existing work mostly follows the two-phase paradigm to interpret a prediction: feature attribution and selection .
Revised: Existing work mostly follows the two-phase paradigm to interpret a prediction: feature attribution and selection .

Original: However, another important component — regularization, which is crucial to facilitate the above paradigm — has been seldom studied.
Revised: However, another important component — regularization, which is crucial to facilitate the above paradigm — has been seldom studied.

Original: In this work, we explore the role of regularization in GNNs explainability from the perspective of information theory.
Revised: In this work, we explore the role of regularization in GNNs explainability from the perspective of information theory.

Original: Our main findings are: 1) regularization is essentially pursuing the balance between two phases, 2) its optimal coefficient is proportional to the sparsity of explanations, 3) existing methods imply an implicit regularization effect of stochastic mechanism, and 4) its contradictory effects on two phases are responsible for the out-of-distribution (OOD) issue in post-hoc explainability.
Revised: Our main findings are: 1) regularization is essentially pursuing the balance between two phases, 2) its optimal coefficient is proportional to the sparsity of explanations, 3) existing methods imply an implicit regularization effect of stochastic mechanism, and 4) its contradictory effects on two phases are responsible for the out-of-distribution (OOD) issue in post-hoc explainability.

Original: Based on these findings, we propose two common optimization methods, which can bolster the performance of the current explanation methods via sparsity-adaptive and OOD-resistant regularization schemes.
Revised: Based on these findings, we propose two common optimization methods, which can bolster the performance of the current explanation methods via sparsity-adaptive and OOD-resistant regularization schemes.

Original: Extensive empirical studies validate our findings and proposed methods.
Revised: Extensive empirical studies validate our findings and proposed methods.

Original: Code is available at https://anonymous.4open.science/r/Rethink_
Revised: Code is available at https://anonymous.4open.science/r/Rethink_

Original: Reg-
Revised: Reg-

Original: 07F0 .
Revised: 07F0 .

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Graph Neural Networks (GNNs) (Dwivedi et al., 2020; Wu et al., 2019) have achieved remarkable progress on various graph-related tasks (Mahmud et al., 2021;
Revised: Graph Neural Networks (GNNs) (Dwivedi et al., 2020; Wu et al., 2019) have achieved remarkable progress on various graph-related tasks (Mahmud et al., 2021;

Original: Zhao et al., 2021; Guo & Wang, 2021).
Revised: Zhao et al., 2021; Guo & Wang, 2021).

Original: However, GNNs usually work as a black box, making the decision-making process obscure and hard to interpret (Ribeiro et al., 2016).
Revised: However, GNNs usually work as a black box, making the decision-making process obscure and hard to interpret (Ribeiro et al., 2016).

Original: Hence, answering the question: “What knowledge does the GNN use to make a certain prediction?” is becoming crucial.
Revised: Hence, answering the question: “What knowledge does the GNN use to make a certain prediction?” is becoming crucial.

Original: To solve this question, most prior studies (Yu et al., 2021; Miao et al., 2022) realize post-hoc explainability by extracting the informative yet sparse subgraphs as explanations, following the principle of graph information bottleneck (GIB)
Revised: To solve this question, most prior studies (Yu et al., 2021; Miao et al., 2022) realize post-hoc explainability by extracting the informative yet sparse subgraphs as explanations, following the principle of graph information bottleneck (GIB)

Original: (Wu et al., 2020).
Revised: (Wu et al., 2020).

Original: The common paradigm of these explainers can be summarized as the relay race of feature attribution and selection.
Revised: The common paradigm of these explainers can be summarized as the relay race of feature attribution and selection.

Original: Specifically, feature attribution distributes the prediction to the input features and traces their importance, and feature selection sequentially fills features into the explanatory subgraph according to the importance rank, where regularization terms are introduced to constrain subgraph properties like size and connectivity.
Revised: Specifically, feature attribution distributes the prediction to the input features and traces their importance, and feature selection sequentially fills features into the explanatory subgraph according to the importance rank, where regularization terms are introduced to constrain subgraph properties like size and connectivity.

Original: However, existing explainers allocate little attention to the role of regularization in them, but is the focus of our work.
Revised: However, existing explainers allocate little attention to the role of regularization in them, but is the focus of our work.

Original: On the one hand, without digging deeper into regularization theoretically, we hardly acquire a plain picture of how regularization specifically affects the process of feature attribution and selection.
Revised: On the one hand, without digging deeper into regularization theoretically, we hardly acquire a plain picture of how regularization specifically affects the process of feature attribution and selection.

Original: Furthermore, some regularization in existing explainers lacks concrete theoretical support and is seemingly not more than an empirical trick.
Revised: Furthermore, some regularization in existing explainers lacks concrete theoretical support and is seemingly not more than an empirical trick.

Original: For example, GNNExplainer (Ying et al., 2019) leverages the l 1 norm to constrain the magnitude of masks and selects the edge with larger importance ( i.e. , larger mask).
Revised: For example, GNNExplainer (Ying et al., 2019) leverages the l 1 norm to constrain the magnitude of masks and selects the edge with larger importance ( i.e. , larger mask).

Original: The key here is not the absolute magnitude of the mask (i.e., l 1 norm), but rather the relative magnitude between the masks.
Revised: The key here is not the absolute magnitude of the mask (i.e., l 1 norm), but rather the relative magnitude between the masks.

Original: Thus, we argue that the necessity of l 1 norm needs more theoretical support.
Revised: Thus, we argue that the necessity of l 1 norm needs more theoretical support.

Original: In sight of this, we endeavor to rethink the role of regularization in GNNs explainability from the perspective of information theory.
Revised: In sight of this, we endeavor to rethink the role of regularization in GNNs explainability from the perspective of information theory.

Original: Before starting, we first reshape the principle of GIB as GIBE ( i.e. , new GIB form tailored for GNNs Explainability) in the language of feature attribution and selection.
Revised: Before starting, we first reshape the principle of GIB as GIBE ( i.e. , new GIB form tailored for GNNs Explainability) in the language of feature attribution and selection.

Original: Specifically, GIBE unifies the current explanation methods via formulating the optimization objective of these two phases.
Revised: Specifically, GIBE unifies the current explanation methods via formulating the optimization objective of these two phases.

Original: It further explores the roles of regularization in two phases respectively.
Revised: It further explores the roles of regularization in two phases respectively.

Original: Guided by these explorations, we reveal the essence of regularization and propose four intriguing propositions in terms of it.
Revised: Guided by these explorations, we reveal the essence of regularization and propose four intriguing propositions in terms of it.

Original: We believe a better theory of regularization is fundamental:
Revised: We believe a better theory of regularization is fundamental:

Original: • The essence of regularization: Regularization in GNNs explainability is essentially the tradeoff scheme to pursue the balance between the phases of feature attribution and selection (Section 3.2).
Revised: • The essence of regularization: Regularization in GNNs explainability is essentially the tradeoff scheme to pursue the balance between the phases of feature attribution and selection (Section 3.2).

Original: • On Sparsity: The optimal coefficients of regularization are proportional to the sparsity of the explanation, that is, high sparsity should require large regularization and vice versa (Section 4.1).
Revised: • On Sparsity: The optimal coefficients of regularization are proportional to the sparsity of the explanation, that is, high sparsity should require large regularization and vice versa (Section 4.1).

Original: • On stochastic mechanism: Existing methods imply an implicit regularization effect of stochastic mechanism, which endows GNNs explainability with better compressibility (Section 4.2).
Revised: • On stochastic mechanism: Existing methods imply an implicit regularization effect of stochastic mechanism, which endows GNNs explainability with better compressibility (Section 4.2).

Original: • On OOD issue: The contradictory effects of regularization on two phases are responsible for the OOD issue in the post-hoc explainability (Section 4.3).
Revised: • On OOD issue: The contradictory effects of regularization on two phases are responsible for the OOD issue in the post-hoc explainability (Section 4.3).

Original: Furthermore, based on these findings, we propose two common optimization methods, which can bolster the performance of current explainers via sparsity-adaptive and OOD-resistant regularization schemes.
Revised: Furthermore, based on these findings, we propose two common optimization methods, which can bolster the performance of current explainers via sparsity-adaptive and OOD-resistant regularization schemes.

Original: Extensive empirical studies validate our findings and proposed methods in Section 5.
Revised: Extensive empirical studies validate our findings and proposed methods in Section 5.

Original:  P RELIMINARY AND R ELATED W ORK
Revised:  P RELIMINARY AND R ELATED W ORK

Original: GNNs explainability.
Revised: GNNs explainability.

Original: While GNNs have achieved remarkable success in node classification (Zhou et al., 2019; Sankar et al., 2019), graph classification (Zhang et al., 2018;
Revised: While GNNs have achieved remarkable success in node classification (Zhou et al., 2019; Sankar et al., 2019), graph classification (Zhang et al., 2018;

Original: Chen et al., 2019), and link prediction (Ying et al., 2018; You et al., 2020) tasks, in this work, we focus on the scenario of interpreting graph classification task comprising the data distribution D and the classifier f ′ .
Revised: Chen et al., 2019), and link prediction (Ying et al., 2018; You et al., 2020) tasks, in this work, we focus on the scenario of interpreting graph classification task comprising the data distribution D and the classifier f ′ .

Original: Specifically, the input graph G =
Revised: Specifically, the input graph G =

Original: ( X , A ) is independent and identically distributed (IID) from D , where X is the features of all nodes and A is the adjacency matrix.
Revised: ( X , A ) is independent and identically distributed (IID) from D , where X is the features of all nodes and A is the adjacency matrix.

Original: GNNs explainers aim to seek for G ∗ ( i.e. , ground-truth subgraph) as the explanation (Ying et al., 2019; Luo et al., 2020) thatbest supports thepredictive label Y . More formally, Y is determined byG ∗ in the sense that
Revised: Following Miao et al. (2022), we assume that there exists a subgraph G ∗ such that the label Y for graph G is determined by

Original: Y =
Revised: Y =

Original: f ( G ∗ )
Revised: f ( G ∗ )

Original: + ϵ for somedeterministic invertible function f (Miao et al., 2022), where randomness ϵ is independent from G .
Revised: + ϵ for some ϵ as noise independent of G , where f is an invertible function projecting the set of subgraphs to the label space.

Original: Guided by this target, explainers mainly leverage feature attribution and selection to extract the subgraph (Vu & Thai, 2020; Yuan et al., 2021; Lin et al., 2021).
Revised: Guided by the target of searching G ∗ , current explainers mainly leverage feature attribution and selection to extract the subgraph (Wang et al., 2021).

Original: Feature attribution.
Revised: Feature attribution.

Original: Current explainers mainly perform feature attribution by leveraging:
Revised: Current explainers mainly perform feature attribution by leveraging:

Original: • Gradient-like signals w.r t.
Revised: • Gradient-like signals w.r.t.

Original: the
Revised: the

Original: graph
Revised: graph

Original: structure
Revised: structure

Original: (Baldassarre & Azizpour, 2019; Pope et al., 2019).
Revised: (Baldassarre & Azizpour, 2019; Pope et al., 2019)

Original: For example, SA (Baldassarre & Azizpour, 2019) directly calculates the gradients of GNN’s loss w.r.t.
Revised: For example, SA (Baldassarre & Azizpour, 2019) directly calculates the gradients of GNN’s loss w.r.t.

Original: adjacency matrix as the importance scores of edges ; • Attention scores of structural features (Luo et al., 2020; Miao et al., 2022).
Revised: adjacency matrix as the importance scores of edges; • Attention scores of structural features (Luo et al., 2020; Miao et al., 2022).

Original: For example, GSAT (in its post-hoc working mode)
Revised: For example, GSAT (in its post-hoc working mode)

Original: (Miao et al., 2022) trains a parameterized predictor to generate the stochastic attention for each edge as their importance; • Mask scores of structural features (Ying et al., 2019; Wang et al., 2021).
Revised: (Miao et al., 2022) trains a parameterized predictor to generate the stochastic attention for each edge as their importance; • Mask scores of structural features (Ying et al., 2019; Wang et al., 2021).

Original: For example, GNNExplainer (Ying et al., 2019) adds soft masks to the input features and trains them by maximizing the mutual information between the masked outcome and target prediction; • Prediction changes on structure perturbations (Yuan et al., 2021; Lin et al., 2021).
Revised: For example, GNNExplainer (Ying et al., 2019) adds soft masks to the input features and trains them by maximizing the mutual information between the masked outcome and target prediction; • Prediction changes on structure perturbations (Yuan et al., 2021; Lin et al., 2021).

Original: For example, PGMExplainer (Vu & Thai, 2020) collects the prediction change on the random node perturbations and learns a Bayesian network from these observations.
Revised: For example, PGMExplainer (Vu & Thai, 2020) collects the prediction change on the random node perturbations and learns a Bayesian network from these observations.

Original: Feature selection.
Revised: Feature selection.

Original: With attribution scores of features, input features are sequentially filled into the set of salient features to generate the explanatory subgraph according to their importance rank.
Revised: With attribution scores of features, input features are sequentially filled into the set of salient features to generate the explanatory subgraph according to their importance rank.

Original: Many regularization terms are introduced to guide this process.
Revised: Many regularization terms are introduced to guide this process.

Original: For example, sparsity constraints (Ying et al., 2019; Schlichtkrull et al., 2021) typically leverage the l 1 norm to guarantee that the selected subgraph remains within a prescribed size; connective constraints (Luo et al., 2020; Wang et al., 2021) give more selective probabilities to the edges connecting with the part selected already;
Revised: For example, sparsity constraints (Ying et al., 2019; Schlichtkrull et al., 2021) typically leverage the l 1 norm to guarantee that the selected subgraph remains within a prescribed size; connective constraints (Luo et al., 2020; Wang et al., 2021) give more selective probabilities to the edges connecting with the part selected already;

Original: more recently, information bottleneck constraints (Miao et al., 2022) are proposed to squeeze the mutual information between the input graph and the selected subgraph.
Revised: more recently, information bottleneck constraints (Miao et al., 2022) are proposed to squeeze the mutual information between the input graph and the selected subgraph.

Original:  R ETHINKING THE R OLE OF R EGULARIZATION
Revised:  R ETHINKING THE R OLE OF R EGULARIZATION

Original: In this section, we rethink the role of regularization in GNNs explainers.
Revised: In this section, we rethink the role of regularization in GNNs explainers.

Original: We start with a new form of graph information bottleneck tailored for explainability and the formulation of the feature attribution and selection (Section 3.1).
Revised: We start with a new form of graph information bottleneck tailored for explainability and the formulation of the feature attribution and selection (Section 3.1).

Original: Guided by the above theory, we analyze the effect of regularization in two phases, respectively (Section 3.2).
Revised: Guided by the above theory, we analyze the effect of regularization in two phases, respectively (Section 3.2).

Original: 3. G RAPH I NFORMATION B OTTLENECK FOR EXPLAINABILITY
Revised: 3. G RAPH I NFORMATION B OTTLENECK FOR EXPLAINABILITY

Original: The principle of GIB is widely leveraged to guide the subgraph generation by prevailing GNNs explainers (Wu et al., 2020;
Revised: The principle of GIB is widely leveraged to guide the subgraph generation by prevailing GNNs explainers (Wu et al., 2020;

Original: Miao et al., 2022).
Revised: Miao et al., 2022).

Original: The general formulation of GIB is shown as follows:
Revised: The general formulation of GIB is shown as follows:

Original: • Definition 1 (GIB (Yu et al., 2021))
Revised: • Definition 1 (GIB (Yu et al., 2021))

Original: Given an input graph G and its label Y , GIB seeks for a maximally informative yet sparse subgraph by optimizing the following objective:
Revised: Given an input graph G and its label Y , GIB seeks for a maximally informative yet sparse subgraph by optimizing the following objective:

Original: [Equation]
Revised: [Equation]

Original: where G sub ( G ) indicates the set of all subgraphs of G and β is the Lagrangian multiplier.
Revised: where G sub ( G ) indicates the set of all subgraphs of G and β is the Lagrangian multiplier.

Original: To
Revised: To

Original: instantiate the objective of GIB
Revised: instantiate the objective of GIB

Original: , considering
Revised: , considering

Original: the discreteness and non differentiability of sub
Revised: the discreteness and non differentiability of sub

Original: - graph G s
Revised: - graph G s

Original: existing
Revised: existing

Original: explanation methods
Revised: explanation methods

Original: typically take G ⊙ M as the proxy of G s
Revised: typically take G ⊙ M as the proxy of G s

Original: where M is the explanatory mask sharing the same size with A .
Revised: where M is the explanatory mask sharing the same size with A .

Original: Concretely, for mask-based methods, M is the trainable masks; for attention-based methods, M is the dualization of the attention matrix
Revised: Concretely, for mask-based methods, M is the trainable masks; for attention-based methods, M is the dualization of the attention matrix

Original: and for perturbation-based
Revised: and for perturbation-based

Original: methods
Revised: methods

Original: , M is the matrix recording corresponding features are perturbed or not.
Revised: , M is the matrix recording corresponding features are perturbed or not.

Original: In sight of this, we first
Revised: In sight of this, we first

Original: replace
Revised: replace

Original: G s
Revised: G s

Original: in
Revised: in

Original: Equation
Revised: Equation

Original: with
Revised: with

Original: G ⊙ M .
Revised: G ⊙ M .

Original: Moreover, according to the invariance of the mutual information (MI) to invertible transformation, we rewrite Equation 1 to introduce the new GIB form tailored for GNNs explainability.
Revised: Moreover, according to the invariance of the mutual information (MI) to invertible transformation, we rewrite Equation 1 to introduce the new GIB form tailored for GNNs explainability.

Original: Detailed derivation is provided in Appendix A.
Revised: Detailed derivation is provided in Appendix A.

Original: • Definition 2 (GIBE)
Revised: • Definition 2 (GIBE)

Original: Given an input graph G and its label Y , GIBE seeks for the explanatory mask M to generate the explanation G ⊙ M by optimizing the following objective:
Revised: Given an input graph G and its label Y , GIBE seeks for the explanatory mask M to generate the explanation G ⊙ M by optimizing the following objective:

Original: [Equation]
Revised: [Equation]

Original: where α is the tradeoff parameter which equals to β/ (1 − β ) for β in Definition 1.
Revised: where α is the tradeoff parameter which equals to β/ (1 − β ) for β in Definition 1.

Original: Theoretically, employing Data Processing Inequality (DPI) (Cover & Thomas, 2006) along the
Revised: Theoretically, employing Data Processing Inequality (DPI) (Cover & Thomas, 2006) along the

Original: Markov chain, G ∗ →
Revised: Markov chain, G ∗ →

Original: G → Y , the optimal solution M of Equation 2 can be proved to be equal to the adjacency matrix of G ∗ .
Revised: G → Y , the optimal solution M of Equation 2 can be proved to be equal to the adjacency matrix of G ∗ .

Original: We provide detailed derivation in Appendix A.2.
Revised: We provide detailed derivation in Appendix A.2.

Original: Note that GIBE is the first attempt to in-depth combine the principle of GIB and the common paradigm of the post-hoc explainability ( i.e. , feature attribution and selection).
Revised: Note that GIBE is the first attempt to in-depth combine the principle of GIB and the common paradigm of the post-hoc explainability ( i.e. , feature attribution and selection).

Original: Specifically, the first term of Equation 2 is the optimization objective of feature attribution, which maps the information of feature importance in G to mask, G (cid:55)→ G ⊙ M ; and the second term is the objective of feature selection, which maps the information in the mask to subgraph, G ⊙ M
Revised: Specifically, the first term of Equation 2 is the optimization objective of feature attribution, which maps the information of feature importance in G to mask, G (cid:55)→ G ⊙ M ; and the second term is the objective of feature selection, which maps the information in the mask to subgraph, G ⊙ M

Original: (cid:55)→ G s .
Revised: (cid:55)→ G s .

Original: In conclusion, GIBE endows the GIB with better ability to directly guide the construction of the explanation methods.
Revised: In conclusion, GIBE endows the GIB with better ability to directly guide the construction of the explanation methods.

Original: 3. R EGULARIZATION IN T WO P HASES
Revised: 3. R EGULARIZATION IN T WO P HASES

Original: Regularization in feature attribution.
Revised: Regularization in feature attribution.

Original: Directly calculating the training objective of feature attribution, I ( G ⊙ M ; Y ) is tough since the latent distribution P ( G ⊙ M , Y ) is notoriously intractable.
Revised: Directly calculating the training objective of feature attribution, I ( G ⊙ M ; Y ) is tough since the latent distribution P ( G ⊙ M , Y ) is notoriously intractable.

Original: Hence, existing explainers typically introduce the parameterized variational approximation P θ ( Y | G ⊙ M ) for P ( Y | G ⊙ M ) according to:
Revised: Hence, existing explainers typically introduce the parameterized variational approximation P θ ( Y | G ⊙ M ) for P ( Y | G ⊙ M ) according to:

Original: [Equation]
Revised: [Equation]

Original: Detailed derivation is provided in Appendix A.3.
Revised: Detailed derivation is provided in Appendix A.3.

Original: Treating classifier f ′ as the proxy function of P θ , the first term of Equation 3 is specified as the cross-entropy between Y and f ′ ( G ⊙ M ) .
Revised: Treating classifier f ′ as the proxy function of P θ , the first term of Equation 3 is specified as the cross-entropy between Y and f ′ ( G ⊙ M ) .

Original: In this case, since the second term H ( Y ) is a constant, the gap between I ( G ⊙ M ; Y ) and above cross-entropy, shown as gap1 in Figure 1, solely depends on the third term of Equation 3.
Revised: In this case, since the second term H ( Y ) is a constant, the gap between I ( G ⊙ M ; Y ) and above cross-entropy, shown as gap1 in Figure 1, solely depends on the third term of Equation 3.

Original: Note that this term is also the Kullback-Leibler (KL) Divergence between P ( Y | G ⊙ M ) and f ′ ( G ⊙ M ) .
Revised: Note that this term is also the Kullback-Leibler (KL) Divergence between P ( Y | G ⊙ M ) and f ′ ( G ⊙ M ) .

Original: To bridge this gap and maximize the effectiveness of feature attribution, regularization should play a role in constraining P ( Y | G ⊙ M ) to f ′ .
Revised: To bridge this gap and maximize the effectiveness of feature attribution, regularization should play a role in constraining P ( Y | G ⊙ M ) to f ′ .

Original: Since f ′ is trained for data distribution P ( G , Y ) , this role transfers to constrain G ⊙ M to G .
Revised: Since f ′ is trained for data distribution P ( G , Y ) , this role transfers to constrain G ⊙ M to G .

Original: In this case, the constraint orientation of M should be loosened to A for squeezing gap in feature attribution to 0, as shown in the left side of Figure 1.
Revised: In this case, the constraint orientation of M should be loosened to A for squeezing gap in feature attribution to 0, as shown in the left side of Figure 1.

Original: Regularization in feature selection.
Revised: Regularization in feature selection.

Original: Feature selection typically binarizes M to achieve the mapping G ⊙ M (cid:55)→ G s .
Revised: Feature selection typically binarizes M to achieve the mapping G ⊙ M (cid:55)→ G s .

Original: To bridge the gap between M and its binaryzation M ′ , shown as gap2 in Figure 1, the optimization objective of feature selection, I ( G ⊙ M ; G ∗ ) − I ( G ⊙ M ; G ) in Equation 2 is usually maximized by: (1) discrete constraint
Revised: To bridge the gap between M and its binaryzation M ′ , shown as gap2 in Figure 1, the optimization objective of feature selection, I ( G ⊙ M ; G ∗ ) − I ( G ⊙ M ; G ) in Equation 2 is usually maximized by: (1) discrete constraint

Original: (Ying et al., 2019) and connectivity constraint (Luo et al., 2020), which are work for maximizing I ( G ⊙ M ; G ∗ ) ; (2) sparsity constraint (Schlichtkrull et al., 2021), which leverages the l 1 or l 2 norm to minimize I ( G ⊙ M ; G ) .
Revised: (Ying et al., 2019) and connectivity constraint (Luo et al., 2020), which are work for maximizing I ( G ⊙ M ; G ∗ ) ; (2) sparsity constraint (Schlichtkrull et al., 2021), which leverages the l 1 or l 2 norm to minimize I ( G ⊙ M ; G ) .

Original: That is, in the phase of feature selection, regularization plays a role in constraining G ⊙ M to G s .
Revised: That is, in the phase of feature selection, regularization plays a role in constraining G ⊙ M to G s .

Original: In other words, the constraint orientation of M should be tightened to A s ( i.e. , adjacency matrix of
Revised: In other words, the constraint orientation of M should be tightened to A s ( i.e. , adjacency matrix of

Original: G s ) for squeezing the gap in feature selection to 0, as shown in the right side of Figure 1.
Revised: G s ) for squeezing the gap in feature selection to 0, as shown in the right side of Figure 1.

Original: In conclusion, while the constraint orientation of M is beneficial to feature attribution and the gap in feature attribution tends to zero, the gap in feature selection will inevitably become large, and vice versa.
Revised: In conclusion, while the constraint orientation of M is beneficial to feature attribution and the gap in feature attribution tends to zero, the gap in feature selection will inevitably become large, and vice versa.

Original: In other words, since the role of regularization is totally different across two phases, regularization is essentially a tradeoff to guarantee the mapping effectiveness both in two phases:
Revised: In other words, since the role of regularization is totally different across two phases, regularization is essentially a tradeoff to guarantee the mapping effectiveness both in two phases:

Original: • Proposition 1 (Essence of Regularization
Revised: • Proposition 1 (Essence of Regularization

Original: ) Regularization in GNNs explainability is essentially the tradeoff scheme to pursue the balance between the phases of feature attribution and selection.
Revised: ) Regularization in GNNs explainability is essentially the tradeoff scheme to pursue the balance between the phases of feature attribution and selection.

Original:  P ROPOSITIONS OF R EGULARIZATION IN E XPLANATION METHODS
Revised:  P ROPOSITIONS OF R EGULARIZATION IN E XPLANATION METHODS

Original: In this section, we derive three intriguing propositions stemming from GIBE and Proposition 1 , which respectively reveal the relation between regularization and three important concepts in GNNs explainability ( i.e. , sparsity (Ying et al., 2019; Lucic et al., 2022), stochastic mechanism (Luo et al., 2020; Wang et al., 2021) and OOD issue (Miao et al., 2022;
Revised: In this section, we derive three intriguing propositions stemming from GIBE and Proposition 1 , which respectively reveal the relation between regularization and three important concepts in GNNs explainability ( i.e. , sparsity (Ying et al., 2019; Lucic et al., 2022), stochastic mechanism (Luo et al., 2020; Wang et al., 2021) and OOD issue (Miao et al., 2022;

Original: Wu et al., 2022)).
Revised: Wu et al., 2022)).

Original: Inspired by these propositions, we propose two simple yet effective regularization schemes.
Revised: Inspired by these propositions, we propose two simple yet effective regularization schemes.

Original: 4. R EGULARIZATION & S PARSITY
Revised: 4. R EGULARIZATION & S PARSITY

Original: We first focus on the relationship between regularization and predefined sparsity η of targeted subgraph G ∗ , where η = 1 − |G ∗ | / |G| .
Revised: We first focus on the relationship between regularization and predefined sparsity η of targeted subgraph G ∗ , where η = 1 − |G ∗ | / |G| .

Original: According to Equation 2, sparsity variation of G ∗ only affect the objective of feature selection,
Revised: According to Equation 2, sparsity variation of G ∗ only affect the objective of feature selection,

Original: I ( G ⊙ M ; G ∗ ) − I ( G ⊙ M ; G ) .
Revised: I ( G ⊙ M ; G ∗ ) − I ( G ⊙ M ; G ) .

Original: Therefore, we aim to derive the relation between η and the phase of feature selection in current explanation methods.
Revised: Therefore, we aim to derive the relation between η and the phase of feature selection in current explanation methods.

Original: The process of derivation is exhibited as follows:
Revised: The process of derivation is exhibited as follows:

Original: To calculate this objective, we first introduce the variational approximation P ϕ
Revised: To calculate this objective, we first introduce the variational approximation P ϕ

Original: ( G ⊙ M | G ∗ ) for P ( G ⊙ M | G ∗ ) and Q for marginal distribution P ( G ⊙ M ) simultaneously.
Revised: ( G ⊙ M | G ∗ ) for P ( G ⊙ M | G ∗ ) and Q for marginal distribution P ( G ⊙ M ) simultaneously.

Original: Then the lower bound of the objective can be derived as:
Revised: Then the lower bound of the objective can be derived as:

Original: [Equation]
Revised: [Equation]

Original: Detailed derivation is shown in Appendix A.4.
Revised: Detailed derivation is shown in Appendix A.4.

Original: While M is inherited from feature attribution, the above lower bound is in positive relation to its first term, E G⊙ M , G ∗ [log P ϕ
Revised: While M is inherited from feature attribution, the above lower bound is in positive relation to its first term, E G⊙ M , G ∗ [log P ϕ

Original: ( G ⊙ M | G ∗ )] since the rest of terms are constants.
Revised: ( G ⊙ M | G ∗ )] since the rest of terms are constants.

Original: To calculate its first term, the variational distribution P ϕ ( G ⊙ M | G ∗ ) is defined as follow: for every two directed node pair ( u, v ) in G ∗ , we sample the elements of M by m uv ∼ Bern ( z ) where z ∈ [0 , 1] is a hyperparameter.
Revised: To calculate its first term, the variational distribution P ϕ ( G ⊙ M | G ∗ ) is defined as follow: for every two directed node pair ( u, v ) in G ∗ , we sample the elements of M by m uv ∼ Bern ( z ) where z ∈ [0 , 1] is a hyperparameter.

Original: Borrowing the above P ϕ , the first term of
Revised: Borrowing the above P ϕ , the first term of

Original: Equation 4 is equal to:
Revised: Equation 4 is equal to:

Original: [Equation]
Revised: [Equation]

Original: where E ∗ is the set of edges in G ∗ .
Revised: where E ∗ is the set of edges in G ∗ .

Original: Note that the sum in Equation 5 has positive relation to the number of terms |E ∗ | , which equals to (1 − η ) |G| , since M is inherent and the values of every terms are fixed.
Revised: Note that the sum in Equation 5 has positive relation to the number of terms |E ∗ | , which equals to (1 − η ) |G| , since M is inherent and the values of every terms are fixed.

Original: That is, in the phase of feature selection, the lower bound of the objective is in negative relation to η .
Revised: That is, in the phase of feature selection, the lower bound of the objective is in negative relation to η .

Original: It is worth to mention that Equation 5 is similar with the regularization terms proposed in GSAT (Miao et al., 2022).
Revised: It is worth to mention that Equation 5 is similar with the regularization terms proposed in GSAT (Miao et al., 2022).

Original: However, it is appropriate for mainly post-hoc explanation methods while terms proposed in GSAT is tailored for itself solely.
Revised: However, it is appropriate for mainly post-hoc explanation methods while terms proposed in GSAT is tailored for itself solely.

Original: Therefore, for an interpreting task appealing to low sparsity, the magnitude of objective in feature selection is large.
Revised: Therefore, for an interpreting task appealing to low sparsity, the magnitude of objective in feature selection is large.

Original: Since regularization is the tradeoff scheme between two phases ( Proposition 1 ), it should be loosened to a favorable direction for feature attribution to keep the balance, and vise versa.
Revised: Since regularization is the tradeoff scheme between two phases ( Proposition 1 ), it should be loosened to a favorable direction for feature attribution to keep the balance, and vise versa.

Original: We illustrate this process in Figure 2.
Revised: We illustrate this process in Figure 2.

Original: More formally:
Revised: More formally:

Original: • Proposition 2 (Sparsity-adaptive Regularization)
Revised: • Proposition 2 (Sparsity-adaptive Regularization)

Original: For a certain interpreting task, let K i and K j are the optimal coefficient vector of regularization under the predefined explanatory sparsity η i and η j .
Revised: For a certain interpreting task, let K i and K j are the optimal coefficient vector of regularization under the predefined explanatory sparsity η i and η j .

Original: For all η ∈ (0 , 1) we have:
Revised: For all η ∈ (0 , 1) we have:

Original: [Equation]
Revised: [Equation]

Original: According to
Revised: According to

Original: Proposition 2 , we propose a simple yet effective scheme called S parsity-adaptive
Revised: Proposition 2 , we propose a simple yet effective scheme called S parsity-adaptive

Original: Regularization Scheme (SRS) to enhance the performance of existing explainers.
Revised: Regularization Scheme (SRS) to enhance the performance of existing explainers.

Original: Concretely, SRS first
Revised: Concretely, SRS first

Original: seeks
Revised: seeks

Original: for the optimal
Revised: for the optimal

Original: coefficients K i under the certain
Revised: coefficients K i under the certain

Original: sparsity
Revised: sparsity

Original: η i
Revised: η i

Original: by grid search,
Revised: by grid search,

Original: then for
Revised: then for

Original: other sparsity
Revised: other sparsity

Original: η j , SRS sets K j according to K j = η
Revised: η j , SRS sets K j according to K j = η

Original: j η
Revised: j η

Original: i K i .
Revised: i K i .

Original: Specific implementation and experimental
Revised: Specific implementation and experimental

Original: verification are provided in Section 5.2.
Revised: verification are provided in Section 5.2.

Original: 4. R EGULARIZATION & S TOCHASTIC M ECHANISM
Revised: 4. R EGULARIZATION & S TOCHASTIC M ECHANISM

Original: In this part we focus on the relationship between the regularization and the stochastic mechanism.
Revised: In this part we focus on the relationship between the regularization and the stochastic mechanism.

Original: Recent years have witnessed a surge in research that focuses on leveraging the stochastic mechanism to enhance the performance of GNNs explainability (Luo et al., 2020;
Revised: Recent years have witnessed a surge in research that focuses on leveraging the stochastic mechanism to enhance the performance of GNNs explainability (Luo et al., 2020;

Original: Wang et al., 2021; Miao et al., 2022).
Revised: Wang et al., 2021; Miao et al., 2022).

Original: Concretely, the probability of graph G is first factorized as P ( G ) =
Revised: Concretely, the probability of graph G is first factorized as P ( G ) =

Original: Π ( i,j ) ∈E P ( e ij ) , where e ij = 1 if the edge ( i, j ) exists, and 0 otherwise.
Revised: Π ( i,j ) ∈E P ( e ij ) , where e ij = 1 if the edge ( i, j ) exists, and 0 otherwise.

Original: Then the Bernoulli distribution is employed to instantiate P ( e ij ) , where the stochastic mechanism is introduced in this process (Luo et al., 2020).
Revised: Then the Bernoulli distribution is employed to instantiate P ( e ij ) , where the stochastic mechanism is introduced in this process (Luo et al., 2020).

Original: Meanwhile, the gumbel-softmax reparameterization trick is applied to make sure the gradient in the Bernoulli distribution is traceable (Jang et al., 2017).
Revised: Meanwhile, the gumbel-softmax reparameterization trick is applied to make sure the gradient in the Bernoulli distribution is traceable (Jang et al., 2017).

Original: However, despite the satisfactory performance of the stochastic mechanism, current researches mainly neglected the theoretical basis of the stochastic mechanism.
Revised: However, despite the satisfactory performance of the stochastic mechanism, current researches mainly neglected the theoretical basis of the stochastic mechanism.

Original: We argue that the stochastic mechanism bolsters the explanation methods via facilitating the feature selection.
Revised: We argue that the stochastic mechanism bolsters the explanation methods via facilitating the feature selection.

Original: Specifically, employing the stochastic mechanism to the training stage of M can be regarded as adding random noise to M (Shwartz-Ziv & Tishby, 2017).
Revised: Specifically, employing the stochastic mechanism to the training stage of M can be regarded as adding random noise to M (Shwartz-Ziv & Tishby, 2017).

Original: The entropy of the mask’s distribution will continue to increase during this process under the constraints of the information bottleneck.
Revised: The entropy of the mask’s distribution will continue to increase during this process under the constraints of the information bottleneck.

Original: That in turn maximizes the conditional entropy H ( G|G ⊙ M ) .
Revised: That in turn maximizes the conditional entropy H ( G|G ⊙ M ) .

Original: Since the entropy of input graph H ( G ) is a constant, mutual information
Revised: Since the entropy of input graph H ( G ) is a constant, mutual information

Original: I ( G ⊙ M , G ) =
Revised: I ( G ⊙ M , G ) =

Original: H ( G ) − H ( G|G ⊙ M ) is minimized contributed to stochastic mechanism.
Revised: H ( G ) − H ( G|G ⊙ M ) is minimized contributed to stochastic mechanism.

Original: In conclusion, the stochastic mechanism can accelerate the process of minimizing I ( G ⊙ M , G ) , thus endowing the compression of the explanatory subgraph.
Revised: In conclusion, the stochastic mechanism can accelerate the process of minimizing I ( G ⊙ M , G ) , thus endowing the compression of the explanatory subgraph.

Original: That is, it can be regarded as the implicit regularization term in favor of the phase of feature selection.
Revised: That is, it can be regarded as the implicit regularization term in favor of the phase of feature selection.

Original: We summarize this rationale as Proposition 3 and verify it in Section 5.3.
Revised: We summarize this rationale as Proposition 3 and verify it in Section 5.3.

Original: • Proposition 3 (Rationale of Stochastic Mechanism)
Revised: • Proposition 3 (Rationale of Stochastic Mechanism)

Original: Stochastic mechanism works as the implicit regularization term, which endows GNNs explainability with better compressibility
Revised: Stochastic mechanism works as the implicit regularization term, which endows GNNs explainability with better compressibility

Original: 4. R EGULARIZATION & OOD
Revised: 4. R EGULARIZATION & OOD

Original: Many recent endeavors have been made towards revealing the out-of-distribution issue of the posthoc explanation methods (Chang et al., 2019;
Revised: Many recent endeavors have been made towards revealing the out-of-distribution issue of the posthoc explanation methods (Chang et al., 2019;

Original: Qiu et al., 2022; Wu et al., 2022; Miao et al., 2022).
Revised: Qiu et al., 2022; Wu et al., 2022; Miao et al., 2022).

Original: Concretely, OOD issue is posed in the data space since the distribution of full graph P ( G ) differs from that of subgraph P ( G s ) w.r.t.
Revised: Concretely, OOD issue is posed in the data space since the distribution of full graph P ( G ) differs from that of subgraph P ( G s ) w.r.t.

Original: some properties of graph data such as size (Bevilacqua et al., 2021), degree (Tang et al., 2020) and homophily (Lei et al., 2022).
Revised: some properties of graph data such as size (Bevilacqua et al., 2021), degree (Tang et al., 2020) and homophily (Lei et al., 2022).

Original: Thus it is fallacious to leverage the original GNN f ′ , which is trained for the full graphs, to estimate the MI in terms of subgraphs.
Revised: Thus it is fallacious to leverage the original GNN f ′ , which is trained for the full graphs, to estimate the MI in terms of subgraphs.

Original: Unfortunately, most previous explainers (Ying et al., 2019; Luo et al., 2020; Vu & Thai, 2020; Miao et al., 2022) simply feed G ⊙ M into f ′ , unaware of that I ( f ( G ⊙ M ); Y ) is not proportional to
Revised: Unfortunately, most previous explainers (Ying et al., 2019; Luo et al., 2020; Vu & Thai, 2020; Miao et al., 2022) simply feed G ⊙ M into f ′ , unaware of that I ( f ( G ⊙ M ); Y ) is not proportional to

Original: I ( G ⊙ M ; Y ) , sometimes not even close.
Revised: I ( G ⊙ M ; Y ) , sometimes not even close.

Original: To reveal the relationship between OOD issue and regularization, we scrutinize the OOD between P ( G ) and P ( G s ) and decouple it into two phases: (1) OOD-AT between P ( G ) and P ( G ⊙ M ) posed in feature attribution and (2) OOD-SE between P ( G ⊙ M ) and P ( G s ) posed in feature selection.
Revised: To reveal the relationship between OOD issue and regularization, we scrutinize the OOD between P ( G ) and P ( G s ) and decouple it into two phases: (1) OOD-AT between P ( G ) and P ( G ⊙ M ) posed in feature attribution and (2) OOD-SE between P ( G ⊙ M ) and P ( G s ) posed in feature selection.

Original: OOD-AT.
Revised: OOD-AT.

Original: We first focus on OOD-AT posed in feature attribution.
Revised: We first focus on OOD-AT posed in feature attribution.

Original: According to Equation 3, feature attribution searches for the optimal M via calculating the cross-entropy H ( f ( G ⊙ M ) , Y ) to approximate I ( G ⊙ M ; Y ) .
Revised: According to Equation 3, feature attribution searches for the optimal M via calculating the cross-entropy H ( f ( G ⊙ M ) , Y ) to approximate I ( G ⊙ M ; Y ) .

Original: The approximation error, also be illustrated as gap1 in Figure 1, is given as the sum of second and third term of Equation 3.
Revised: The approximation error, also be illustrated as gap1 in Figure 1, is given as the sum of second and third term of Equation 3.

Original: For the second term, H ( Y ) , since it is the constant across different candidates M , it will not affect the search results.
Revised: For the second term, H ( Y ) , since it is the constant across different candidates M , it will not affect the search results.

Original: Unfortunately, for the third term, KL ( P ( Y | G ⊙ M )
Revised: Unfortunately, for the third term, KL ( P ( Y | G ⊙ M )

Original: ||
Revised: ||

Original: P θ
Revised: P θ

Original: ( Y | G ⊙ M )) , OOD-AT makes this part far away from zero and inevitably fluctuate across different M .
Revised: ( Y | G ⊙ M )) , OOD-AT makes this part far away from zero and inevitably fluctuate across different M .

Original: This fluctuation degrades the fairness of the searching process of optimal M , and further degenerates the effectiveness of the mapping, G (cid:55)→ G ⊙ M in feature attribution.
Revised: This fluctuation degrades the fairness of the searching process of optimal M , and further degenerates the effectiveness of the mapping, G (cid:55)→ G ⊙ M in feature attribution.

Original: Thus we formulate the tangible impact caused by OOD-AT as:
Revised: Thus we formulate the tangible impact caused by OOD-AT as:

Original: [Equation]
Revised: [Equation]

Original: where D is the Variance.
Revised: where D is the Variance.

Original: If regularization pushes P ( G ⊙ M ) close to P ( G ) via constraining M to A , the Equation 7 will close to zero and OOD-AT will cause little degradation.
Revised: If regularization pushes P ( G ⊙ M ) close to P ( G ) via constraining M to A , the Equation 7 will close to zero and OOD-AT will cause little degradation.

Original: Note that this process is similar to the left side of Figure 1: the constrain orientation is loosened to A to eliminate the gap.
Revised: Note that this process is similar to the left side of Figure 1: the constrain orientation is loosened to A to eliminate the gap.

Original: OOD-SE.
Revised: OOD-SE.

Original: We then focus on OOD-SE posed in feature selection.
Revised: We then focus on OOD-SE posed in feature selection.

Original: While feature selection binarizes
Revised: While feature selection binarizes

Original: M to get M ′ and generates G s =
Revised: M to get M ′ and generates G s =

Original: G ⊙ M ′ , the degree of OOD-SE can be simplified as the distance between M and A s .
Revised: G ⊙ M ′ , the degree of OOD-SE can be simplified as the distance between M and A s .

Original: That is, OOD-SE issue will be remedied if regularization pushes P ( G ⊙ M ) close to P ( G s ) via constraining M to A s .
Revised: That is, OOD-SE issue will be remedied if regularization pushes P ( G ⊙ M ) close to P ( G s ) via constraining M to A s .

Original: In conclusion, just similar to the dilemma of regularization between two phases described in Section 3.2, the optimal regularization scheme for remedying OOD-AT and OOD-SE is totally different.
Revised: In conclusion, just similar to the dilemma of regularization between two phases described in Section 3.2, the optimal regularization scheme for remedying OOD-AT and OOD-SE is totally different.

Original: That is, for a certain interpreting task, there is inherent uncooperativeness between OOD in feature attribution and OOD in feature selection.
Revised: That is, for a certain interpreting task, there is inherent uncooperativeness between OOD in feature attribution and OOD in feature selection.

Original: Note that the above analysis also explains why the OOD issue is the inherent limitation of post-hoc explainability: since we can only remedy one side and ignore the other side, OOD can not be thoroughly settled by adjusting the training objective and hyperparameters solely.
Revised: Note that the above analysis also explains why the OOD issue is the inherent limitation of post-hoc explainability: since we can only remedy one side and ignore the other side, OOD can not be thoroughly settled by adjusting the training objective and hyperparameters solely.

Original: We formulate this proposition as:
Revised: We formulate this proposition as:

Original: Proposition 3 (Rationale of OOD)
Revised: Proposition 3 (Rationale of OOD)

Original: For a certain interpreting task, the OOD issue is imputed to the contradictory effects of regularization on the phases of feature attribution and selection.
Revised: For a certain interpreting task, the OOD issue is imputed to the contradictory effects of regularization on the phases of feature attribution and selection.

Original: Even though the OOD issue is inherent, we introduce a simple yet effective regularization scheme called OOD-resistant Regularization Scheme (ORS) to alleviate the above dilemma to some extent.
Revised: Even though the OOD issue is inherent, we introduce a simple yet effective regularization scheme called OOD-resistant Regularization Scheme (ORS) to alleviate the above dilemma to some extent.

Original: Specifically, in the early stage of training, loose regularization of M is performed in favor of alleviating OOD-AT for the more precise mapping G (cid:55)→ G ⊙ M .
Revised: Specifically, in the early stage of training, loose regularization of M is performed in favor of alleviating OOD-AT for the more precise mapping G (cid:55)→ G ⊙ M .

Original: Similarly, we then gradually tighten the regularization of M to alleviate the OOD-SE for the mapping
Revised: Similarly, we then gradually tighten the regularization of M to alleviate the OOD-SE for the mapping

Original: G ⊙ M
Revised: G ⊙ M

Original: (cid:55)→ G s .
Revised: (cid:55)→ G s .

Original: ORS can achieve better explanation performance than the common invariable regularization scheme.
Revised: ORS can achieve better explanation performance than the common invariable regularization scheme.

Original: Specific implementation and experimental verification are provided in Section 5.3.
Revised: Specific implementation and experimental verification are provided in Section 5.3.

Original:  E XPERIMENT
Revised:  E XPERIMENT

Original: In this section, we conduct extensive experiments to answer the following questions:
Revised: In this section, we conduct extensive experiments to answer the following questions:

Original: • RQ1: Can Proposition 1 provided in Section 3.2 and Proposition 2 provided in Section 4.1 be verified?
Revised: • RQ1: Can Proposition 1 provided in Section 3.2 and Proposition 2 provided in Section 4.1 be verified?

Original: Furthermore, how does the SRS perform?
Revised: Furthermore, how does the SRS perform?

Original: • RQ2: Can Proposition 3 provided in Section 4.2 and Proposition 4 provided in Section 4.3 be verified?
Revised: • RQ2: Can Proposition 3 provided in Section 4.2 and Proposition 4 provided in Section 4.3 be verified?

Original: Furthermore, how does the ORS perform?
Revised: Furthermore, how does the ORS perform?

Original: 5. S ETTINGS
Revised: 5. S ETTINGS

Original: Datasets and Target GNNs.
Revised: Datasets and Target GNNs.

Original: We use one synthetic dataset and two real datasets which are publicly accessible and train three popular GNN models for them.
Revised: We use one synthetic dataset and two real datasets which are publicly accessible and train three popular GNN models for them.

Original: More details can be found in Appendix B.
Revised: More details can be found in Appendix B.

Original: • BA-3motifs (Ying et al., 2019) contains 3,000 graphs attaching with the motif types: house, cycle, or grid, for which a trained ASAP (Ranjan et al., 2020), has achieved a 99.3% testing accuracy.
Revised: • BA-3motifs (Ying et al., 2019) contains 3,000 graphs attaching with the motif types: house, cycle, or grid, for which a trained ASAP (Ranjan et al., 2020), has achieved a 99.3% testing accuracy.

Original: • MUTAG (Kazius et al., 2005) contains 4,337 molecule graphs categorized into two classes based on their mutagenic effect on the Gram-negative bacterium.
Revised: • MUTAG (Kazius et al., 2005) contains 4,337 molecule graphs categorized into two classes based on their mutagenic effect on the Gram-negative bacterium.

Original: The well-trained Graph Isomorphism Network (GIN) (Xu et al., 2019) has achieved a 97.7% testing accuracy.
Revised: The well-trained Graph Isomorphism Network (GIN) (Xu et al., 2019) has achieved a 97.7% testing accuracy.

Original: • MNIST (Monti et al., 2017) superpixel converts 70,000 images into the graphs of superpixel adjacency.
Revised: • MNIST (Monti et al., 2017) superpixel converts 70,000 images into the graphs of superpixel adjacency.

Original: A trained Spline-based GNN (Fey et al., 2018) has achieved a 94.6% testing accuracy.
Revised: A trained Spline-based GNN (Fey et al., 2018) has achieved a 94.6% testing accuracy.

Original: Evaluation Metrics and Baselines.
Revised: Evaluation Metrics and Baselines.

Original: We select three commonly used metrics to evaluate our results:
Revised: We select three commonly used metrics to evaluate our results:

Original: Accuracy, Precision, and Fidelity.
Revised: Accuracy, Precision, and Fidelity.

Original: Moreover, we leverage six state-of-the-art methods to verify proposed propositions and schemes, covering the GNNExplainer (Ying et al., 2019), PGExplainer (Luo et al., 2020), GraphMask (Schlichtkrull et al., 2021), CF-GNNExplainer (Lucic et al., 2022),
Revised: Moreover, we leverage six state-of-the-art methods to verify proposed propositions and schemes, covering the GNNExplainer (Ying et al., 2019), PGExplainer (Luo et al., 2020), GraphMask (Schlichtkrull et al., 2021), CF-GNNExplainer (Lucic et al., 2022),

Original: Refine (Wang et al., 2021) and GSAT (Miao et al., 2022).
Revised: Refine (Wang et al., 2021) and GSAT (Miao et al., 2022).

Original: More details is provided in Appendix B.
Revised: More details is provided in Appendix B.

Original: 5. E VALUATION OF P ROPOSITION 1, P ROPOSITION 2 AND SRS (RQ1)
Revised: 5. E VALUATION OF P ROPOSITION 1, P ROPOSITION 2 AND SRS (RQ1)

Original: We first focus on verifying the Proposition 1 and Proposition 2 provided in Section 3.2 and Section 4.1 simultaneously.
Revised: We first focus on verifying the Proposition 1 and Proposition 2 provided in Section 3.2 and Section 4.1 simultaneously.

Original: Since the role of regularization is to keep the balance between two phases ( Proposition 1 ), the variation of its coefficient K will break the balance and ulteriorly impose opposite impacts on the performance under different sparsity η ( Proposition 2 ).
Revised: Since the role of regularization is to keep the balance between two phases ( Proposition 1 ), the variation of its coefficient K will break the balance and ulteriorly impose opposite impacts on the performance under different sparsity η ( Proposition 2 ).

Original: In sight of this, we start with the optimal coefficients of regularization for ACC-AUC by grid search, then we deliberately increase K and record the accuracy under different η .
Revised: In sight of this, we start with the optimal coefficients of regularization for ACC-AUC by grid search, then we deliberately increase K and record the accuracy under different η .

Original: Experiment results are shown in Figure 4, where the horizontal axis represents the multiple of the increment of K and the length of marker shows the variance.
Revised: Experiment results are shown in Figure 4, where the horizontal axis represents the multiple of the increment of K and the length of marker shows the variance.

Original: Note that the performance of other baselines have similar trends to the results in Figure 4.
Revised: Note that the performance of other baselines have similar trends to the results in Figure 4.

Original: [Table]
Revised: [Table]

Original: According to Figure 4 we have the following observations:
Revised: According to Figure 4 we have the following observations:

Original: • While K is changed, the fluctuation of overall performance (
Revised: • While K is changed, the fluctuation of overall performance (

Original: i
Revised: i

Original: e. , ACC-AUC) is not too strong ( ± 7 .
Revised: e. , ACC-AUC) is not too strong ( ± 7 .

Original: 25% ) on average.
Revised: 25% ) on average.

Original: However, the fluctuation of ACC in high sparsity and low sparsity is obviously larger ( ± 21 .
Revised: However, the fluctuation of ACC in high sparsity and low sparsity is obviously larger ( ± 21 .

Original: 86% ) than that of ACC-AUC.
Revised: 86% ) than that of ACC-AUC.

Original: These observations verify the role of regularization: a tradeoff to keep the balance and guarantee the overall performance at a high level.
Revised: These observations verify the role of regularization: a tradeoff to keep the balance and guarantee the overall performance at a high level.

Original: • ACC under low sparsity and ACC-AUC are decreasing with the increment of the regularization coefficient.
Revised: • ACC under low sparsity and ACC-AUC are decreasing with the increment of the regularization coefficient.

Original: On the contrary, ACC in high sparsity is increasing by a large margin ( 29 .
Revised: On the contrary, ACC in high sparsity is increasing by a large margin ( 29 .

Original: 63% ↑ ) on average.
Revised: 63% ↑ ) on average.

Original: These increases up to a maximum of 67 .
Revised: These increases up to a maximum of 67 .

Original: 83% in MUTAG are counterintuitive and eccentric.
Revised: 83% in MUTAG are counterintuitive and eccentric.

Original: Whereas, these observations exactly conform to the theoretical analysis in Propositon 2 : loose constraint ( i.e. , large K ) is beneficial to the performance in high sparsity, while tight constraint ( i.e. , small K ) is in favor of the performance under low sparsity.
Revised: Whereas, these observations exactly conform to the theoretical analysis in Propositon 2 : loose constraint ( i.e. , large K ) is beneficial to the performance in high sparsity, while tight constraint ( i.e. , small K ) is in favor of the performance under low sparsity.

Original: While the above observations verify the Proposition 1 and Proposition 2 , we now evaluate the effectiveness of the derived scheme, SRS.
Revised: While the above observations verify the Proposition 1 and Proposition 2 , we now evaluate the effectiveness of the derived scheme, SRS.

Original: Specifically, for a certain interpreting task, we first seek for the optimal K under η = 0 .
Revised: Specifically, for a certain interpreting task, we first seek for the optimal K under η = 0 .

Original: 5 by grid search, then we reduce or enlarge K proportionally to the variation of η in (0 . 1 , ..., 0 . 9) .
Revised: 5 by grid search, then we reduce or enlarge K proportionally to the variation of η in (0 . 1 , ..., 0 . 9) .

Original: Results are summarized in Table 1.
Revised: Results are summarized in Table 1.

Original: We have the following observations:
Revised: We have the following observations:

Original: • The baseline explainers enhanced by SRS outperform themselves in all cases.
Revised: • The baseline explainers enhanced by SRS outperform themselves in all cases.

Original: To be more specific, SRS achieves significant improvements over the six baselines w.r.t.
Revised: To be more specific, SRS achieves significant improvements over the six baselines w.r.t.

Original: fidelity by 7.9% and 7.2% averagely in MUTAG and BA3-motif, respectively.
Revised: fidelity by 7.9% and 7.2% averagely in MUTAG and BA3-motif, respectively.

Original: The improvement is up to a maximum of 12.8% when GNNExplainer is employed to interpret graphs in MUTAG.
Revised: The improvement is up to a maximum of 12.8% when GNNExplainer is employed to interpret graphs in MUTAG.

Original: It demonstrates the effectiveness and universality of SRS, and verifies that SRS can be leveraged to boost the accuracy of current explainers.
Revised: It demonstrates the effectiveness and universality of SRS, and verifies that SRS can be leveraged to boost the accuracy of current explainers.

Original: We attribute these improvements to the ability that SRS can allocate the most adaptive coefficient of regularization for diverse explanatory sparsity.
Revised: We attribute these improvements to the ability that SRS can allocate the most adaptive coefficient of regularization for diverse explanatory sparsity.

Original: • SRS provides much stabler explanations than the baselines for the much smaller variance.
Revised: • SRS provides much stabler explanations than the baselines for the much smaller variance.

Original: More specifically, STD of SRS outperforms baseline by a larger margin (22.8% ↓ ) on average.
Revised: More specifically, STD of SRS outperforms baseline by a larger margin (22.8% ↓ ) on average.

Original: This phenomenon verifies that the performance of most explainers is not stable since it depends on the matching degree between the task and regularization terms, while SRS can avoid this dilemma by adaptively allocating regularization terms.
Revised: This phenomenon verifies that the performance of most explainers is not stable since it depends on the matching degree between the task and regularization terms, while SRS can avoid this dilemma by adaptively allocating regularization terms.

Original: 5. E VALUATION OF P ROPOSITION 3, P ROPOSITION 4 AND ORS (RQ2)
Revised: 5. E VALUATION OF P ROPOSITION 3, P ROPOSITION 4 AND ORS (RQ2)

Original: We now focus on verifying the Proposition 3 proposed in Section 4.2 and the Proposition 4 proposed in Section 4.3.
Revised: We now focus on verifying the Proposition 3 proposed in Section 4.2 and the Proposition 4 proposed in Section 4.3.

Original: According to Proposition 3 , the stochastic mechanism endows GNNs explainability with better compressibility, which has a similar rationale to the regularization term.
Revised: According to Proposition 3 , the stochastic mechanism endows GNNs explainability with better compressibility, which has a similar rationale to the regularization term.

Original: In sight of this, we gradually increase the magnitude of stochasticity in explainers and observe whether the performance has the same trends as the results in Figure 4.
Revised: In sight of this, we gradually increase the magnitude of stochasticity in explainers and observe whether the performance has the same trends as the results in Figure 4.

Original: The overall results are summarized in
Revised: The overall results are summarized in

Original: Figure 5.
Revised: Figure 5.

Original: We have the following observations:
Revised: We have the following observations:

Original: • The ACC under low sparsity and ACC-AUC are decreasing ( 8 .
Revised: • The ACC under low sparsity and ACC-AUC are decreasing ( 8 .

Original: 76% ↓ and 19 .
Revised: 76% ↓ and 19 .

Original: 85% ↓ ) with the increment of stochasticity magnitude, while the ACC under high sparsity is increasing ( 14 . 52% ↑ ) in this case.
Revised: 85% ↓ ) with the increment of stochasticity magnitude, while the ACC under high sparsity is increasing ( 14 . 52% ↑ ) in this case.

Original: These increases up to a maximum of 37 .
Revised: These increases up to a maximum of 37 .

Original: 03% in MUTAG.
Revised: 03% in MUTAG.

Original: Therefore, the large magnitude of stochasticity is beneficial to the performance in high sparsity and vice versa.
Revised: Therefore, the large magnitude of stochasticity is beneficial to the performance in high sparsity and vice versa.

Original: This effect is similar to the effect of regularization terms.
Revised: This effect is similar to the effect of regularization terms.

Original: In conclusion, the same trends between Figure
Revised: In conclusion, the same trends between Figure

Original: and Figure 5 verify that the stochastic mechanism has a similar rationale with the regularization terms, and further verify the Proposition 3 .
Revised: and Figure 5 verify that the stochastic mechanism has a similar rationale with the regularization terms, and further verify the Proposition 3 .

Original: [Table]
Revised: [Table]

Original: In terms of Proposition 4 , since the degree of OOD is intractable, we directly leverage the derived scheme: ORS to remedy the
Revised: In terms of Proposition 4 , since the degree of OOD is intractable, we directly leverage the derived scheme: ORS to remedy the

Original: OOD issue in explainers.
Revised: OOD issue in explainers.

Original: We surmise the results in Table 2.
Revised: We surmise the results in Table 2.

Original: We find that:
Revised: We find that:

Original: • The explainers enhanced by ORS get the higher accuracy on the graph classification tasks than themselves.
Revised: • The explainers enhanced by ORS get the higher accuracy on the graph classification tasks than themselves.

Original: Specifically, ORS achieves significant improvements over the strongest baselines w.r.t.
Revised: Specifically, ORS achieves significant improvements over the strongest baselines w.r.t.

Original: ACCAUC by 4.1% and 3.5% in MUTAG and BA3-motif, respectively.
Revised: ACCAUC by 4.1% and 3.5% in MUTAG and BA3-motif, respectively.

Original: These improvements verify the reliability and effectiveness of the ORS.
Revised: These improvements verify the reliability and effectiveness of the ORS.

Original: We contribute these improvements mainly to the advantage of remedying the OOD issue in the phases of feature attribution and selection.
Revised: We contribute these improvements mainly to the advantage of remedying the OOD issue in the phases of feature attribution and selection.

Original: This observation further validates the theoretical analysis of the OOD issue in the above two phases ( Proposition 4 ) indirectly.
Revised: This observation further validates the theoretical analysis of the OOD issue in the above two phases ( Proposition 4 ) indirectly.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: In this work, we rethink the role of regularization in GNNs explainability from the perspective of information theory.
Revised: In this work, we rethink the role of regularization in GNNs explainability from the perspective of information theory.

Original: We first retrospect the concept of GIB and derive the new GIB form tailored for GNNs explainability: GIBE.
Revised: We first retrospect the concept of GIB and derive the new GIB form tailored for GNNs explainability: GIBE.

Original: The role of regularization in the phases of feature attribution and selection are explored respectively under the guidance of GIBE.
Revised: The role of regularization in the phases of feature attribution and selection are explored respectively under the guidance of GIBE.

Original: Moreover, four intriguing propositions and two common optimization schemes of regularization are introduced inspired by the above insight.
Revised: Moreover, four intriguing propositions and two common optimization schemes of regularization are introduced inspired by the above insight.

Original: Extensive experiments are conducted on both synthetic and real-world datasets to validate the rationality of our propositions and the superiority of our scheme.
Revised: Extensive experiments are conducted on both synthetic and real-world datasets to validate the rationality of our propositions and the superiority of our scheme.

Original: This work represents an initial attempt of digging deeper into the regularization in explainability theoretically, which provides a new perspective to understand the role of regularization.
Revised: This work represents an initial attempt of digging deeper into the regularization in explainability theoretically, which provides a new perspective to understand the role of regularization.

Original: E THICS S TATEMENT
Revised: E THICS S TATEMENT

Original: In this work, we rethink the role of regularization in GNNs explainability from the perspective of information theory, where no human subject is related.
Revised: In this work, we rethink the role of regularization in GNNs explainability from the perspective of information theory, where no human subject is related.

Original: We believe digging deeper into the regularization theoretically is beneficial for performing the better explanations and consequently improving the model’s transparency in real-world applications.
Revised: We believe digging deeper into the regularization theoretically is beneficial for performing the better explanations and consequently improving the model’s transparency in real-world applications.

Original: R EPRODUCIBILITY
Revised: R EPRODUCIBILITY

Original: We summarize the efforts made to ensure reproducibility in this work.
Revised: We summarize the efforts made to ensure reproducibility in this work.

Original: (1) Datasets: We use one synthetic dataset and two real datasets which are publicly accessible, where the processing details are included in Appendix B. (2) Model Training: We provide the training details (including hyperparameter settings) in Appendix B which are consistent with our implementation in the code ( cf.
Revised: (1) Datasets: We use one synthetic dataset and two real datasets which are publicly accessible, where the processing details are included in Appendix B. (2) Model Training: We provide the training details (including hyperparameter settings) in Appendix B which are consistent with our implementation in the code ( cf.

Original: the anonymous link provided in Abstract).
Revised: the anonymous link provided in Abstract).

Original: (3) Theoretical Results: All assumptions and proofs can be referred to Appendix A.
Revised: (3) Theoretical Results: All assumptions and proofs can be referred to Appendix A.

Original: A T HE P ROCESS OF D ERIVATION IN TERMS OF I NFORMATION T HEORY A. D ERIVATION OF GIBE
Revised: A T HE P ROCESS OF D ERIVATION IN TERMS OF I NFORMATION T HEORY A. D ERIVATION OF GIBE

Original: We focus on the detailed derivation of GIBE.
Revised: We focus on the detailed derivation of GIBE.

Original: Firstly, the proportion of mutual information: invariance to invertible transformations (Shwartz-Ziv & Tishby, 2017) can be formulated as:
Revised: Firstly, the proportion of mutual information: invariance to invertible transformations (Shwartz-Ziv & Tishby, 2017) can be formulated as:

Original: [Equation]
Revised: [Equation]

Original: for any invertible functions ψ and ϕ .
Revised: for any invertible functions ψ and ϕ .

Original: Since Y is determined by G ∗ in the sense that Y =
Revised: Since Y is determined by G ∗ in the sense that Y =

Original: f ( G ∗ )
Revised: f ( G ∗ )

Original: + ϵ for some deterministic invertible function f , according to Equation 8, I ( G s ; Y ) in the objective of
Revised: + ϵ for some deterministic invertible function f , according to Equation 8, I ( G s ; Y ) in the objective of

Original: GIB ( i.e. , Equation 1) can be rewrote as (Miao et al., 2022):
Revised: GIB ( i.e. , Equation 1) can be rewrote as (Miao et al., 2022):

Original: [Equation]
Revised: [Equation]

Original: Thus we can decompose the first term in the objective of GIB into:
Revised: Thus we can decompose the first term in the objective of GIB into:

Original: [Equation]
Revised: [Equation]

Original: for any β ∈ (0 , 1) .
Revised: for any β ∈ (0 , 1) .

Original: Comparing Equation 10 and Equation 1, the objective of GIB can be rewritten as:
Revised: Comparing Equation 10 and Equation 1, the objective of GIB can be rewritten as:

Original: [Equation]
Revised: [Equation]

Original: We then substitute the coefficient α = β/
Revised: We then substitute the coefficient α = β/

Original: (1 − β ) into above equation:
Revised: (1 − β ) into above equation:

Original: [Equation]
Revised: [Equation]

Original: Taking G ⊙ M as the substitution of G s , the new form of GIB for explainability can be derived as:
Revised: Taking G ⊙ M as the substitution of G s , the new form of GIB for explainability can be derived as:

Original: [Equation]
Revised: [Equation]

Original: which is the objective of GIBE.
Revised: which is the objective of GIBE.

Original: A. P ROOF OF T HE C ONVERGENCE OF GIBE
Revised: A. P ROOF OF T HE C ONVERGENCE OF GIBE

Original: Theorem A.1.
Revised: Theorem A.1.

Original: For any α ∈ (0 , + ∞ ) we have: G ⊙ M =
Revised: For any α ∈ (0 , + ∞ ) we have: G ⊙ M =

Original: G ∗ maximizes the objective of GIBE:
Revised: G ∗ maximizes the objective of GIBE:

Original: I ( G ⊙ M ; Y ) + α
Revised: I ( G ⊙ M ; Y ) + α

Original: [ I ( G ⊙ M ; G ∗ ) − I ( G ⊙ M ; G )] .
Revised: [ I ( G ⊙ M ; G ∗ ) − I ( G ⊙ M ; G )] .

Original: Proof .
Revised: Proof .

Original: Consider the following derivation,
Revised: Consider the following derivation,

Original: [Equation]
Revised: [Equation]

Original: Since I ( Y ; G ∗ ) is a constant, for any α ∈ (0 , + ∞ ) , G ⊙ M that maximizes the objective of GIBE also minimizes I ( G ∗ ; Y | G ⊙ M ) + αI ( G ∗ ; G ⊙ M | Y ) .
Revised: Since I ( Y ; G ∗ ) is a constant, for any α ∈ (0 , + ∞ ) , G ⊙ M that maximizes the objective of GIBE also minimizes I ( G ∗ ; Y | G ⊙ M ) + αI ( G ∗ ; G ⊙ M | Y ) .

Original: As I ( G ∗ ; Y | G ⊙ M ) > 0 and
Revised: As I ( G ∗ ; Y | G ⊙ M ) > 0 and

Original: I ( G ∗ ; G ⊙ M | Y ) > 0 , so the lower bound of I ( G ∗ ; Y | G ⊙ M ) + αI ( G ∗ ; G ⊙ M | Y ) is 0.
Revised: I ( G ∗ ; G ⊙ M | Y ) > 0 , so the lower bound of I ( G ∗ ; Y | G ⊙ M ) + αI ( G ∗ ; G ⊙ M | Y ) is 0.

Original: We have G ⊙ M =
Revised: We have G ⊙ M =

Original: G ∗ can make I ( G ∗ ; Y | G ⊙ M ) + αI ( G ∗ ; G ⊙ M | Y )
Revised: G ∗ can make I ( G ∗ ; Y | G ⊙ M ) + αI ( G ∗ ; G ⊙ M | Y )

Original: = 0 .
Revised: = 0 .

Original: This is because (1)
Revised: This is because (1)

Original: Y =
Revised: Y =

Original: f ( G ∗ )
Revised: f ( G ∗ )

Original: + ϵ where ϵ is independent of G so that I ( G ∗ ; Y | G⊙ M )
Revised: + ϵ where ϵ is independent of G so that I ( G ∗ ; Y | G⊙ M )

Original: = 

Revised: =

Original: and (2) G ∗ =
Revised: and (2) G ∗ =

Original: f − 1 ( Y − ϵ ) where ϵ is independent of G so that I ( G ∗ ; G ⊙ M | Y ) = 0 .
Revised: f − 1 ( Y − ϵ ) where ϵ is independent of G so that I ( G ∗ ; G ⊙ M | Y ) = 0 .

Original: Therefore, G ⊙ M = G ∗ maximizes the objective of GIBE in Equation 2.
Revised: Therefore, G ⊙ M = G ∗ maximizes the objective of GIBE in Equation 2.

Original: A. E STIMATION OF M UTUAL I NFORMATION IN F EATURE A TTRIBUTION
Revised: A. E STIMATION OF M UTUAL I NFORMATION IN F EATURE A TTRIBUTION

Original: According to the definition of mutual information:
Revised: According to the definition of mutual information:

Original: [Equation]
Revised: [Equation]

Original: Since P ( Y | G ⊙ M ) is intractable, we introduce a variational approximation P θ ( Y | G ⊙ M ) for it.
Revised: Since P ( Y | G ⊙ M ) is intractable, we introduce a variational approximation P θ ( Y | G ⊙ M ) for it.

Original: Then, we can rewrite Equation 15 as:
Revised: Then, we can rewrite Equation 15 as:

Original: [Equation]
Revised: [Equation]

Original: A.
Revised: A.

Original: V ARIATIONAL B OUNDS FOR M UTUAL I NFORMATION IN F EATURE S ELECTION
Revised: V ARIATIONAL B OUNDS FOR M UTUAL I NFORMATION IN F EATURE S ELECTION

Original: We now focus on the lower bounds for mutual information in the phase of feature selection:
Revised: We now focus on the lower bounds for mutual information in the phase of feature selection:

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: For the first term I ( G ⊙ M ; G ∗ ) , by definition:
Revised: For the first term I ( G ⊙ M ; G ∗ ) , by definition:

Original: [Equation]
Revised: [Equation]

Original: Since P ( G ⊙ M | G ∗ ) is intractable
Revised: Since P ( G ⊙ M | G ∗ ) is intractable

Original: [Equation]
Revised: [Equation]

Original: we introduce a variational approximation P ϕ
Revised: we introduce a variational approximation P ϕ

Original: ( G ⊙ M | G ∗ ) for it.
Revised: ( G ⊙ M | G ∗ ) for it.

Original: Then
Revised: Then

Original: [Equation]
Revised: [Equation]

Original: we obtain a lower bound for Equation 18:
Revised: we obtain a lower bound for Equation 18:

Original: [Equation]
Revised: [Equation]

Original: Then for the second term I ( G ⊙ M ; G ) we have:
Revised: Then for the second term I ( G ⊙ M ; G ) we have:

Original: [Equation]
Revised: [Equation]

Original: Since P ( G ⊙ M ) is intractable, we introduce a variational approximation Q for it.
Revised: Since P ( G ⊙ M ) is intractable, we introduce a variational approximation Q for it.

Original: Then, we obtain a upper bound for Equation 20:
Revised: Then, we obtain a upper bound for Equation 20:

Original: [Equation]
Revised: [Equation]

Original: Plugging in Equation 19 and Equation [Equation] we obtain a variational lower bound of Equation 17 as the objective of feature selection:
Revised: Plugging in Equation 19 and Equation 21, we obtain a variational lower bound of Equation 17 as the objective of feature selection:

Original: [Equation]
Revised: [Equation]

Original: B E XPERIMENT S ETTING
Revised: B E XPERIMENT S ETTING

Original: Datasets and Target GNNs.
Revised: Datasets and Target GNNs.

Original: We use one synthetic dataset and two real datasets which are publicly accessible.
Revised: We use one synthetic dataset and two real datasets which are publicly accessible.

Original: Three popular GNN models are trained to perform graph classification.
Revised: Three popular GNN models are trained to perform graph classification.

Original: The statistics of datasets and the configurations of GNN models are summarized in table 3.
Revised: The statistics of datasets and the configurations of GNN models are summarized in table 3.

Revised: Note that some benchmark datasets may not satisfy the assumption in Section 2, for further exploration, we still take them into consideration.

Original: • MUTAG (Kazius et al., 2005; Riesen & Bunke, 2008) contains 4,337 molecule graphs categorized into two classes based on their mutagenic effect on the Gram-negative bacterium.
Revised: • MUTAG (Kazius et al., 2005; Riesen & Bunke, 2008) contains 4,337 molecule graphs categorized into two classes based on their mutagenic effect on the Gram-negative bacterium.

Original: • BA-3motifs (Ying et al., 2019; Luo et al., 2020) contains 3,000 graphs attaching with one of three motif types: house, cycle, and grid, where Barabasi-Albert (BA) graphs are adopted as the base.
Revised: • BA-3motifs (Ying et al., 2019; Luo et al., 2020) contains 3,000 graphs attaching with one of three motif types: house, cycle, and grid, where Barabasi-Albert (BA) graphs are adopted as the base.

Original: • MNIST (Monti et al., 2017; Deng, 2012) superpixel dataset converts 70,000 images into the graphs of superpixel adjacency, where every graph is labeled as one of ten digit classes.
Revised: • MNIST (Monti et al., 2017; Deng, 2012) superpixel dataset converts 70,000 images into the graphs of superpixel adjacency, where every graph is labeled as one of ten digit classes.

Original: GIN (Xu et al., 2019) suppresses the popular GNN variants, such as Graph Convolutional Networks and GraphSAGE in terms of expressive power, as it generalizes the Weisfeiler Lehman graph isomorphism test and hence achieves maximum discriminative power under the neighborhood aggregation framework.
Revised: GIN (Xu et al., 2019) suppresses the popular GNN variants, such as Graph Convolutional Networks and GraphSAGE in terms of expressive power, as it generalizes the Weisfeiler Lehman graph isomorphism test and hence achieves maximum discriminative power under the neighborhood aggregation framework.

Original: • ASAP (Ranjan et al., 2020) utilizes a self-attention network along with a modified GNN formulation to capture the importance of each node in a given graph, and learns a sparse soft cluster assignment for nodes at each layer to effectively pool the subgraphs to form the pooled graph.
Revised: • ASAP (Ranjan et al., 2020) utilizes a self-attention network along with a modified GNN formulation to capture the importance of each node in a given graph, and learns a sparse soft cluster assignment for nodes at each layer to effectively pool the subgraphs to form the pooled graph.

Original: • Spline-based GNN (Fey et al., 2018) adopts a novel convolution operator based on B-splines, which operates in the spatial domain and aggregates local features, applying a trainable continuous kernel function parametrized by trainable B-spline control values and allowing very fast training and inference computation.[Table]
Revised: • Spline-based GNN (Fey et al., 2018) adopts a novel convolution operator based on B-splines, which operates in the spatial domain and aggregates local features, applying a trainable continuous kernel function parametrized by trainable B-spline control values and allowing very fast training and inference computation.

Original: Evaluation Metrics.
Revised: Evaluation Metrics.

Original: It is of crucial importance to evaluate the explanations quantitatively since human evaluations are highly dependent on their subjective understanding.
Revised: It is of crucial importance to evaluate the explanations quantitatively since human evaluations are highly dependent on their subjective understanding.

Original: Prior studies have proposed some metrics to quantitatively assess the explanations(Yuan et al., 2020; Dwivedi et al., 2020), among which we select three commonly used metrics to evaluate our results.
Revised: Prior studies have proposed some metrics to quantitatively assess the explanations(Yuan et al., 2020; Dwivedi et al., 2020), among which we select three commonly used metrics to evaluate our results.

Original: For clarity, we denote
Revised: For clarity, we denote

Original: G Ks as the explanatory subgraph by taking top- K edges in G , and |G| as the number of edges in graph G .
Revised: G K s as the explanatory subgraph by taking top- K edges in G , and |G| as the number of edges in graph G .

Original: • Predictive Accuracy (ACC @ η ) (Chen et al., 2018).
Revised: • Predictive Accuracy (ACC @ η ) (Chen et al., This metric feeds the

Revised: explanatory

Original: This metric feeds the explanatory subgraph into the target model and measures the performance of the explanation by auditing how well it recovers the target prediction, where η is the predefined sparsity.
Revised: subgraph into the target model and measures the performance of the explanation by auditing how well it recovers the target prediction, where η is the predefined sparsity.

Original: Formally, given the trained GNN models f , we have
Revised: Formally, given the trained GNN models f , we have

Original: [Equation]
Revised: [Equation]

Revised: [Table]

Original: where I ( · , · ) is the indicator function that takes value 1 when its two arguments are equal and takes value 0 otherwise.
Revised: where I ( · , · ) is the indicator function that takes value 1 when its two arguments are equal and takes value 0 otherwise.

Original: Moreover, we plot the curve of ACC over different sparsity η ∈ (0 , 0 .1 , ...,
Revised: Moreover, we plot the curve of ACC over different sparsity η ∈ (0 , 0 .

Original:  9) on the test set and denote ACC-AUC as the area under the ACC curve.
Revised: 1 , ..., 0 . 9) on the test set and denote ACC-AUC as the area under the ACC curve.

Original: Note that ACC @
Revised: Note that ACC @

Original: η and ACC-AUC do not rely on ground truth labels, and thus are suitable for all the datasets.
Revised: η and ACC-AUC do not rely on ground truth labels, and thus are suitable for all the datasets.

Original: • Precision @
Revised: • Precision @

Original: N (Ying et al., 2019).
Revised: N (Ying et al., 2019).

Original: This metric measures the consistency between the explanatory subgraph G s and the ground-truth subgraph G ∗ .
Revised: This metric measures the consistency between the explanatory subgraph G s and the ground-truth subgraph G ∗ .

Original: Concretely, the edges within G ∗ are positive in G , while the remains are negative.
Revised: Concretely, the edges within G ∗ are positive in G , while the remains are negative.

Original: In this case, precision can be adopted as the evaluation protocol.
Revised: In this case, precision can be adopted as the evaluation protocol

Original: More formally,
Revised: More formally,

Original: [Equation]
Revised: [Equation]

Revised: [Equation]

Original: • Fidelity @ p (Yuan et al., 2020).
Revised: • Fidelity @ p (Yuan et al., 2020).

Original: The Fidelity metric studies the prediction change by removing
Revised: The Fidelity metric studies the prediction change by removing

Original: important input features identified by explanation methods.
Revised: important input features identified by explanation methods.

Original: Formally,
Revised: Formally,

Original: [Equation]
Revised: [Equation]

Original: Baselines.
Revised: Baselines.

Original: We leverage the state-of-the-art methods to verify proposed propositions and optimization schemes, covering the followings:
Revised: We leverage the state-of-the-art methods to verify proposed propositions and optimization schemes, covering the followings:

Original: • GNNExplainer (Ying et al., 2019) directly learns an adjacency matrix mask through maximizing the mutual information between a GNN’s prediction and distribution of possible subgraph structures, which is performed via multiplying the mask to the input features.
Revised: • GNNExplainer (Ying et al., 2019) directly learns an adjacency matrix mask through maximizing the mutual information between a GNN’s prediction and distribution of possible subgraph structures, which is performed via multiplying the mask to the input features.

Original: • PGExplainer (Luo et al., 2020) adopts a deep neural network to parameterize the generation process of explanations, which makes it a natural approach to explaining multiple instances collectively.
Revised: • PGExplainer (Luo et al., 2020) adopts a deep neural network to parameterize the generation process of explanations, which makes it a natural approach to explaining multiple instances collectively.

Original: It can also provide global explanations for a certain class.
Revised: It can also provide global explanations for a certain class.

Original: • GraphMask
Revised: • GraphMask

Original: (Schlichtkrull et al., 2021) learns a simple classifier that, for every edge in every layer, predicts if that edge can be dropped, in a fully differentiable fashion.
Revised: (Schlichtkrull et al., 2021) learns a simple classifier that, for every edge in every layer, predicts if that edge can be dropped, in a fully differentiable fashion.

Original: Then by dropping edges without deteriorating the performance of the model, the remaining edges naturally form an explanation for the model’s prediction.
Revised: Then by dropping edges without deteriorating the performance of the model, the remaining edges naturally form an explanation for the model’s prediction.

Original: • CF-GNNExplainer (Lucic et al., 2022) focuses on the counterfactual explanations by figuring out the minimal perturbation to the input (graph) data such that the prediction changes.
Revised: • CF-GNNExplainer (Lucic et al., 2022) focuses on the counterfactual explanations by figuring out the minimal perturbation to the input (graph) data such that the prediction changes.

Original: By instantiating the perturbation with only edge deletions, they find out the edges which are crucial for the original predictions.
Revised: By instantiating the perturbation with only edge deletions, they find out the edges which are crucial for the original predictions.

Original: • Refine (Wang et al., 2021) develops a explainer that can generate multi-grained explanations by exploiting the pre-training and fine-tuning idea.
Revised: • Refine (Wang et al., 2021) develops a explainer that can generate multi-grained explanations by exploiting the pre-training and fine-tuning idea.

Original: Specifically, the pre-training phase exhibits global explanations with the prototypical patterns, and the fine-tuning phase further adapts the global explanations in the local context with high fidelity.
Revised: Specifically, the pre-training phase exhibits global explanations with the prototypical patterns, and the fine-tuning phase further adapts the global explanations in the local context with high fidelity.

Original: • GSAT (Miao et al., 2022) leverages stochastic attention to block the information from the taskirrelevant graph components while learning stochasticity-reduced attention to select the taskrelevant subgraphs for interpretation.
Revised: • GSAT (Miao et al., 2022) leverages stochastic attention to block the information from the taskirrelevant graph components while learning stochasticity-reduced attention to select the taskrelevant subgraphs for interpretation.

Original: Though it’s naturally an inherently interpretable method, it also works in a post-hoc way through a fine-tuning fashion.
Revised: Though it’s naturally an inherently interpretable method, it also works in a post-hoc way through a fine-tuning fashion.

Original: We adopt this post-hoc working mode as one baseline.
Revised: We adopt this post-hoc working mode as one baseline.

Original: Training Optimization and Early Stopping.
Revised: Training Optimization and Early Stopping.

Original: All experiments are done on a single Tesla V
Revised: All experiments are done on a single Tesla V

Original: SXM2 GPU (32 GB).
Revised: SXM2 GPU (32 GB).

Original: During training, we use Adam (Kingma & Ba, 2015) optimizer.
Revised: During training, we use Adam (Kingma & Ba, 2015) optimizer.

Original: The maximum number of epochs is 200 for all datasets.
Revised: The maximum number of epochs is 200 for all datasets.

Original: We use Stochastic Gradient Descent (SGD) for the optimization of all GNNs models.
Revised: We use Stochastic Gradient Descent (SGD) for the optimization of all GNNs models.

Original: The initial learning rate is set to: 10 − 3 for BA3-motif , 10 −for MNIST, and 10 − 3 for MUTAG.
Revised: The initial learning rate is set to: 10 − 3 for BA3-motif , 10 − for MNIST, and 10 − 3 for MUTAG.

Original: Also, we exhibit early stopping to avoid overfitting the training dataset.
Revised: Also, we exhibit early stopping to avoid overfitting the training dataset.

Original: If the model’s performance on the validation dataset is without improvement ( i.e. , validation accuracy begins to decrease) for five epochs, we stop the training process to prevent increased generalization error.
Revised: If the model’s performance on the validation dataset is without improvement ( i.e. , validation accuracy begins to decrease) for five epochs, we stop the training process to prevent increased generalization error.

Original: Hyperparameter Settings.
Revised: Hyperparameter Settings.

Original: The most crucial hyperparameter in this work is the coefficients of regularization.
Revised: The most crucial hyperparameter in this work is the coefficients of regularization.

Original: For MUTAG: the initial coefficient of sparsity constraints is 0 .
Revised: For MUTAG: the initial coefficient of sparsity constraints is 0 .

Original: and it grows at a rate of 50% each epoch; the coefficient of discrete constraints is 0 .
Revised: and it grows at a rate of 50% each epoch; the coefficient of discrete constraints is 0 .

Original: and it grows at a rate of 50% each epoch;
Revised: and it grows at a rate of 50% each epoch;

Original: For BA3, these coefficients are set to { 0 . 04 , 50% , 0 . 4 , 50% } ; For MNIST: these coefficients are set to { 5 × 10 − 5 , 100% , 5 × 10 − 4 , 50% } .
Revised: For BA3, these coefficients are set to { 0 . 04 , 50% , 0 . 4 , 50% } ; For MNIST: these coefficients are set to { 5 × 10 − 5 , 100% , 5 × 10 − 4 , 50% } .

Original: For other hyperparameters in baseline methods, we adopt a grid search for the optimal parameters using the validation datasets.
Revised: For other hyperparameters in baseline methods, we adopt a grid search for the optimal parameters using the validation datasets.

Original: To be more specific, the learning rate of Adam is tuned in { 10 − 3 , 10 − 2 , 10 − 1 } , and the weight decay is searched in { 10 − 5 , 10 − 4 , 10 − 3 } .
Revised: To be more specific, the learning rate of Adam is tuned in { 10 − 3 , 10 − 2 , 10 − 1 } , and the weight decay is searched in { 10 − 5 , 10 − 4 , 10 − 3 } .

Original: Other model-specific hyperparameters are set as follows: For PGExplainer, the temperature for reparameterization is 0.1;
Revised: Other model-specific hyperparameters are set as follows: For PGExplainer, the temperature for reparameterization is 0.1;

Original: For Refine, the temperature hyperparameter β is 1 and the trade-off hyperparameter γ is 5; For
Revised: For Refine, the temperature hyperparameter β is 1 and the trade-off hyperparameter γ is 5; For

Original: GSAT, the parameter of Bernoulli distribution r is fixed as 0.6.
Revised: GSAT, the parameter of Bernoulli distribution r is fixed as 0.6.

Original: C Q UALITATIVE E VALUATION OF SRS
Revised: C Q UALITATIVE E VALUATION OF SRS

Original: To have visual inspections on the explanatory subgraphs generated by different explainers and the effectiveness of our proposed scheme SRS, we randomly choose graph instances from the synthetic dataset BA3-motif and present them in Figure 6.
Revised: To have visual inspections on the explanatory subgraphs generated by different explainers and the effectiveness of our proposed scheme SRS, we randomly choose graph instances from the synthetic dataset BA3-motif and present them in Figure 6.

Original: For each baseline explainer, we highlight the edges which havethe top- K importance scores by red lines, where K = 6 . The ground-truth nodes are highlighted in green while the turbulence nodes w.r.t. nodes in BA-motif are distinguished in blue.
Revised: For each baseline explainer, we highlight the edges which have

Original: According to Figure 6 we can observe that:
Revised: According to Figure 6 we can observe that:

Original: • Some blue edges which is not belonging to the ground-truth motifs are select by the baseline explanation methods.
Revised: • Some blue edges which is not belonging to the ground-truth motifs are select by the baseline explanation methods.

Original: On the contrary, current explainers enhanced by our proposed scheme SRS inherit the ground-truth edges extracted by the original methods, and eliminate the edges which belong to the Barabasi-Albert (BA) graphs.
Revised: On the contrary, current explainers enhanced by our proposed scheme SRS inherit the ground-truth edges extracted by the original methods, and eliminate the edges which belong to the Barabasi-Albert (BA) graphs.

Original: • Current explainers enhanced by SRS take count of both accuracy and completeness.
Revised: • Current explainers enhanced by SRS take count of both accuracy and completeness.

Original: That is, SRS mainly focuses on the edges belonging to one completed motif while the graph contains more than one ground-truth motifs.
Revised: That is, SRS mainly focuses on the edges belonging to one completed motif while the graph contains more than one ground-truth motifs.

Original: However, baseline explainers are often distracted by multiple groundtruth motifs.
Revised: However, baseline explainers are often distracted by multiple groundtruth motifs.

Original: • For blue nodes in Barabasi-Albert (BA) graphs, there are several more intrusive nodes which are connected to ground-truth motifs.
Revised: • For blue nodes in Barabasi-Albert (BA) graphs, there are several more intrusive nodes which are connected to ground-truth motifs.

Original: These nodes will cause higher interference to the generation of explanatory subgraphs.
Revised: These nodes will cause higher interference to the generation of explanatory subgraphs.

Original: Fortunately, SRS framework can avoid these traps by extracting less turbulence nodes.
Revised: Fortunately, SRS framework can avoid these traps by extracting less turbulence nodes.

Original: This phenomenon demonstrates the robustness of our proposed SRS.
Revised: This phenomenon demonstrates the robustness of our proposed SRS.