Original: T ASK -A GNOSTIC G RAPH E XPLANATIONS
Revised: T ASK -A GNOSTIC G RAPH E XPLANATIONS

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Graph Neural Networks (GNNs) have emerged as powerful tools to encode graph structured data.
Revised: Graph Neural Networks (GNNs) have emerged as powerful tools to encode graph structured data.

Original: Due to their broad applications, there is an increasing need to develop tools to explain how GNNs make decisions given graph structured data.
Revised: Due to their broad applications, there is an increasing need to develop tools to explain how GNNs make decisions given graph structured data.

Original: Existing learning-based GNN explanation approaches are task-speciﬁc in training and hence suffer from crucial drawbacks.
Revised: Existing learning-based GNN explanation approaches are task-speciﬁc in training and hence suffer from crucial drawbacks.

Original: Speciﬁcally, they are incapable of producing explanations for a multitask prediction model with a single explainer.
Revised: Speciﬁcally, they are incapable of producing explanations for a multitask prediction model with a single explainer.

Original: They are also unable to provide explanations in cases where the GNN is trained in a self-supervised manner, and the resulting representations are used in future downstream tasks.
Revised: They are also unable to provide explanations in cases where the GNN is trained in a self-supervised manner, and the resulting representations are used in future downstream tasks.

Original: To address these limitations, we propose a Task-Agnostic GNN
Revised: To address these limitations, we propose a Task-Agnostic GNN

Original: Explainer (T AGE ) trained under self-supervision with no knowledge of downstream tasks.
Revised: Explainer (T AGE ) trained under self-supervision with no knowledge of downstream tasks.

Original: T AGE enables the explanation of GNN embedding models without downstream tasks and allows efﬁcient explanation of multitask models.
Revised: T AGE enables the explanation of GNN embedding models without downstream tasks and allows efﬁcient explanation of multitask models.

Original: Our extensive experiments show that T AGE can signiﬁcantly speed up the explanation efﬁciency by using the same model to explain predictions for multiple downstream tasks while achieving explanation quality as good as or even better than current state-of-the-art GNN explanation approaches.
Revised: Our extensive experiments show that T AGE can signiﬁcantly speed up the explanation efﬁciency by using the same model to explain predictions for multiple downstream tasks while achieving explanation quality as good as or even better than current state-of-the-art GNN explanation approaches.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Graph neural networks (GNNs) (Kipf & Welling, 2017; Veli ˇ ckovi ´ c et al., 2018; Xu et al., 2019) have achieved remarkable success in learning from real-world graph-structured data due to their unique ability to capture both feature-wise and topological information.
Revised: Graph neural networks (GNNs) (Kipf & Welling, 2017; Veli ˇ ckovi ´ c et al., 2018; Xu et al., 2019) have achieved remarkable success in learning from real-world graph-structured data due to their unique ability to capture both feature-wise and topological information.

Original: Extending their success, GNNs are widely applied in various research ﬁelds and industrial applications including quantum chemistry (Gilmer et al., 2017), drug discovery (Wu et al., 2018;
Revised: Extending their success, GNNs are widely applied in various research ﬁelds and industrial applications including quantum chemistry (Gilmer et al., 2017), drug discovery (Wu et al., 2018;

Original: Wang et al., 2020), social networks (Fan et al., 2019), and recommender systems (Ying et al., 2018).
Revised: Wang et al., 2020), social networks (Fan et al., 2019), and recommender systems (Ying et al., 2018).

Original: While multiple approaches have been proposed and studied to improve GNN performance, GNN explainability is an emerging area and has a smaller body of research behind it.
Revised: While multiple approaches have been proposed and studied to improve GNN performance, GNN explainability is an emerging area and has a smaller body of research behind it.

Original: Recently, explainability has gained more attention due to an increasing desire for GNNs with more security, fairness, and reliability.
Revised: Recently, explainability has gained more attention due to an increasing desire for GNNs with more security, fairness, and reliability.

Original: Being able to provide a good explanation to a GNN prediction increases model reliability and reduces the risk of incorrect predictions, which is crucial in ﬁelds such as molecular biology, chemistry, fraud detection, etc.
Revised: Being able to provide a good explanation to a GNN prediction increases model reliability and reduces the risk of incorrect predictions, which is crucial in ﬁelds such as molecular biology, chemistry, fraud detection, etc.

Original: Existing methods adapting the explanation methods for convolutional neural networks (CNNs) or speciﬁcally designed for GNNs have shown promising explanations on multiple types of graph data.
Revised: Existing methods adapting the explanation methods for convolutional neural networks (CNNs) or speciﬁcally designed for GNNs have shown promising explanations on multiple types of graph data.

Original: A recent survey (Yuan et al., 2020) categorizes existing explanation methods into gradient-based, perturbation, decomposition, and surrogate methods.
Revised: A recent survey (Yuan et al., 2020) categorizes existing explanation methods into gradient-based, perturbation, decomposition, and surrogate methods.

Original: In particular, perturbation methods involve learning or optimization (Ying et al., 2019; Luo et al., 2020; Yuan et al., 2021; Lin et al., 2021) and, while bearing higher computational costs, generally achieve state-of-the-art performance in terms of explanation quality.
Revised: In particular, perturbation methods involve learning or optimization (Ying et al., 2019; Luo et al., 2020; Yuan et al., 2021; Lin et al., 2021) and, while bearing higher computational costs, generally achieve state-of-the-art performance in terms of explanation quality.

Original: These methods train post-hoc explanation models on top of the prediction model to be explained.
Revised: These methods train post-hoc explanation models on top of the prediction model to be explained.

Original: Earlier approaches like GNNExplainer (Ying et al., 2019) require training or optimizing an individual explainer for each data instance, i.e., a graph or a node to be explained.
Revised: Earlier approaches like GNNExplainer (Ying et al., 2019) require training or optimizing an individual explainer for each data instance, i.e., a graph or a node to be explained.

Original: In contrast, PGExplainer (Luo et al., 2020) performs inductive learning, i.e., it only requires a one-time training, and the explainer can be generalized to explain all data instances without individual optimization.
Revised: In contrast, PGExplainer (Luo et al., 2020) performs inductive learning, i.e., it only requires a one-time training, and the explainer can be generalized to explain all data instances without individual optimization.

Original: Compared to other optimization-based explanation methods, PGExplainer signiﬁcantly improves the efﬁciency in terms of time cost without performance loss by learning.
Revised: Compared to other optimization-based explanation methods, PGExplainer signiﬁcantly improves the efﬁciency in terms of time cost without performance loss by learning.

Original: However, even state-of-the-art explanation methods like PGExplainer are still task-speciﬁc at training and hence suffer from two crucial drawbacks.
Revised: However, even state-of-the-art explanation methods like PGExplainer are still task-speciﬁc at training and hence suffer from two crucial drawbacks.

Original: First, current methods are inefﬁcient in explaining multitask prediction for graph-structured data.
Revised: First, current methods are inefﬁcient in explaining multitask prediction for graph-structured data.

Original: For example, one may need to predict multiple chemical properties in drug discovery for a molecular graph.
Revised: For example, one may need to predict multiple chemical properties in drug discovery for a molecular graph.

Original: In particular, ToxCast from MoleculeNet has
Revised: In particular, ToxCast from MoleculeNet has

Original: [Figure]
Revised: [Figure]

Original: 167 prediction tasks.
Revised: 167 prediction tasks.

Original: In these cases, it is common to apply a single GNN model with multiple output dimensions to make predictions for all tasks.
Revised: In these cases, it is common to apply a single GNN model with multiple output dimensions to make predictions for all tasks.

Original: However, one is unable to employ a single explainer to explain the above model, since current explainers are trained speciﬁcally to explain one prediction task.
Revised: However, one is unable to employ a single explainer to explain the above model, since current explainers are trained speciﬁcally to explain one prediction task.

Original: As a result, in the case of ToxCast, one must train 167 explainers to explain the GNN model.
Revised: As a result, in the case of ToxCast, one must train 167 explainers to explain the GNN model.

Original: Second, in industry settings, it is common to train GNN models in a two-stage fashion due to scaling, latency, and label sparsity issues.
Revised: Second, in industry settings, it is common to train GNN models in a two-stage fashion due to scaling, latency, and label sparsity issues.

Original: The ﬁrst stage trains a GNN-based embedding model with a massive amount of unlabeled data in an unsupervised manner to learn embeddings for nodes or graphs.
Revised: The ﬁrst stage trains a GNN-based embedding model with a massive amount of unlabeled data in an unsupervised manner to learn embeddings for nodes or graphs.

Original: The second stage trains lightweight models such as multilayer perceptrons (MLPs) using the frozen embeddings as input to predict the downstream tasks.
Revised: The second stage trains lightweight models such as multilayer perceptrons (MLPs) using the frozen embeddings as input to predict the downstream tasks.

Original: In the ﬁrst stage, the downstream tasks are usually unknown or undeﬁned, and existing task-speciﬁc explainers cannot be applied.
Revised: In the ﬁrst stage, the downstream tasks are usually unknown or undeﬁned, and existing task-speciﬁc explainers cannot be applied.

Original: Also, there can be tens to hundreds of downstream tasks trained on these GNN embeddings, and training a separate explainer for each task is undesirable and downright impossible.
Revised: Also, there can be tens to hundreds of downstream tasks trained on these GNN embeddings, and training a separate explainer for each task is undesirable and downright impossible.

Original: To address the above limitations, we present a new task-agnostic explanation pipeline, shown in
Revised: To address the above limitations, we present a new task-agnostic explanation pipeline, shown in

Original: Figure 1, where we decompose a prediction model into a GNN embedding model and a downstream model, designing separate explainers for each component.
Revised: Figure 1, where we decompose a prediction model into a GNN embedding model and a downstream model, designing separate explainers for each component.

Original: We design the downstream explainers to cooperate with the embedding explainer.
Revised: We design the downstream explainers to cooperate with the embedding explainer.

Original: The embedding explainer is trained using a self-supervised training framework, which we dub Task-Agnostic GNN Explainer (T AGE ), with no knowledge of downstream tasks, models, or labels.
Revised: The embedding explainer is trained using a self-supervised training framework, which we dub Task-Agnostic GNN Explainer (T AGE ), with no knowledge of downstream tasks, models, or labels.

Original: In contrast to existing explainers, the learning objective for
Revised: In contrast to existing explainers, the learning objective for

Original: T AGE is computed at the graph or node embeddings without involving task-related predictions.
Revised: T AGE is computed at the graph or node embeddings without involving task-related predictions.

Original: In addition to eliminating the need for downstream tasks in T AGE , we argue that the self-supervision performed on the embeddings can bring additional performance boost in terms of the explanation quality compared to existing task-speciﬁc baselines such as GNNExplainer and PGExplainer.
Revised: In addition to eliminating the need for downstream tasks in T AGE , we argue that the self-supervision performed on the embeddings can bring additional performance boost in terms of the explanation quality compared to existing task-speciﬁc baselines such as GNNExplainer and PGExplainer.

Original: We summarize our contributions as follows: 1) We introduce the task-agnostic explanation problem and propose a two-stage explanation pipeline involving an embedding explainer and a downstream explainer.
Revised: We summarize our contributions as follows: 1) We introduce the task-agnostic explanation problem and propose a two-stage explanation pipeline involving an embedding explainer and a downstream explainer.

Original: This enables the explanation of multiple downstream tasks with a single embedding explainer.
Revised: This enables the explanation of multiple downstream tasks with a single embedding explainer.

Original: We propose a self-supervised training framework T AGE , which is based on conditioned contrastive learning to train the embedding explainer.
Revised: We propose a self-supervised training framework T AGE , which is based on conditioned contrastive learning to train the embedding explainer.

Original: T AGE requires no knowledge of downstream tasks.
Revised: T AGE requires no knowledge of downstream tasks.

Original: We perform experiments on real-world datasets and observe that T AGE outperforms existing learning-based explanation baselines in terms of explanation quality, universal explanation ability, and the time required for training and inference.
Revised: We perform experiments on real-world datasets and observe that T AGE outperforms existing learning-based explanation baselines in terms of explanation quality, universal explanation ability, and the time required for training and inference.

Original:  T ASK -A GNOSTIC E XPLANATIONS 2. N OTATIONS AND L EARNING - BASED GNN EXPLANATION
Revised:  T ASK -A GNOSTIC E XPLANATIONS 2. N OTATIONS AND L EARNING - BASED GNN EXPLANATION

Original: Our study considers the attributed graph G with node set V and edge set E .
Revised: Our study considers the attributed graph G with node set V and edge set E .

Original: We formulate the attributed graph as a tuple of matrices ( A , X ) , where A ∈ { 0 , 1 } | V |×| V | denotes the adjacency matrix and X ∈ R | V |× d f denotes the feature matrix with feature dimension of d f .
Revised: We formulate the attributed graph as a tuple of matrices ( A , X ) , where A ∈ { 0 , 1 } | V |×| V | denotes the adjacency matrix and X ∈ R | V |× d f denotes the feature matrix with feature dimension of d f .

Original: We assume that the prediction model F that is to be explained operates on graph-structured data through two components: a GNN-based embedding model and lighter downstream models.
Revised: We assume that the prediction model F that is to be explained operates on graph-structured data through two components: a GNN-based embedding model and lighter downstream models.

Original: Denoting the input space by G , a node-level embedding model E n : G → R | V |× d takes a graph as input and computes
Revised: Denoting the input space by G , a node-level embedding model E n : G → R | V |× d takes a graph as input and computes

Original: [Table]
Revised: [Table]

Original: embeddings of dimension d for all nodes in the graph, whereas a graph-level embedding model
Revised: embeddings of dimension d for all nodes in the graph, whereas a graph-level embedding model

Original: E g : G → R 1 × d computes an embedding for the input graph.
Revised: E g : G → R 1 × d computes an embedding for the input graph.

Original: Subsequently, the downstream model
Revised: Subsequently, the downstream model

Original: D : R d → R computes predictions for the downstream task based on the embeddings.
Revised: D : R d → R computes predictions for the downstream task based on the embeddings.

Original: Typical GNN explainers consider a task-speciﬁc GNN-based model as a complete unit, i.e. , F :=
Revised: Typical GNN explainers consider a task-speciﬁc GNN-based model as a complete unit, i.e. , F :=

Original: D ◦ E .
Revised: D ◦ E .

Original: Given a graph G and the GNN-based model F to be explained , our goal is to identify the subgraph G sub that contributes the most to the ﬁnal prediction made by F .
Revised: Given a graph G and the GNN-based model F to be explained , our goal is to identify the subgraph G sub that contributes the most to the ﬁnal prediction made by F .

Original: In other words, we claim that a given prediction is made because F captures crucial information provided by some subgraph G sub .
Revised: In other words, we claim that a given prediction is made because F captures crucial information provided by some subgraph G sub .

Original: The learning-based (or optimization-based) GNN explanation employs a parametric explainer T θ associated with the GNN model F to compute the subgraph G sub of the given graph data.
Revised: The learning-based (or optimization-based) GNN explanation employs a parametric explainer T θ associated with the GNN model F to compute the subgraph G sub of the given graph data.

Original: Concretely, the explainer T θ computes the importance score for each node or edge, denoted as w i or w ij , or masks for node attributes denoted as m .
Revised: Concretely, the explainer T θ computes the importance score for each node or edge, denoted as w i or w ij , or masks for node attributes denoted as m .

Original: It then selects the subgraph G sub induced by important nodes and edges, i.e. , whose scores exceed a threshold t , and by masking the unimportant attributes.
Revised: It then selects the subgraph G sub induced by important nodes and edges, i.e. , whose scores exceed a threshold t , and by masking the unimportant attributes.

Original: In our study, we follow Luo et al.
Revised: In our study, we follow Luo et al.

Original: (2020), focusing on the importance of edges to provide explanations to GNNs.
Revised: (2020), focusing on the importance of edges to provide explanations to GNNs.

Original: Formally, we have G sub :=
Revised: Formally, we have G sub :=

Original: ( V, E sub ) =
Revised: ( V, E sub ) =

Original: T θ ( G ) , where E sub
Revised: T θ ( G ) , where E sub

Original: = { ( v i , v j ) : ( v i , v j ) ∈ E, w ij ≥ t } .
Revised: = { ( v i , v j ) : ( v i , v j ) ∈ E, w ij ≥ t } .

Original: 2. T ASK -A GNOSTIC E XPLANATIONS
Revised: 2. T ASK -A GNOSTIC E XPLANATIONS

Original: As introduced in Section 1, all existing learning-based or optimization-based explanation approaches are task-speciﬁc and hence suffer from infeasiblity or inefﬁciency in many real-application scenarios.
Revised: As introduced in Section 1, all existing learning-based or optimization-based explanation approaches are task-speciﬁc and hence suffer from infeasiblity or inefﬁciency in many real-application scenarios.

Original: In particular, they are of limited use when downstream tasks are unknown or undeﬁned and fail to employ a single explainer to explain a multitask prediction model.
Revised: In particular, they are of limited use when downstream tasks are unknown or undeﬁned and fail to employ a single explainer to explain a multitask prediction model.

Original: To enable the explanation of GNNs in two-stage training and multitask scenarios, we introduce a new explanation paradigm called the task-agnostic explanation.
Revised: To enable the explanation of GNNs in two-stage training and multitask scenarios, we introduce a new explanation paradigm called the task-agnostic explanation.

Original: The task-agnostic explanation considers a whole prediction model as an embedding model followed by any number of downstream models.
Revised: The task-agnostic explanation considers a whole prediction model as an embedding model followed by any number of downstream models.

Original: It focuses on explaining the embedding model regardless of the number or the existence of downstream models.
Revised: It focuses on explaining the embedding model regardless of the number or the existence of downstream models.

Original: In particular, the task-agnostic explanation trains only one explainer T ( tag ) θ to explain the embedding model E , which should satisfy the following features.
Revised: In particular, the task-agnostic explanation trains only one explainer T ( tag ) θ to explain the embedding model E , which should satisfy the following features.

Original: First, given an input graph G , the explainer T ( tag ) θ should be able to provide different explanations according to speciﬁc downstream tasks being studied.
Revised: First, given an input graph G , the explainer T ( tag ) θ should be able to provide different explanations according to speciﬁc downstream tasks being studied.

Original: Table 1 compares the properties of common GNN explanation methods and the desired task-agnostic explainers in multitask scenarios.
Revised: Table 1 compares the properties of common GNN explanation methods and the desired task-agnostic explainers in multitask scenarios.

Original: Second, the explainer T ( tag ) θ can be trained when only the embedding model is available, e.g. , at the ﬁrst stage of a two-stage training paradigm, regardless of the presence of downstream tasks.
Revised: Second, the explainer T ( tag ) θ can be trained when only the embedding model is available, e.g. , at the ﬁrst stage of a two-stage training paradigm, regardless of the presence of downstream tasks.

Original: When downstream tasks and models are unknown,
Revised: When downstream tasks and models are unknown,

Original: T ( tag ) θ can still identify which components of the input graph are important for certain embedding dimensions of interest.
Revised: T ( tag ) θ can still identify which components of the input graph are important for certain embedding dimensions of interest.

Original:  T HE T AGE F RAMEWORK
Revised:  T HE T AGE F RAMEWORK

Original: Our explanation framework T AGE follows the typical scheme of GNN explanation introduced in the previous section.
Revised: Our explanation framework T AGE follows the typical scheme of GNN explanation introduced in the previous section.

Original: It provides explanations by identifying important edges in a given graph and removing the edges that lead to signiﬁcant changes in the ﬁnal prediction.
Revised: It provides explanations by identifying important edges in a given graph and removing the edges that lead to signiﬁcant changes in the ﬁnal prediction.

Original: Speciﬁcally, the goal of the T AGE is to predict the importance score for each edge in a given graph.
Revised: Speciﬁcally, the goal of the T AGE is to predict the importance score for each edge in a given graph.

Original: Different from existing methods, the proposed T AGE breaks down typical end-to-end GNN explainers into two components.
Revised: Different from existing methods, the proposed T AGE breaks down typical end-to-end GNN explainers into two components.

Original: We now provide general descriptions and detailed formulations to the proposed framework.
Revised: We now provide general descriptions and detailed formulations to the proposed framework.

Original: 3. T ASK -A GNOSTIC E XPLANATION P IPELINE
Revised: 3. T ASK -A GNOSTIC E XPLANATION P IPELINE

Original: Following the principle of the desired task-agnostic explanations, we introduce the task-agnostic explanation pipeline, where a typical explanation procedure is performed in two steps.
Revised: Following the principle of the desired task-agnostic explanations, we introduce the task-agnostic explanation pipeline, where a typical explanation procedure is performed in two steps.

Original: In particular, we decompose the typical end-to-end learning-based GNN explainer into two parts: the embedding explainer T E and the downstream explainer T down , corresponding to the two components in the two-stage training and prediction procedure.
Revised: In particular, we decompose the typical end-to-end learning-based GNN explainer into two parts: the embedding explainer T E and the downstream explainer T down , corresponding to the two components in the two-stage training and prediction procedure.

Original: We compare the typical explanation pipeline and the two-stage explanation pipeline in Figure 1.
Revised: We compare the typical explanation pipeline and the two-stage explanation pipeline in Figure 1.

Original: The embedding explainer and downstream explainers can be trained or constructed independently from each other.
Revised: The embedding explainer and downstream explainers can be trained or constructed independently from each other.

Original: In addition, the embedding explainer can cooperate with any downstream explainers to perform end-to-end explanations on input graphs.
Revised: In addition, the embedding explainer can cooperate with any downstream explainers to perform end-to-end explanations on input graphs.

Original: The downstream explainer aims to explain task-speciﬁc downstream models.
Revised: The downstream explainer aims to explain task-speciﬁc downstream models.

Original: As downstream models are usually lightweight MLPs, we simply adopt gradient-based explainers for downstream explainers without training.
Revised: As downstream models are usually lightweight MLPs, we simply adopt gradient-based explainers for downstream explainers without training.

Original: The downstream explainer takes a downstream model and the graph or node embedding vector as inputs and computes the importance score of each dimension on the embedding vector.
Revised: The downstream explainer takes a downstream model and the graph or node embedding vector as inputs and computes the importance score of each dimension on the embedding vector.

Original: The importance scores then serve as a condition vector input to the embedding explainer.
Revised: The importance scores then serve as a condition vector input to the embedding explainer.

Original: Given the condition vector, the embedding explainer explains the GNN-based embedding model by identifying an important subgraph from the input graph data.
Revised: Given the condition vector, the embedding explainer explains the GNN-based embedding model by identifying an important subgraph from the input graph data.

Original: In other words, given different condition vectors associated with different downstream tasks or models, the embedding explainer can provide corresponding explanations for the same embedding model.
Revised: In other words, given different condition vectors associated with different downstream tasks or models, the embedding explainer can provide corresponding explanations for the same embedding model.

Original: Formally, we denote the downstream explainer for models from D by T down : D × R d → R d , which maps input models and embeddings into importance scores m for all embedding dimensions.
Revised: Formally, we denote the downstream explainer for models from D by T down : D × R d → R d , which maps input models and embeddings into importance scores m for all embedding dimensions.

Original: We denote the embedding explainer associated with the embedding model E by T E : R d × G → G , which maps a given graph into a subgraph of higher importance, conditioned on the embedding dimension importance m ∈ R d .
Revised: We denote the embedding explainer associated with the embedding model E by T E : R d × G → G , which maps a given graph into a subgraph of higher importance, conditioned on the embedding dimension importance m ∈ R d .

Original: The training procedures of the embedding explainer are independent of downstream tasks or downstream explainers.
Revised: The training procedures of the embedding explainer are independent of downstream tasks or downstream explainers.

Original: In particular, the downstream explainer is obtained from the downstream model only, and the training of the embedding explainer only requires the embedding model and the input graphs.
Revised: In particular, the downstream explainer is obtained from the downstream model only, and the training of the embedding explainer only requires the embedding model and the input graphs.

Original: As downstream models are usually constructed as stacked fully connected (FC) layers and the explanation of FC layers has been well studied, our study mainly focuses on the non-trivial training procedure and design of the embedding explainer.
Revised: As downstream models are usually constructed as stacked fully connected (FC) layers and the explanation of FC layers has been well studied, our study mainly focuses on the non-trivial training procedure and design of the embedding explainer.

Original: 3. T RAINING EMBEDDING EXPLAINER UNDER S ELF - SUPERVISION
Revised: 3. T RAINING EMBEDDING EXPLAINER UNDER S ELF - SUPERVISION

Original: A straightforward idea of explaining an embedding model with no knowledge of downstream tasks is to employ existing explainers and perform explanation on the pretext task, such as graph reconstruction (Kipf & Welling, 2016) or context prediction (Hu et al., 2020), used during the pre-training of GNNs.
Revised: A straightforward idea of explaining an embedding model with no knowledge of downstream tasks is to employ existing explainers and perform explanation on the pretext task, such as graph reconstruction (Kipf & Welling, 2016) or context prediction (Hu et al., 2020), used during the pre-training of GNNs.

Original: However, such explanations cannot generalize to future downstream tasks as there are limited dependencies between the pretext task and downstream tasks.
Revised: However, such explanations cannot generalize to future downstream tasks as there are limited dependencies between the pretext task and downstream tasks.

Original: Therefore, training an embedding explainer without downstream models or labels is challenging, and it is desirable to develop a generalizable training approach for the embedding explainer.
Revised: Therefore, training an embedding explainer without downstream models or labels is challenging, and it is desirable to develop a generalizable training approach for the embedding explainer.

Original: To this end, we propose a self-supervised learning framework for the embedding explainer.
Revised: To this end, we propose a self-supervised learning framework for the embedding explainer.

Original: The learning objective of the proposed framework seeks to maximize the mutual information (MI) between two embeddings with certain dimensions masked, i.e., one of the given graphs and one of the corresponding subgraph of high importance induced by the explainer.
Revised: The learning objective of the proposed framework seeks to maximize the mutual information (MI) between two embeddings with certain dimensions masked, i.e., one of the given graphs and one of the corresponding subgraph of high importance induced by the explainer.

Original: We introduce a masking vector
Revised: We introduce a masking vector

Original: p ∈
Revised: p ∈

Original: R
Revised: R

Original: d to indicate speciﬁc dimensions of embeddings on which to maximize MI.
Revised: d to indicate speciﬁc dimensions of embeddings on which to maximize MI.

Original: During explanation, we obtain the masking vector from the importance vector computed by any downstream explainer T down .
Revised: During explanation, we obtain the masking vector from the importance vector computed by any downstream explainer T down .

Original: As no downstream importance vector is available at training, we sample the masking vector p from a multivariate Laplace distribution due to the sparse gradient assumption, i.e. , only a few dimensions are of high importance.
Revised: As no downstream importance vector is available at training, we sample the masking vector p from a multivariate Laplace distribution due to the sparse gradient assumption, i.e. , only a few dimensions are of high importance.

Original: Formally, the MI-based learning objective is
Revised: Formally, the MI-based learning objective is

Original: [Equation]
Revised: [Equation]

Original: (1) where MI ( · , · ) computes the mutual information between two random vectors, p denotes the random masking vector sampled from a certain distribution, T θ ( p , G ) computes the subgraph of high importance, and ⊗ denotes the element-wise multiplication, which applies masking to the embeddings
Revised: (1) where MI ( · , · ) computes the mutual information between two random vectors, p denotes the random masking vector sampled from a certain distribution, T θ ( p , G ) computes the subgraph of high importance, and ⊗ denotes the element-wise multiplication, which applies masking to the embeddings

Original: E ( · ) .
Revised: E ( · ) .

Original: Figure 2 outlines the training framework and objective.
Revised: Figure 2 outlines the training framework and objective.

Original: Intuitively, given an input graph and the desired embedding dimensions to be explained, the explainer T θ predicts the subgraph whose
Revised: Intuitively, given an input graph and the desired embedding dimensions to be explained, the explainer T θ predicts the subgraph whose

Original: [Table]
Revised: [Table]

Original: /agXcxOEmYVndpmkBPhIAyaSaa8mGWCJCeiI8zoEe3HlZWieV2zNjxfF6vUsjhw6RieohGx0iaroAdVQAxH0jMboHX0YL8ar8Wl8TZ+uGLOZIzRXxvcPnaCrJQ==</latexit> <latexit sha1_base64="agaFn6C551lw2UK48HW/AQbYOnA=">AAACL3icbZDLSsNAFIYnXmu9VV26CRahBSmJCLosipdlhd6gCWEynbZDJ5kwcyKUkDdy46t0I6KIW9/CSVvEth4Y5uM/c5j//H7EmQLLejNWVtfWNzZzW/ntnd29/cLBYVOJWBLaIIIL2faxopyFtAEMOG1HkuLA57TlD2+yfuuJSsVEWIdRRN0A90PWYwSDlrzCneML3lWjQF9JlDoCWECVE2AYEMyT27T0y/XUc2BAAZfmZ87uy2WvULQq1qTMZbBnUESzqnmFsdMVJA5oCIRjpTq2FYGbYAmMcJrmnVjRCJMh7tOOxhBrV24y2Tc1T7XSNXtC6hOCOVH/TiQ4UJk9/TIzrxZ7mfhfrxND78pNWBjFQEMy/agXcxOEmYVndpmkBPhIAyaSaa8mGWCJCeiI8zoEe3HlZWieV2zNjxfF6vUsjhw6RieohGx0iaroAdVQAxH0jMboHX0YL8ar8Wl8TZ+uGLOZIzRXxvcPnaCrJQ==</latexit> <latexit sha1_base64="agaFn6C551lw2UK48HW/AQbYOnA=">AAACL3icbZDLSsNAFIYnXmu9VV26CRahBSmJCLosipdlhd6gCWEynbZDJ5kwcyKUkDdy46t0I6KIW9/CSVvEth4Y5uM/c5j//H7EmQLLejNWVtfWNzZzW/ntnd29/cLBYVOJWBLaIIIL2faxopyFtAEMOG1HkuLA57TlD2+yfuuJSsVEWIdRRN0A90PWYwSDlrzCneML3lWjQF9JlDoCWECVE2AYEMyT27T0y/XUc2BAAZfmZ87uy2WvULQq1qTMZbBnUESzqnmFsdMVJA5oCIRjpTq2FYGbYAmMcJrmnVjRCJMh7tOOxhBrV24y2Tc1T7XSNXtC6hOCOVH/TiQ4UJk9/TIzrxZ7mfhfrxND78pNWBjFQEMy/agXcxOEmYVndpmkBPhIAyaSaa8mGWCJCeiI8zoEe3HlZWieV2zNjxfF6vUsjhw6RieohGx0iaroAdVQAxH0jMboHX0YL8ar8Wl8TZ+uGLOZIzRXxvcPnaCrJQ==</latexit>
Revised: /agXcxOEmYVndpmkBPhIAyaSaa8mGWCJCeiI8zoEe3HlZWieV2zNjxfF6vUsjhw6RieohGx0iaroAdVQAxH0jMboHX0YL8ar8Wl8TZ+uGLOZIzRXxvcPnaCrJQ==</latexit> <latexit sha1_base64="agaFn6C551lw2UK48HW/AQbYOnA=">AAACL3icbZDLSsNAFIYnXmu9VV26CRahBSmJCLosipdlhd6gCWEynbZDJ5kwcyKUkDdy46t0I6KIW9/CSVvEth4Y5uM/c5j//H7EmQLLejNWVtfWNzZzW/ntnd29/cLBYVOJWBLaIIIL2faxopyFtAEMOG1HkuLA57TlD2+yfuuJSsVEWIdRRN0A90PWYwSDlrzCneML3lWjQF9JlDoCWECVE2AYEMyT27T0y/XUc2BAAZfmZ87uy2WvULQq1qTMZbBnUESzqnmFsdMVJA5oCIRjpTq2FYGbYAmMcJrmnVjRCJMh7tOOxhBrV24y2Tc1T7XSNXtC6hOCOVH/TiQ4UJk9/TIzrxZ7mfhfrxND78pNWBjFQEMy/agXcxOEmYVndpmkBPhIAyaSaa8mGWCJCeiI8zoEe3HlZWieV2zNjxfF6vUsjhw6RieohGx0iaroAdVQAxH0jMboHX0YL8ar8Wl8TZ+uGLOZIzRXxvcPnaCrJQ==</latexit> <latexit sha1_base64="agaFn6C551lw2UK48HW/AQbYOnA=">AAACL3icbZDLSsNAFIYnXmu9VV26CRahBSmJCLosipdlhd6gCWEynbZDJ5kwcyKUkDdy46t0I6KIW9/CSVvEth4Y5uM/c5j//H7EmQLLejNWVtfWNzZzW/ntnd29/cLBYVOJWBLaIIIL2faxopyFtAEMOG1HkuLA57TlD2+yfuuJSsVEWIdRRN0A90PWYwSDlrzCneML3lWjQF9JlDoCWECVE2AYEMyT27T0y/XUc2BAAZfmZ87uy2WvULQq1qTMZbBnUESzqnmFsdMVJA5oCIRjpTq2FYGbYAmMcJrmnVjRCJMh7tOOxhBrV24y2Tc1T7XSNXtC6hOCOVH/TiQ4UJk9/TIzrxZ7mfhfrxND78pNWBjFQEMy/agXcxOEmYVndpmkBPhIAyaSaa8mGWCJCeiI8zoEe3HlZWieV2zNjxfF6vUsjhw6RieohGx0iaroAdVQAxH0jMboHX0YL8ar8Wl8TZ+uGLOZIzRXxvcPnaCrJQ==</latexit>

Original: Practically, the mutual information is intractable and is hence hard to directly compute.
Revised: Practically, the mutual information is intractable and is hence hard to directly compute.

Original: A common approach to achieve efﬁcient computation and optimization is to adopt the upper bound estimations of mutual information, namely, the Jenson-Shannon Estimator (JSE)
Revised: A common approach to achieve efﬁcient computation and optimization is to adopt the upper bound estimations of mutual information, namely, the Jenson-Shannon Estimator (JSE)

Original: (Nowozin et al., 2016) and the InfoNCE (Gutmann & Hyv ¨ arinen, 2010).
Revised: (Nowozin et al., 2016) and the InfoNCE (Gutmann & Hyv ¨ arinen, 2010).

Original: These upper bound estimations are also referred to as contrastive loss and are widely applied in self-supervised representation learning (Hjelm et al., 2019;
Revised: These upper bound estimations are also referred to as contrastive loss and are widely applied in self-supervised representation learning (Hjelm et al., 2019;

Original: Sun et al., 2019;
Revised: Sun et al., 2019;

Original: Veli ˇ ckovi ´ c et al., 2019) for both images and graphs.
Revised: Veli ˇ ckovi ´ c et al., 2019) for both images and graphs.

Original: Adopting these estimators, the objectives are efﬁciently computed as
Revised: Adopting these estimators, the objectives are efﬁciently computed as

Original: [Equation]
Revised: [Equation]

Original: for JSE and InfoNCE, respectively, where N denotes the number of samples in a mini-batch, σ denotes Sigmoid function, z i and z i,θ are embeddings of the original graph G i and its subgraph
Revised: for JSE and InfoNCE, respectively, where N denotes the number of samples in a mini-batch, σ denotes Sigmoid function, z i and z i,θ are embeddings of the original graph G i and its subgraph

Original: T θ ( G i ) , or target nodes of the two graphs.
Revised: T θ ( G i ) , or target nodes of the two graphs.

Original: Our objective involves condition vectors as masks on the embeddings, which differs from typical contrastive loss used in self-supervised representation learning.
Revised: Our objective involves condition vectors as masks on the embeddings, which differs from typical contrastive loss used in self-supervised representation learning.

Original: We hence call the proposed objective the conditioned contrastive loss.
Revised: We hence call the proposed objective the conditioned contrastive loss.

Original: To restrict the size of subgraphs given by the explainer, we additionally add a size regularization term R , computed as the averaged importance score, to the above objectives.
Revised: To restrict the size of subgraphs given by the explainer, we additionally add a size regularization term R , computed as the averaged importance score, to the above objectives.

Original: In the case where edge importance scores w ij ∈ [0 , 1] are computed, the regularization term is computed as
Revised: In the case where edge importance scores w ij ∈ [0 , 1] are computed, the regularization term is computed as

Original: [Equation]
Revised: [Equation]

Original: where λ s and λ e are hyper-parameters controlling the size and the entropy of edge importance scores, respectively.
Revised: where λ s and λ e are hyper-parameters controlling the size and the entropy of edge importance scores, respectively.

Original: 3. E XPLAINER A RCHITECTURES
Revised: 3. E XPLAINER A RCHITECTURES

Original: Embedding explainers .
Revised: Embedding explainers .

Original: Inspired by explainer architectures used by PGExplainer, we adopt the multilayer perceptron (MLP) to predict the importance score w ij for each edge ( u i , u j ) ∈ E , on top of learned embeddings z i and z j of the two nodes connected by the edge.
Revised: Inspired by explainer architectures used by PGExplainer, we adopt the multilayer perceptron (MLP) to predict the importance score w ij for each edge ( u i , u j ) ∈ E , on top of learned embeddings z i and z j of the two nodes connected by the edge.

Original: Edges with scores higher than a threshold are considered as important edges that remain in the selected subgraph.
Revised: Edges with scores higher than a threshold are considered as important edges that remain in the selected subgraph.

Original: In order for the embedding explainer to cooperate with different downstream explainers and provide diverse explanations for different tasks, it additionally requires a condition vector as input indicating the speciﬁc downstream task to be explained.
Revised: In order for the embedding explainer to cooperate with different downstream explainers and provide diverse explanations for different tasks, it additionally requires a condition vector as input indicating the speciﬁc downstream task to be explained.

Original: We handle the condition vector in a similar manner to
Revised: We handle the condition vector in a similar manner to

Original: Conditional GAN (Mirza & Osindero, 2014).
Revised: Conditional GAN (Mirza & Osindero, 2014).

Original: Formally, the graph-level embedding explainer takes the embeddings, z i and z j , and the condition vector p as inputs and computes the importance score by
Revised: Formally, the graph-level embedding explainer takes the embeddings, z i and z j , and the condition vector p as inputs and computes the importance score by

Original: [Equation]
Revised: [Equation]

Original: where [ · ; · ] denotes the concatenation along the feature dimension, ⊗ denotes the element-wise mul- tiplication, σ denotes the activation function, and
Revised: where [ · ; · ] denotes the concatenation along the feature dimension, ⊗ denotes the element-wise mul- tiplication, σ denotes the activation function, and

Original: f g : R d → R 2 d is a linear projection.
Revised: f g : R d → R 2 d is a linear projection.

Original: The node-level embedding explainer takes an additional node embedding as its input, as the explainers are expected to predict different scores for the same edge when explaining different target nodes.
Revised: The node-level embedding explainer takes an additional node embedding as its input, as the explainers are expected to predict different scores for the same edge when explaining different target nodes.

Original: The formulation of computing the importance score is as follows,
Revised: The formulation of computing the importance score is as follows,

Original: [Equation]
Revised: [Equation]

Original: ) where f g : R d → R 3 d is a linear projection, and z target denotes the embedding of the target node whose prediction is to be explained.
Revised: ) where f g : R d → R 3 d is a linear projection, and z target denotes the embedding of the target node whose prediction is to be explained.

Original: Downstream explainers .
Revised: Downstream explainers .

Original: We adopt the gradient-based explainer to explain the downstream models.
Revised: We adopt the gradient-based explainer to explain the downstream models.

Original: Formally, given an input embedding z and its prediction probabilities D ( z ) ∈ [0 , 1] C among all C classes, we compute the gradient of the maximal probability w.r.t.
Revised: Formally, given an input embedding z and its prediction probabilities D ( z ) ∈ [0 , 1] C among all C classes, we compute the gradient of the maximal probability w.r.t.

Original: the input embedding:
Revised: the input embedding:

Original: [Equation]
Revised: [Equation]

Original: where D ( z )[ c ] denotes the probability for class c .
Revised: where D ( z )[ c ] denotes the probability for class c .

Original: To convert the gradient into the condition vector, we further perform normalization and only take positive values reﬂecting only positive inﬂuence to the predicted class probability
Revised: To convert the gradient into the condition vector, we further perform normalization and only take positive values reﬂecting only positive inﬂuence to the predicted class probability

Original: , i.e. , p = ReLU ( norm ( g T )) .
Revised: , i.e. , p = ReLU ( norm ( g T )) .

Original:  E XPERIMENTAL S TUDIES
Revised:  E XPERIMENTAL S TUDIES

Original: We conduct two groups of quantitative studies evaluating the explanation quality and the universal explanation ability, i.e. , training a single explainer to explain all downstream tasks, of T AGE .
Revised: We conduct two groups of quantitative studies evaluating the explanation quality and the universal explanation ability, i.e. , training a single explainer to explain all downstream tasks, of T AGE .

Original: We then compare the efﬁciency of multiple learning-based GNN explainers in terms of training and explanation time cost.
Revised: We then compare the efﬁciency of multiple learning-based GNN explainers in terms of training and explanation time cost.

Original: We further provide visualizations to demonstrate the explanation quality as well as the ability to explain GNN models without downstream tasks.
Revised: We further provide visualizations to demonstrate the explanation quality as well as the ability to explain GNN models without downstream tasks.

Original: 4. D ATASETS
Revised: 4. D ATASETS

Original: To demonstrate the effectiveness of the proposed T AGE on both node-level and graph-level tasks, we evaluate T AGE on three groups of real-world datasets that contain potentially multiple tasks.
Revised: To demonstrate the effectiveness of the proposed T AGE on both node-level and graph-level tasks, we evaluate T AGE on three groups of real-world datasets that contain potentially multiple tasks.

Original: The datasets are described as follows and their statistics are summarized in Appendix A.
Revised: The datasets are described as follows and their statistics are summarized in Appendix A.

Original: MoleculeNet .
Revised: MoleculeNet .

Original: The MoleculeNet (Wu et al., 2018) library provides a collection of molecular graph datasets for the prediction of different molecule properties.
Revised: The MoleculeNet (Wu et al., 2018) library provides a collection of molecular graph datasets for the prediction of different molecule properties.

Original: In a molecular graph, each atom in the molecule is considered as a node, and each bond is considered as an edge.
Revised: In a molecular graph, each atom in the molecule is considered as a node, and each bond is considered as an edge.

Original: The prediction of molecule properties is a graph-level task.
Revised: The prediction of molecule properties is a graph-level task.

Original: We include three graph classiﬁcation tasks from MoleculeNet to evaluate the explanation of graph-level tasks: HIV, SIDER, and BACE.
Revised: We include three graph classiﬁcation tasks from MoleculeNet to evaluate the explanation of graph-level tasks: HIV, SIDER, and BACE.

Original: Protein-Protein Interaction .
Revised: Protein-Protein Interaction .

Original: The Protein-Protein Interaction (PPI) (Zitnik & Leskovec, 2017) dataset documents the physical interactions between proteins in 24 different human tissues.
Revised: The Protein-Protein Interaction (PPI) (Zitnik & Leskovec, 2017) dataset documents the physical interactions between proteins in 24 different human tissues.

Original: In PPI graphs, each protein is considered as a node with its motif and immunological features, and there is an edge between two proteins if they interact with each other.
Revised: In PPI graphs, each protein is considered as a node with its motif and immunological features, and there is an edge between two proteins if they interact with each other.

Original: Each node in the graphs hasbinary labels associated with different protein functions.
Revised: Each node in the graphs hasbinary labels associated with different protein functions.

Original: As different protein functions are not exclusive to each other, the prediction of each protein function is considered an individual task.
Revised: As different protein functions are not exclusive to each other, the prediction of each protein function is considered an individual task.

Original: We utilize the ﬁrst ﬁve out of 121 tasks to evaluate the explanation of node-level tasks.
Revised: We utilize the ﬁrst ﬁve out of 121 tasks to evaluate the explanation of node-level tasks.

Original: E-commerce Product Network .
Revised: E-commerce Product Network .

Original: The E-commerce Product Network (EPN) 1 is constructed with subsampled, anonymized logs from an e-commerce store, where entities including buyers, products, merchants, and reviews are considered as nodes, and interactions between entities are considered as edges.
Revised: The E-commerce Product Network (EPN) 1 is constructed with subsampled, anonymized logs from an e-commerce store, where entities including buyers, products, merchants, and reviews are considered as nodes, and interactions between entities are considered as edges.

Original: We subsample the data for the sake of experimental evaluations and the dataset characteristics do not mirror actual production trafﬁc.
Revised: We subsample the data for the sake of experimental evaluations and the dataset characteristics do not mirror actual production trafﬁc.

Original: We study the explanation of the classiﬁcation of fraudulent entities (nodes), where the prediction for different types of entities are considered as individual tasks.
Revised: We study the explanation of the classiﬁcation of fraudulent entities (nodes), where the prediction for different types of entities are considered as individual tasks.

Original: We evaluate our framework speciﬁcally on classiﬁcations of the buyer, merchant, and review nodes.
Revised: We evaluate our framework speciﬁcally on classiﬁcations of the buyer, merchant, and review nodes.

Original: 4. E XPERIMENT S ETTINGS AND E VALUATION M ETRICS
Revised: 4. E XPERIMENT S ETTINGS AND E VALUATION M ETRICS

Original: For each real-world dataset, we evaluate explainers on multiple downstream tasks that share a single embedding model.
Revised: For each real-world dataset, we evaluate explainers on multiple downstream tasks that share a single embedding model.

Original: For consistency with industrial use cases, we perform the two-stage training paradigm to obtain GNN models to be explained.
Revised: For consistency with industrial use cases, we perform the two-stage training paradigm to obtain GNN models to be explained.

Original: In particular, we ﬁrst use unlabeled graphs to train the GNN-based embedding model in an unsupervised fashion.
Revised: In particular, we ﬁrst use unlabeled graphs to train the GNN-based embedding model in an unsupervised fashion.

Original: We then freeze the embedding model and use the learned embeddings to train individual downstream models structured as 2-layer MLPs.
Revised: We then freeze the embedding model and use the learned embeddings to train individual downstream models structured as 2-layer MLPs.

Original: Speciﬁcally, for graph-level classiﬁcation tasks in MoleculeNet, we employ the GNN pretraining strategy context prediction (Hu et al., 2020) to train a 5-layer GIN (Xu et al., 2019) as the embedding model on ZINC-2M (Sterling & Irwin, 2015) containing 2 million unlabeled molecules.
Revised: Speciﬁcally, for graph-level classiﬁcation tasks in MoleculeNet, we employ the GNN pretraining strategy context prediction (Hu et al., 2020) to train a 5-layer GIN (Xu et al., 2019) as the embedding model on ZINC-2M (Sterling & Irwin, 2015) containing 2 million unlabeled molecules.

Original: For the node-level classiﬁcation on PPI, we employ the self-supervised training method GRACE (Zhu et al., 2020) to train a 2-layer GCN (Kipf & Welling, 2017) on all 21 graphs from PPI without using labels.
Revised: For the node-level classiﬁcation on PPI, we employ the self-supervised training method GRACE (Zhu et al., 2020) to train a 2-layer GCN (Kipf & Welling, 2017) on all 21 graphs from PPI without using labels.

Original: For the larger-scale node-level classiﬁcation on EPN, we use graph autoencoder (GAE) (Kipf &
Revised: For the larger-scale node-level classiﬁcation on EPN, we use graph autoencoder (GAE) (Kipf &

Original: Welling, 2016) to train the embedding model on sampled subgraphs of EPN.
Revised: Welling, 2016) to train the embedding model on sampled subgraphs of EPN.

Original: More implementation details are provided in Appendix B.
Revised: More implementation details are provided in Appendix B.

Original: As the involved real-world datasets do not have ground truth for explanations, we follow previous studies (Pope et al., 2019; Yuan et al., 2020; 2021) to adopt a ﬁdelity score and a sparsity score to quantitatively evaluate the explanations.
Revised: As the involved real-world datasets do not have ground truth for explanations, we follow previous studies (Pope et al., 2019; Yuan et al., 2020; 2021) to adopt a ﬁdelity score and a sparsity score to quantitatively evaluate the explanations.

Original: Intuitively, the ﬁdelity score measures the level of change in the probability of the predicted class when removing important nodes or edges, whereas the sparsity score measures the relative amount of important nodes or nodes associated with important edges.
Revised: Intuitively, the ﬁdelity score measures the level of change in the probability of the predicted class when removing important nodes or edges, whereas the sparsity score measures the relative amount of important nodes or nodes associated with important edges.

Original: A formulation of the scores are provided in Appendix B. Note that compared to explanation evaluation with ground truths, the ﬁdelity score is considered more faithful to the model, especially when the model makes incorrect predictions, in which case the explanation ground truths become inconsistent with evidence to making the wrong predictions.
Revised: A formulation of the scores are provided in Appendix B. Note that compared to explanation evaluation with ground truths, the ﬁdelity score is considered more faithful to the model, especially when the model makes incorrect predictions, in which case the explanation ground truths become inconsistent with evidence to making the wrong predictions.

Original: In practice, one needs to trade off between the ﬁdelity score and the sparsity score by selecting the proper threshold for the importance.
Revised: In practice, one needs to trade off between the ﬁdelity score and the sparsity score by selecting the proper threshold for the importance.

Original: 4. Q UANTITATIVE S TUDIES
Revised: 4. Q UANTITATIVE S TUDIES

Original: We conduct two groups of quantitatively experimental comparisons.
Revised: We conduct two groups of quantitatively experimental comparisons.

Original: We ﬁrst demonstrate the explanation quality of individual tasks in terms of the ﬁdelity score and the sparsity score.
Revised: We ﬁrst demonstrate the explanation quality of individual tasks in terms of the ﬁdelity score and the sparsity score.

Original: We do this by comparing T AGE with multiple baseline methods including non-learning-based methods GradCAM (Pope et al., 2019) and DeepLIFT (Shrikumar et al., 2017), as well as learning-based methods
Revised: We do this by comparing T AGE with multiple baseline methods including non-learning-based methods GradCAM (Pope et al., 2019) and DeepLIFT (Shrikumar et al., 2017), as well as learning-based methods

Original: GNNExplainer (Ying et al., 2019) and PGExplainer (Luo et al., 2020).
Revised: GNNExplainer (Ying et al., 2019) and PGExplainer (Luo et al., 2020).

Original: We do not include other optimization or search-based methods such as Monte-Carlo tree search (Jin et al., 2020) due to the
Revised: We do not include other optimization or search-based methods such as Monte-Carlo tree search (Jin et al., 2020) due to the

Original: [Table]
Revised: [Table]

Original: signiﬁcant time cost on real-world datasets.
Revised: signiﬁcant time cost on real-world datasets.

Original: Note that to show the effectiveness of universal explanations over different downstream tasks, we only train one embedding explainer for all tasks in a dataset, on top of which a gradient-based downstream explainer is applied to explain multiple downstream tasks.
Revised: Note that to show the effectiveness of universal explanations over different downstream tasks, we only train one embedding explainer for all tasks in a dataset, on top of which a gradient-based downstream explainer is applied to explain multiple downstream tasks.

Original: In contrast, for existing learning-based methods, we need to train multiple explainers to explain downstream tasks individually.
Revised: In contrast, for existing learning-based methods, we need to train multiple explainers to explain downstream tasks individually.

Original: For all methods, we vary the threshold for selecting important nodes or edges and compare how ﬁdelity scores change over sparsity scores on each task and dataset.
Revised: For all methods, we vary the threshold for selecting important nodes or edges and compare how ﬁdelity scores change over sparsity scores on each task and dataset.

Original: The results are shown in Figure 4.
Revised: The results are shown in Figure 4.

Original: In particular, T AGE outperforms other learning-based explainers on BACE, SIDER, and PPI (tasks 0 and 1).
Revised: In particular, T AGE outperforms other learning-based explainers on BACE, SIDER, and PPI (tasks 0 and 1).

Original: For HIV and PPI (task 2), T AGE is more effective at higher sparsity levels, i.e. , when fewer nodes are considered important and masked.
Revised: For HIV and PPI (task 2), T AGE is more effective at higher sparsity levels, i.e. , when fewer nodes are considered important and masked.

Original: To justify the necessity of task-agnostic explanation and demonstrate the universal explanation ability of T AGE , we include PGExplainer as our baseline and compare the explanation quality when adopting a single explainer to explain multiple downstream tasks.
Revised: To justify the necessity of task-agnostic explanation and demonstrate the universal explanation ability of T AGE , we include PGExplainer as our baseline and compare the explanation quality when adopting a single explainer to explain multiple downstream tasks.

Original: For PGExplainer, we train multiple explainers on different downstream tasks and evaluate each explainer on different downstream tasks.
Revised: For PGExplainer, we train multiple explainers on different downstream tasks and evaluate each explainer on different downstream tasks.

Original: For T AGE , we train one explainer without downstream tasks and evaluate it on different downstream tasks.
Revised: For T AGE , we train one explainer without downstream tasks and evaluate it on different downstream tasks.

Original: Results shown in Table 2 (PPI) and Appendix D (MoleculeNet and EPN) indicate that task-speciﬁc explainers fail to generalize to different downstream tasks and hence are unable to provide universal explanations.
Revised: Results shown in Table 2 (PPI) and Appendix D (MoleculeNet and EPN) indicate that task-speciﬁc explainers fail to generalize to different downstream tasks and hence are unable to provide universal explanations.

Original: On the other hand, the task-agnostic explainer, although trained without downstream tasks, can provide explanations with even higher quality for any downstream tasks.
Revised: On the other hand, the task-agnostic explainer, although trained without downstream tasks, can provide explanations with even higher quality for any downstream tasks.

Original: GNNExplainer and PGExplainer should generally outperform task-agnostic explainers, as they are speciﬁc to data examples or tasks.
Revised: GNNExplainer and PGExplainer should generally outperform task-agnostic explainers, as they are speciﬁc to data examples or tasks.

Original: This should especially be true when T AGE and PGExplainer have the same level of parameters.
Revised: This should especially be true when T AGE and PGExplainer have the same level of parameters.

Original: However, we surprisingly ﬁnd that T AGE outperforms the learningbased baselines.
Revised: However, we surprisingly ﬁnd that T AGE outperforms the learningbased baselines.

Original: One possible reason can be the non-injective characteristic of the downstream
Revised: One possible reason can be the non-injective characteristic of the downstream

Original: MLPs on top of GNNs, where different embeddings can produce similar downstream prediction results.
Revised: MLPs on top of GNNs, where different embeddings can produce similar downstream prediction results.

Original: Due to this characteristic, the learning objective of T AGE computed between embeddings brings stronger supervision than the objective computed between ﬁnal predictions, as the latter objective does not guarantee consistency between embeddings or between input graphs and subgraphs.
Revised: Due to this characteristic, the learning objective of T AGE computed between embeddings brings stronger supervision than the objective computed between ﬁnal predictions, as the latter objective does not guarantee consistency between embeddings or between input graphs and subgraphs.

Original: 4. M ULTITASK E XPLANATION E FFICIENCY
Revised: 4. M ULTITASK E XPLANATION E FFICIENCY

Original: A major advantage of the task-agnostic explanation is that it removes the need for training individual explainers, which consumes the majority of the total time cost to explain a model on a dataset.
Revised: A major advantage of the task-agnostic explanation is that it removes the need for training individual explainers, which consumes the majority of the total time cost to explain a model on a dataset.

Original: We hence evaluate the efﬁciency of T AGE in terms of time cost for explanation and compare it to the two learning-based explainer baselines.
Revised: We hence evaluate the efﬁciency of T AGE in terms of time cost for explanation and compare it to the two learning-based explainer baselines.

Original: We record the time cost for the training and inference of different explanation methods on the same dataset and device, shown in Table 3.
Revised: We record the time cost for the training and inference of different explanation methods on the same dataset and device, shown in Table 3.

Original: All results are obtained from running the explanation on the PPI dataset with 121 node classiﬁcation tasks with a single Nvidia Tesla V100 GPU.
Revised: All results are obtained from running the explanation on the PPI dataset with 121 node classiﬁcation tasks with a single Nvidia Tesla V100 GPU.

Original: Although the inference time cost of T AGE is slightly higher than that of PGExplainer, the results show T AGE costs signiﬁcantly less time than GNNExplainer and
Revised: Although the inference time cost of T AGE is slightly higher than that of PGExplainer, the results show T AGE costs signiﬁcantly less time than GNNExplainer and

Original: PGExplainer, especially in the multitask cases ( T > 1 ).
Revised: PGExplainer, especially in the multitask cases ( T > 1 ).

Original: T AGE allows the explanation of many downstream tasks within a reasonable time duration.
Revised: T AGE allows the explanation of many downstream tasks within a reasonable time duration.

Original: 4. V ISUALIZATIONS AND E XPLANATION TO E MBEDDING D IMENSIONS
Revised: 4. V ISUALIZATIONS AND E XPLANATION TO E MBEDDING D IMENSIONS

Original: We visualize the explanations of the three learning-based explanation methods on the BACE task, which aims to predict the inhibitor effect of molecules to human β -secretase 1 (BACE-1) (Wu et al., 2018).
Revised: We visualize the explanations of the three learning-based explanation methods on the BACE task, which aims to predict the inhibitor effect of molecules to human β -secretase 1 (BACE-1) (Wu et al., 2018).

Original: Additional visualizations on HIV and SIDER are also provided in
Revised: Additional visualizations on HIV and SIDER are also provided in

Original: Appendix E.
Revised: Appendix E.

Original: The visualization results are shown in Figure 4.
Revised: The visualization results are shown in Figure 4.

Original: Each molecule visualization shows the top 10% important edges
Revised: Each molecule visualization shows the top 10% important edges

Original: [Table]
Revised: [Table]

Original: F
Revised: F

Original: N N N
Revised: N N N

Original: H 2 N N
Revised: H 2 N N

Original: O
Revised: O

Original: O F F F
Revised: O F F F

Original: F
Revised: F

Original: F
Revised: F

Original: NH
Revised: NH

Original: O
Revised: O

Original: N
Revised: N

Original: NH O
Revised: NH O

Original: O
Revised: O

Original: OH
Revised: OH

Original: H 2 N +
Revised: H 2 N +

Original: O
Revised: O

Original: N N + H 

Revised: N N + H

Original: OH
Revised: OH

Original: NH O
Revised: NH O

Original: O
Revised: O

Original: [Table]
Revised: [Table]

Original: O
Revised: O

Original: [Table]
Revised: [Table]

Original: F F
Revised: F F

Original: NH
Revised: NH

Original: O
Revised: O

Original: N
Revised: N

Original: NH O
Revised: NH O

Original: O
Revised: O

Original: OH
Revised: OH

Original: H 2 N +
Revised: H 2 N +

Original: [Table]
Revised: [Table]

Original: O
Revised: O

Original: N N + H 

Revised: N N + H

Original: OH
Revised: OH

Original: NH O
Revised: NH O

Original: O
Revised: O

Original: O O
Revised: O O

Original: F
Revised: F

Original: N N N
Revised: N N N

Original: H 2 N N
Revised: H 2 N N

Original: O
Revised: O

Original: O F F F
Revised: O F F F

Original: [Table]
Revised: [Table]

Original: N
Revised: N

Original: [Table]
Revised: [Table]

Original: O
Revised: O

Original: OH
Revised: OH

Original: [Table]
Revised: [Table]

Original: F
Revised: F

Original: N N N
Revised: N N N

Original: H 2 N
Revised: H 2 N

Original: [Table]
Revised: [Table]

Original: OH
Revised: OH

Original: [Table]
Revised: [Table]

Original: O
Revised: O

Original: N N + H 

Revised: N N + H

Original: OH
Revised: OH

Original: NH O
Revised: NH O

Original: O
Revised: O

Original: O O
Revised: O O

Original: F
Revised: F

Original: N N N
Revised: N N N

Original: H 2 N N
Revised: H 2 N N

Original: O
Revised: O

Original: O F F F (0.0193) (0.0494) (0.6287)
Revised: O F F F (0.0193) (0.0494) (0.6287)

Original: TAGE-Single Dimension
Revised: TAGE-Single Dimension

Original: F F
Revised: F F

Original: NH
Revised: NH

Original: O
Revised: O

Original: N
Revised: N

Original: NH O
Revised: NH O

Original: O
Revised: O

Original: OH
Revised: OH

Original: H 2 N +
Revised: H 2 N +

Original: O
Revised: O

Original: N N + H 

Revised: N N + H

Original: OH
Revised: OH

Original: NH O
Revised: NH O

Original: O
Revised: O

Original: O O
Revised: O O

Original: F
Revised: F

Original: N N N
Revised: N N N

Original: H 2 N N
Revised: H 2 N N

Original: O
Revised: O

Original: O F F F (0.8153) (0.6183) (0.3705)
Revised: O F F F (0.8153) (0.6183) (0.3705)

Original: [Figure]
Revised: [Figure]

Original: (bonds) predicted by an explainer with red shadow, together with the ﬁdelity score on the molecule.
Revised: (bonds) predicted by an explainer with red shadow, together with the ﬁdelity score on the molecule.

Original: The left task are explanations by T AGE to two speciﬁc graph embedding dimensions, without downstream mod els.
Revised: The left task are explanations by T AGE to two speciﬁc graph embedding dimensions, without downstream mod els.

Original: Embedding dimensions with greater among are selected in the visualizations.
Revised: Embedding dimensions with greater among are selected in the visualizations.

Original: To obtain explanations to certain embedding dimensions, we input the one-hot vectors to the embedding explainer as condition vectors.
Revised: To obtain explanations to certain embedding dimensions, we input the one-hot vectors to the embedding explainer as condition vectors.

Original: The visualization results indicate that while baseline methods select scattered edges as important, T AGE tends to select edges that form a connected substructure, which is more reasonable when explaining molecule property predictions where a certain functional group is important for the property.
Revised: The visualization results indicate that while baseline methods select scattered edges as important, T AGE tends to select edges that form a connected substructure, which is more reasonable when explaining molecule property predictions where a certain functional group is important for the property.

Original: In addition, the right three columns that dimensions in the embedding correspond to different substructures and T AGE is able to provide explanations to the dimensions without downstream tasks.
Revised: In addition, the right three columns that dimensions in the embedding correspond to different substructures and T AGE is able to provide explanations to the dimensions without downstream tasks.

Original:  C ONCLUSIONS
Revised:  C ONCLUSIONS

Original: Existing task-speciﬁc learning-based explainers become inapplicable under real scenarios when downstream tasks or models are unavailable and suffer from inefﬁciency when explaining real-world graph datasets with multiple downstream tasks.
Revised: Existing task-speciﬁc learning-based explainers become inapplicable under real scenarios when downstream tasks or models are unavailable and suffer from inefﬁciency when explaining real-world graph datasets with multiple downstream tasks.

Original: We introduced T AGE , including the task-agnostic
Revised: We introduced T AGE , including the task-agnostic

Original: GNN explanation pipeline and the self-supervised training framework to train the embedding explainer without knowing downstream tasks or models.
Revised: GNN explanation pipeline and the self-supervised training framework to train the embedding explainer without knowing downstream tasks or models.

Original: Our experiments demonstrate that the T AGE generally achieves higher explanation quality in terms of ﬁdelity and sparsity with the signiﬁcantly reduced explanation time cost.
Revised: Our experiments demonstrate that the T AGE generally achieves higher explanation quality in terms of ﬁdelity and sparsity with the signiﬁcantly reduced explanation time cost.

Original: We discuss potential limitations and their solutions in Appendix G.
Revised: We discuss potential limitations and their solutions in Appendix G.

Original: R EPRODUCIBILITY S TATEMENT
Revised: R EPRODUCIBILITY S TATEMENT

Original: The experiment setting and evaluation protocol are described in Section 4.
Revised: The experiment setting and evaluation protocol are described in Section 4.

Original: In particular, we describe how the models to be explained are trained.
Revised: In particular, we describe how the models to be explained are trained.

Original: To further ensure reproducibility, we provide implementation details including explainer conﬁgurations, training settings, and how evaluation are performed in Appendix B.
Revised: To further ensure reproducibility, we provide implementation details including explainer conﬁgurations, training settings, and how evaluation are performed in Appendix B.

Original: The detailed computation of the sparsity and ﬁdelity scores are provided in Appendix C.
Revised: The detailed computation of the sparsity and ﬁdelity scores are provided in Appendix C.

Original: The code to fully reproduce the results will be released upon acceptance.
Revised: The code to fully reproduce the results will be released upon acceptance.

Original: A D ATASET S TATISTICS
Revised: A D ATASET S TATISTICS

Original: The statistics of datasets used for evaluating T AGE are shown in Table 4.
Revised: The statistics of datasets used for evaluating T AGE are shown in Table 4.

Original: [Table]
Revised: [Table]

Original: B I MPLEMENTATION D ETAILS
Revised: B I MPLEMENTATION D ETAILS

Original: Structure of explainer .
Revised: Structure of explainer .

Original: Our implementation is based on Pytorch (Paszke et al., 2019), Pytorch Geometric (Fey & Lenssen, 2019), and Dive-into-graphs (Liu et al., 2021).
Revised: Our implementation is based on Pytorch (Paszke et al., 2019), Pytorch Geometric (Fey & Lenssen, 2019), and Dive-into-graphs (Liu et al., 2021).

Original: We implement the explainer with a linear projection f p that maps the condition vector p to the same dimension as concatenated embeddings, and a 2-layer MLP with ReLU activation that maps concatenated embeddings with mask to the important score.
Revised: We implement the explainer with a linear projection f p that maps the condition vector p to the same dimension as concatenated embeddings, and a 2-layer MLP with ReLU activation that maps concatenated embeddings with mask to the important score.

Original: Implementation of training objectives .
Revised: Implementation of training objectives .

Original: We adopt the Jason-Shannon Estimator as the lower bound for mutual information maximization for the two public datasets.
Revised: We adopt the Jason-Shannon Estimator as the lower bound for mutual information maximization for the two public datasets.

Original: For graph-level tasks, given a minibatch of N samples, we consider the embeddings of a graph G and its subgraph G s as a positive pair (with N positive pairs in total), and the embeddings of a graph G i and the subgraph G j,s of another sample as a negative pair (with N 2 − N pairs in total).
Revised: For graph-level tasks, given a minibatch of N samples, we consider the embeddings of a graph G and its subgraph G s as a positive pair (with N positive pairs in total), and the embeddings of a graph G i and the subgraph G j,s of another sample as a negative pair (with N 2 − N pairs in total).

Original: For node-level tasks, we still randomly sample N nodes from the entire graph at each iteration and compute the contrastive losses on original embeddings of the N nodes, and embedding of the N nodes when the important subgraph is selected, respectively for each node.
Revised: For node-level tasks, we still randomly sample N nodes from the entire graph at each iteration and compute the contrastive losses on original embeddings of the N nodes, and embedding of the N nodes when the important subgraph is selected, respectively for each node.

Original: Similar to graph-level tasks, we consider embeddings of the same node (in original graph or in subgraph) as a positive pair, and a embeddings of node i in full graph and node j in selected subgraph as a negative pair ( N 2 − N in total).
Revised: Similar to graph-level tasks, we consider embeddings of the same node (in original graph or in subgraph) as a positive pair, and a embeddings of node i in full graph and node j in selected subgraph as a negative pair ( N 2 − N in total).

Original: Training conﬁgurations .
Revised: Training conﬁgurations .

Original: We set the hyperparameters in the size regularization term to λ s = 0 .and
Revised: We set the hyperparameters in the size regularization term to λ s = 0 .and

Original: λ
Revised: λ

Original: e = 0 .
Revised: e = 0 .

Original: 002 , respectively.
Revised: 002 , respectively.

Original: For graph-level explanation on MoleculeNet, we train the embedding explainer on ZINC-2M with learning rate 1 e − 4 and mini-batch size 256 for 1 epoch.
Revised: For graph-level explanation on MoleculeNet, we train the embedding explainer on ZINC-2M with learning rate 1 e − 4 and mini-batch size 256 for 1 epoch.

Original: The random condition vectors are generated from laplace distribution Laplace (0 , 0 . 2) .
Revised: The random condition vectors are generated from laplace distribution Laplace (0 , 0 . 2) .

Original: For node-level explanation on PPI, we train the embedding explainer on PPI without labels with learning rate 5 e −and mini-batch size 4 for 1 epoch.
Revised: For node-level explanation on PPI, we train the embedding explainer on PPI without labels with learning rate 5 e −and mini-batch size 4 for 1 epoch.

Original: The random condition vectors are generated from laplace distribution Laplace (0 , 0 . 1) .
Revised: The random condition vectors are generated from laplace distribution Laplace (0 , 0 . 1) .

Original: For the EPN dataset, we train the embedding explainer with InfoNCE loss, learning rate 1 e − 4 and mini-batch size 16 .
Revised: For the EPN dataset, we train the embedding explainer with InfoNCE loss, learning rate 1 e − 4 and mini-batch size 16 .

Original: The random condition vectors are generated from laplace distribution Laplace (0 , 0 . 25) .
Revised: The random condition vectors are generated from laplace distribution Laplace (0 , 0 . 25) .

Original: The hyperparameters in the size regularization term are set to λ s = 0 .
Revised: The hyperparameters in the size regularization term are set to λ s = 0 .

Original: 5 and λ e = 0 for the stable training with InfoNCE.
Revised: 5 and λ e = 0 for the stable training with InfoNCE.

Original: Evaluation .
Revised: Evaluation .

Original: In molecule and protein property prediction, we are usually interested in the positive samples, i.e. , the existence of what substructure lead to a certain property.
Revised: In molecule and protein property prediction, we are usually interested in the positive samples, i.e. , the existence of what substructure lead to a certain property.

Original: For learning-based baseline methods, we ﬁnd it common that only one class of the two have good explanation, and the class with higher explanation quality is not necessarily the positive class.
Revised: For learning-based baseline methods, we ﬁnd it common that only one class of the two have good explanation, and the class with higher explanation quality is not necessarily the positive class.

Original: For example, PGExplainer has a near-to-zero ﬁdelity score for the positive class of SIDER.
Revised: For example, PGExplainer has a near-to-zero ﬁdelity score for the positive class of SIDER.

Original: We hence compare only the higher ﬁdelity score among two classes for all explanation methods and datasets.
Revised: We hence compare only the higher ﬁdelity score among two classes for all explanation methods and datasets.

Original: C F IDELITY AND S PARSITY
Revised: C F IDELITY AND S PARSITY

Original: Given a set of graphs { G i } and node masks m predicted by the explainer, the ﬁdelity score and the sparsity score are computed as follows.
Revised: Given a set of graphs { G i } and node masks m predicted by the explainer, the ﬁdelity score and the sparsity score are computed as follows.

Original: [Equation]
Revised: [Equation]

Original: where N denotes the number of graphs or nodes to be explained, f denotes the GNN model associated with a speciﬁc downstream task, c i denotes the class of interest, which can be either the labeled class or the original predicted class, G i and G 1 − m i
Revised: where N denotes the number of graphs or nodes to be explained, f denotes the GNN model associated with a speciﬁc downstream task, c i denotes the class of interest, which can be either the labeled class or the original predicted class, G i and G 1 − m i

Original: i denote the original graph and graph with important nodes removed, respectively.
Revised: i denote the original graph and graph with important nodes removed, respectively.

Original: Explanations with both scores higher are better.
Revised: Explanations with both scores higher are better.

Original: D A DDITIONAL R ESULTS FOR U NIVERSAL E XPLANATION A BILITY
Revised: D A DDITIONAL R ESULTS FOR U NIVERSAL E XPLANATION A BILITY

Original:  Evaluation results for the universal explanation ability on the MoleculeNet and EPN datasets are shown in Table 5 and Table 6, respectively. 
Revised: Universal explanation performance on MoleculeNet and EPN . Evaluation results for the universal explanation ability on the MoleculeNet and EPN datasets are shown in Table 5 and Table 6, respectively. [Table] [Table]

Original: On the BACE dataset and task, we additionally compare
Revised: [Table] Comparison of explanation performance when trained on different datasets . Speciﬁcally for the MoleculeNet dataset, as there is a larger unlabeled dataset, ZINC, available for the ﬁrst stage training of encoder, the training of our exlainer is also performed on the ZINC dataset. For a more strict comparison with the baseline explainer who is trained on individual MoleculeNet dataset, we additional evaluate the explanation quality when the same individual MoleculeNet dataset is used to train TAGE.

Revised: The results are shown in Table 7.

Revised: When trained on the same datasets individually,

Original: TAGE  with another recent SOTA learning-based method GEM (Lin et al., 2021) whose explainer is trained based on the Granger causality in Table 7.
Revised: TAGE still performs better than the baseline explainer in terms of ﬁdelity scores. In the individual dataset case, we need to train different explainers, similarly to the training of PGExplainer, as the datasets for the four tasks are different. Evaluation on the synthetic dataset BA-Shapes . On the BACE dataset and task, we additionally compare TAGE with another recent SOTA learning-based method GEM (Lin et al., 2021) whose explainer is trained based on the Granger causality in Table 8.

Original: Note that GEM is not originally proposed under our setting.
Revised: Note that GEM is not originally proposed under our setting.

Original: It assumes that there is a ﬁxed number of important nodes when performing explanation and hence the ﬁnal explanation is a boolean selection of nodes.
Revised: It assumes that there is a ﬁxed number of important nodes when performing explanation and hence the ﬁnal explanation is a boolean selection of nodes.

Original: We adapt GEM to compute ﬁdelity scores under different sparsity scores by varying the threshold when generating explanation groundtruth with Granger causality.
Revised: We adapt GEM to compute ﬁdelity scores under different sparsity scores by varying the threshold when generating explanation groundtruth with Granger causality.

Original: [Table]
Revised: [Table]

Original: [Table]

Original: [Table]
Revised: [Table]

Original: E V ISUALIZATIONS ON HIV AND SIDER
Revised: E V ISUALIZATIONS ON HIV AND SIDER

Original: Visualizations on HIV and SIDER are shown in Figure 5 and Figure 6, respectively.
Revised: Visualizations on HIV and SIDER are shown in Figure 5 and Figure [Table] respectively.

Original: [Figure]
Revised: [Figure]

Revised: (0.0140) (3.099e-6)

Original: O
Revised: O

Original: OH OH
Revised: OH OH

Original: O OH
Revised: O OH

Original: [Figure]
Revised: OH O HO HO HO

Original: OH
Revised: OH

Original: [Figure]
Revised: Cl

Original: HO
Revised: Cl OH Cl Cl Cl

Original:  N
Revised: O O N Fe 3- Cl N

Original: [Figure]
Revised: O HO N + O HO N +

Original: O
Revised: O

Original: OH 
Revised: OH OH O OH

Original: OH 
Revised: OH O OH OH

Original: O 
Revised: O OH OH O OH

Original: OH 
Revised: OH O HO HO HO OH

Revised: Cl

Original:  OH 
Revised: Cl OH Cl Cl Cl O O N Fe

Original:  O 
Revised: 3- Cl N O HO N + O HO

Original:  OH 
Revised: N + O OH OH O HO HO

Original:  OH 
Revised: HO O OH OH O HO HO

Original:  O 
Revised: HO O OH OH O HO HO HO

Original: OH 
Revised: OH Cl Cl OH Cl Cl Cl

Original: OH
Revised: OH

Original: [Figure]
Revised: Cl Cl

Revised: OH Cl Cl Cl

Original: OH
Revised: OH

Revised: Cl

Revised: Cl

Revised: OH Cl Cl Cl

Revised: O

Original: [Figure]
Revised: O N Fe 3- Cl N O HO N + O

Original: OH
Revised: HO

Original: [Figure]
Revised: N + O O N Fe 3- Cl N

Revised: O HO

Revised: N + O

Original: OH
Revised: HO

Original: [Figure]
Revised: N + O O N Fe 3- Cl N

Original: are explanations to two certain embedding dimensions without downstream tasks.
Revised: O HO N + O HO N +

Original: [Figure]
Revised: [Figure]

Original: TAGE
Revised: TAGE

Original: [Figure]
Revised: [Figure]

Original: TAGE
Revised: TAGE

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Figure]
Revised: [Figure]

Original: Cl
Revised: Cl

Original: [Figure]
Revised: [Figure]

Original: N
Revised: N

Original: S
Revised: S

Original: Cl
Revised: Cl

Original: N
Revised: N

Original: S
Revised: S

Original: Cl
Revised: Cl

Original: N
Revised: N

Original: S
Revised: S

Original: Cl
Revised: Cl

Original: O
Revised: O

Original: Cl
Revised: Cl

Original: [Figure]
Revised: [Figure]

Original: Cl
Revised: Cl

Original: [Figure]
Revised: [Figure]

Original: O
Revised: O

Original: Cl
Revised: Cl

Original: [Figure]
Revised: [Figure]

Original: F E XPERIMENTAL STUDIES ON THE SYNTHETIC DATASETS BA-S HAPES
Revised: F E XPERIMENTAL STUDIES ON THE SYNTHETIC DATASETS BA-S HAPES

Original: We perform an additional evaluation on the BA-Shapes synthetic datasets used in GNNExplainer Ying et al.
Revised: We perform an additional evaluation on the BA-Shapes synthetic datasets used in GNNExplainer Ying et al.

Original: and provided by Pytorch-Geometric (Fey & Lenssen, 2019).
Revised: and provided by Pytorch-Geometric (Fey & Lenssen, 2019).

Original: The synthetic dataset is less complicated compared to the real-world datasets.
Revised: The synthetic dataset is less complicated compared to the real-world datasets.

Original: We train a 3-layer GCN for the node-classiﬁcation with a training accuracy of 0.95.
Revised: We train a 3-layer GCN for the node-classiﬁcation with a training accuracy of 0.95.

Original: The AUC score (for importance edges) of
Revised: The AUC score (for importance edges) of

Original: TAGE is 0.999 compared to 0.963 and 0.925 of PGExplainer and GNNExplainer, respectively.
Revised: TAGE is 0.999 compared to 0.963 and 0.925 of PGExplainer and GNNExplainer, respectively.

Original: Note that the baseline scores are from the PGExplainer paper and some re-implementations 23 of PGExplainer can also achieve an AUC score of 0.999.
Revised: Note that the baseline scores are from the PGExplainer paper and some re-implementations 23 of PGExplainer can also achieve an AUC score of 0.999.

Original: Our purpose to show our score on BA-Shapes is to demonstrate that TAGE is on par with its baselines even when considering the typical singletask setting.
Revised: Our purpose to show our score on BA-Shapes is to demonstrate that TAGE is on par with its baselines even when considering the typical singletask setting.

Original: Figure 7 visualizes 20 examples of explanations.
Revised: Figure 7 visualizes 20 examples of explanations.

Original: TAGE is able to provide accurate explanations for all the 20 examples.
Revised: TAGE is able to provide accurate explanations for all the 20 examples.

Original: G D ISCUSSION OF LIMITATIONS AND POTENTIAL SOLUTIONS
Revised: G D ISCUSSION OF LIMITATIONS AND POTENTIAL SOLUTIONS

Original: Inductive learning of explanations.
Revised: Inductive learning of explanations.

Original: Our study focus on the setting of inductive learning of the explanation, i.e. , to train the explainer on a given dataset and perform inference on new coming data.
Revised: Our study focus on the setting of inductive learning of the explanation, i.e. , to train the explainer on a given dataset and perform inference on new coming data.

Original: There are many work conducted under the inductive setting, such as PGExplainer.
Revised: There are many work conducted under the inductive setting, such as PGExplainer.

Original: All methods under this setting may have a potential limitation that the explainer may suffer from some dataset bias when training data and the data to be explained are inconsistent.
Revised: All methods under this setting may have a potential limitation that the explainer may suffer from some dataset bias when training data and the data to be explained are inconsistent.

Original: This is an interesting problem that requires further investigation.
Revised: This is an interesting problem that requires further investigation.

Original: However, we believe that this is a separate problem and applies to all inductive learning methods.
Revised: However, we believe that this is a separate problem and applies to all inductive learning methods.

Original: In addition, the size of graph could be inconsistent for training and inference.
Revised: In addition, the size of graph could be inconsistent for training and inference.

Original: To tackle this issue, we obtain the substructure by selecting top k percentage of edges according to their important scores.
Revised: To tackle this issue, we obtain the substructure by selecting top k percentage of edges according to their important scores.

Original: Black-box explanations.
Revised: Black-box explanations.

Original: Similarly to our baseline method PGExplainer, our explainer relies on node embeddings as inputs to the explainer.
Revised: Similarly to our baseline method PGExplainer, our explainer relies on node embeddings as inputs to the explainer.

Original: In particular, the node embeddings serve as representations to allow explainers identify each node.
Revised: In particular, the node embeddings serve as representations to allow explainers identify each node.

Original: It is required by any (inductively) learning based explanations to tell neural network-based explainers which edge they are looking at.
Revised: It is required by any (inductively) learning based explanations to tell neural network-based explainers which edge they are looking at.

Original: A limitation of the inductive methods is that when the node embeddings may become unavailable when explaining a black-box model.
Revised: A limitation of the inductive methods is that when the node embeddings may become unavailable when explaining a black-box model.

Original: The study of explaining black-box models (where only output is available) is a different direction of study in scenarios like attacking.
Revised: The study of explaining black-box models (where only output is available) is a different direction of study in scenarios like attacking.

Original: Many current SOTA explanation approaches, such as Grad-Cam, GNN-LRP, and PGExplainer, fail under the black-box setting.
Revised: Many current SOTA explanation approaches, such as Grad-Cam, GNN-LRP, and PGExplainer, fail under the black-box setting.

Original: However, if one would like to adapt our approach to the black-box setting, it is still feasible by adopting a surrogate model for the black-box model and perform explanation on the surrogate model.
Revised: However, if one would like to adapt our approach to the black-box setting, it is still feasible by adopting a surrogate model for the black-box model and perform explanation on the surrogate model.

Original: In addition, as mentioned above, the node embeddings are mainly used to identify which node the explainer is looking at, we does not necessarily require the original embedding.
Revised: In addition, as mentioned above, the node embeddings are mainly used to identify which node the explainer is looking at, we does not necessarily require the original embedding.

Original: When node embedding are unavailable, we can still use any representation of nodes as long as it can identify the node based on its feature and topology.
Revised: When node embedding are unavailable, we can still use any representation of nodes as long as it can identify the node based on its feature and topology.