[Re] Parameterized Explainer for Graph Neural Network
Anonymous Author(s)
AfﬁliationAddress email
Scope of Reproducibility
In this work we perform a replication study of the paper Parameterized Explainer for Graph Neural Network .
Thereplication experiment focuses on three main claims: (1) Is it possible to reimplement the proposed method in a differentframework?
(2) Do the main claims with respect to the GNNExplainer hold?
(3) Is the used evaluation method a validmethod for explaining the classiﬁcation decision by Graph Neural Networks?
Methodology
The authors’ TensorFlow code was largely used as starting point for our reimplementation in PyTorch.
However, largeparts of the evaluation setup were missing and differences were found between the listed conﬁgurations in the paper andthe code.
As a result, our codebase contains a large portion of novel code and introduces a different method for trackingexperimental conﬁgurations.
Using the new codebase all experiments are replicated.
In addition to this, a short ablationstudy is performed.
Results
Due to numerous inconsistencies between code and paper, it is not possible to replicate the original results using thepaper alone.
With help of the original codebase, a number of the original results can be retrieved.
The main comparisonclaim of the paper, to improve over the preceding GNNExplainer, does hold.
However, after performing the replicationexperiments, some questions regarding the validity of the used evaluation setup in the original paper remain.
What was easy
The method proposed by the authors for explaining the Graph Neural Networks is easy to comprehend and intuitive.
Re-implementation of the method is straightforward using a modern deep learning framework.
The datasets used forthe experimental setup were all provided together with their codebase.
What was difﬁcult
The main difﬁculty arose from the difference between the experimental conﬁgurations discussed in the paper andimplemented in the code.
There were a number of small inconsistencies (eg. incorrect hyperparameter settings), butalso some major ones (eg.
using batch-normalization in training mode during evaluation).
This issue was worsened bythe fractured reporting of conﬁgurations in the code.
Communication with original authors
Contact was made with the authors on two occasions.
During the ﬁrst exchange the authors conﬁrmed a number of clarifying questions and conﬁrmed that the conﬁgurations as presented in the codebase were to be used instead of those provided in the paper.
In the second exchange our reservations concerning the used experimental evaluation were conveyed to the authors.
The authors did not share our concerns.
Introduction
Graph Neural Networks (GNNs) emerged as state-of-the-art models in machine learning, capturing both graph structureand node features through recursively incorporating a graph’s previous node information.
GNNs are able to deliverstate-of-the-art performances in matters such as graph/node classiﬁcation and link prediction.
As for most Neural Networks, the ’reasoning’ towards classiﬁcation inside GNNs is not intuitive to humans.
The authorsof the paper GNNExplainer: Generating Explanations for Graph Neural Networks [9] address this problem and tryto solve it by introducing GNNE XPLAINER ; an optimization task that maximizes the mutual information between a
GNN’s prediction and a distribution of possible sub-graph structures.
The GNNExplainer’s algorithm can identify thesub-graph and node structure responsible for a given classiﬁcation.
Based on the work done in [9], Luo, D. et al.
claimto have further developed GNNExplainer in their paper Parameterized Explainer for Graph Neural Network [5].
Thepaper introduces PGE XPLAINER ; a general parameterized explainer that applies to any GNN based models in bothtransductive and inductive settings.
The authors of the paper ﬁrst formulate the learning objective of PGExplainer.
Using the same datasets as [9], theyclaim to outperform GNNExplainer up to 24 .
7% in Area under the ROC Curve (AUC) [4] score.
Furthermore theauthors state that PGExplainer can speed up computations up to 108 times faster than GNNExplainer.
These and furtherclaims made in the PGExplainer paper will be evaluated in this report by replicating and extending the performedevaluation in a replication study.
Scope of reproducibility The focus of our reproducibility study is on the experimental comparison between the
PGExplainer and the preceding GNNExplainer.
The authors of the original PGExplainer paper include a number ofother benchmarks in their evaluation, but focus their comparison primarily on the GNNExplainer.
For this reason itmakes sense for us to do the same.
In contrast to the original paper, we will base our entire comparison on reimplementations of both methods.
In theoriginal paper, the authors partly copy the results from the GNNExplainer and partly use their own re-implementationto obtain the GNNExplainer scores.
In communication the authors stated that the decision to partly copy the results wasmade due to lackluster results in their own re-implementation.
As the quality of an explanation is highly dependent onthe model it aims to explain, we believe that it would be beneﬁcial to re-implement both methods in the same frameworkand perform their evaluation on equal footing.
We will use PyTorch as the framework for doing so.
For the reimplementation of the PGExplainer the authors’ own TensorFlow-based codebase provided in their paperwill be used as the main starting-point.
However, during inspection of the codebase, we found that there are a numberof signiﬁcant differences between the conﬁgurations used for both the trained models that we wish to explain andthe PGExplainer itself between what is described in the paper and what is actually implemented in the code.
Afterdiscussing with the authors, the conclusion was reached that the conﬁgurations used in the code should serve as thestarting point for the replication.
Part of our reproduction experiment will focus on validating if these are indeed thecorrect conﬁgurations.
In short, our replication experiment aims to validate the following aspects of the original paper.
(a) Given the original codebase and conﬁguration ﬁles provided therein, is it possible to reimplement the PGEx-
plainer method using a different framework?
And if so, are the provided conﬁgurations sufﬁcient to obtain the
presented quantitative, qualitative and efﬁciency results.
(b) The authors claim that their PGExplaimer greatly improves over the previously proposed GNNExplainer.
We
aim to validate that this claim holds with both methods evaluated using the same framework and evaluation.
(c) Evaluation of explanation methods is notoriously hard.
We wish to validate if the evaluation method used in
the original paper is a sound approach
for doing so.
The remainder of this work will be structured as follows.
In the next section we will provide the needed background onthe PGExplainer.
Following this, we will provide a short overview of the codebase accompanying this reproduction.
Insection 4, we will discuss the original experimental setup in depth and highlight some key components not discussedin the original paper.
Section 5 will present the replicated results and compare them to the original paper.
Based onthe highlighted components in section 4 and some results presented in section 5, section 6 will raise some questionregarding the evaluation setup used.
In the last section, we will summarize our replication.
PGExplainer
The authors start by dividing an input graph G o in two subgraphs, such that G o =
G s + ∆ G .
G s represents theexplanatory graph that makes important contributions towards the graph classiﬁcation, while ∆ G represents the remainder of the initial graph.
The main task therefore is to ﬁnd the optimal subgraph G s .
This is achieved through
Mutual Information ( MI ) maximization:
[Equation]
Which uses the GNN’s classiﬁcation prediction Y o and its input G o .
The MI maximization is done by deducting theconditional entropy from the marginal entropy.
Which is equivalent to minimizing the conditional entropy.
To avoid having an exploding exponential amount of candidates, the authors assume the explanatory graphs used are
Gilbert random graphs [3], where selections of edges from the original input graph G o are conditionally independent toeach other.
Using relaxation, the learning objective is rewritten as
[Equation]
where q (Θ) is the distribution of the parameterized explanatory graph.
Each graph edge obtains a continuous variablein range (0 , 1) .
A random graph ˆ G s is sampled from edge distributions and fed to the trained GNN model obtaining prediction ˆ Y s .
Following [9], the authors modify the conditional entropy with cross-entropy H ( Y o , ˆ Y s ) , where ˆ Y s is the prediction ofthe GNN model with ˆ G s as input.
Using Monte Carlo approximation, the learning objective becomes
[Equation]
) with Φ as the parameters in the trained GNN, K as the number of sampled graphs, C as the number of labels and ˆ G ( k ) sthe k -th sampled graph, parameterized by Ω .
Furthermore, PGExplainer is used to collectively provide explainations for multiple instances I .
The authors presentthe learning objective of this set of instances as follows.min Ψ − (cid:88)
i ∈I
K (cid:88)
[Equation]
(cid:88)
[Equation] [Equation]
Here Ψ are parameters in the explanation network, G ( i ) the input graph and ˆ G ( i,k ) the k -th sampled graph for the i -thinstance.
Using the above, the authors consider two explainer instances; one for node classiﬁcation and one for graphclassiﬁcation.
Both cases use a MLP parameterized by Ψ .
Reimplementation of code
This section shortly summarizes the main structure of the code accompanying this reproducibility check and providesthe information needed to reproduce the experiments presented.
Our reimplementation of the PGExplainer is based onthe PyTorch [6] framework.
More speciﬁcally, it uses the third party extension of PyTorch for Graph Neural Networkscalled PyTorch-Geometric [2].
The codebase is structured for the two main tasks performed in this paper; training the GNNs that will be explainedby the PGExplainer and performing a replication of the original experiments.
Additional scripts are included forperforming the evaluations presented in the appendix.
Each script is self-contained, handling things such as loadingthe dataset, loading the correct model and setting the hyperparameters.
Each of these things are predeﬁned in jsonconﬁguration ﬁles.
For sections directly related to a part of the codebase we have added a link to the correspondingmodule on GitHub.
(Links will be added after review)
3.1 Experiment conﬁguration ﬁles
[configs/selector.py]
The codebase contains a large number of predeﬁned conﬁguration ﬁles.
These conﬁguration ﬁles are the main workinghorse for making the experiments presented in this work reproducible.
There are two different types of conﬁgurations,one for each of the two main tasks mentioned previously.
Shared between tasks is the common occurrence of the dataset,model and seed used.
If a task is to be performed a number of times to achieve an average, the seed is replaced with alist of seeds.
A full description of the conﬁguration ﬁle setup can be found in Appendix A.
As these conﬁguration ﬁles provide a reliable source for all relevant information needed to perform our evaluation,we will—for the remainder of this paper—only disclose the information needed to comprehend the experiment.
For details irrelevant to understanding the results—e.g. the used learning rate and speciﬁc framework versions—we refer tothe provided conﬁguration and codebase 1 .
We understand that this breaks the papers self-containment.
However, webelieve that regarding the balance between page restrictions and replicability completeness, separating the concern ofreplicability from paper to codebase is the correct way to go.
A single source of replicability information also preventsinconsistencies between the paper and the code base.
As the paper under consideration will highlight, this is a concern.
Experiment Setup
In this section we will introduce the setup of the experimental evaluation performed by the authors of the PGExplainer.
While replicating their evaluation, we found that a number of steps were making assumptions that were not welldocumented.
This includes the samples used for calculating the AUC score.
In this section we will spend time on thesesteps.
Additionally, some minor mistakes made in the original evaluation were rectiﬁed during our reproduction.
Thesechanges will also be highlighted here.
The experimental setup used by the authors of the PGExplainer follows that of the GNNExplainer [9] with a number ofextensions.
To clarify, the authors’ proposed method serves the purpose of explaining the classiﬁcation decision of a
GNN.
Hence, the experiments used to evaluate the PGExplainer focus on the explanations provided by the PGExplainerfor the underlying model.
Speciﬁcally, the evaluation is repeated for six different datasets, and thus, for six differentunderlying models.
The six datasets span two different classiﬁcation tasks; node-classiﬁcation and graph-classiﬁcation.
4.1 Datasets
[datasets/dataset_loaders.py]
The node classiﬁcation task is performed using four synthetic datasets (a-d).
All of which are ﬁrst introduced in the
GNNExplainer paper [9].
The graph classiﬁcation task is performed using two datasets (e-f), one synthetic and one real.
A reoccurring concept in all synthetic datasets is the so called motif .
Motifs are highly structured subgraphs—e.g. 9nodes connected in a 2D grid.
These subgraphs are then expanded by attaching them to a randomly generated graph ofa different structural form—e.g. Barabasi-Albert (BA) graph [1] or trees.
Motifs play a crucial role in determiningground-truth explanations for our evaluations, as we will see later.
BA
Shapes dataset consists of single base graph to random BA closely
BA
4.2 Model
[models/GNN_paper.py]
There are a number of large differences between the implementation of the models trained for each dataset and howthey are described in the paper.
These changes are different between the node and graph classiﬁcation tasks.
Node classiﬁcation The authors describe the model for node classiﬁcation to be three consecutive Graph Convolutionlayers feeding directly into the fully connected classiﬁcation.
The model in the codebase however ﬁrst concatenates thethree intermediate outputs of the Graph Convolution layers before using this enlarged embedding as the input for thefully connected classiﬁcation layer.
The coded version of the models is similar to what is used for evaluation in the
GNNExplainer paper [9].
To keep the evaluation consistent, we will therefore use the coded model version instead ofthe one described in the paper for our evaluation.
Moreover, we were not able to get the model described in the paper totrain to the same accuracy using the provided hyperparameters.
In addition to the architecture change, we found the node classiﬁcation models to use an undocumented batchnormalization layer after the ﬁrst and second Graph Convolution layer.
Unfortunately, the original codebase containedan error that resulted in these batch-normalization layers being kept in training mode during evaluation.
This observationwas conﬁrmed by the authors in communication and has since been resolved.
In the same communication the authorsexpressed that to be able to reproduce their results, the batch normalization layers will have to be kept in training mode.
We believe that this will compromise the usability of our reproducibility experiment and therefore decided to removethe batch normalization layers all together.
For completeness full replication of the authors evaluation with a modelcontaining batch normalization is included in Appendix B.
Graph classiﬁcation
The graph classiﬁcation models are more in line with the models described in the paper thanthe node classiﬁcation models.
The difference is the use of both max and mean pooling over the output of the ﬁnal
Graph Convolution layer.
These two pooling types are concatenated to form inputs for fully connected layers.
4.3 Evaluation metrics
[tasks/replication.py]
For each dataset, the explanations are evaluated using three broad categories; quantitative, qualitative and efﬁciency.
4.3.1 Quantitative evaluation
[evaluation/AUCEvaluation.py]
For each dataset the explanations provided by the PGExplainer are compared to ground-truth explanations.
Theseground-truths describe for each sample which edges should or should not be included in the explanation.
Using thismethodology, the quantitative evaluation can be performed similar to a binary classiﬁcation task.
For this reason, theauthors present the quantitative score using the AUC scoring metric.
Ground Truth
For node classiﬁcation the ground-truth explanation is determined globally—i.e. for all node samplesthe edges have the same ground-truth explanation label.
Speciﬁcally, for each edge it is determined if the two nodes itconnects are part of a motif.
When this is the case, the edge is labelled as positive for the ground-truth explanation.
Otherwise, the edge is labelled as negative for the ground-truth explanation.
For graph classiﬁcations this is dependenton the dataset used and how the ground-truth explanations are generated.
For the BA-2motif dataset, being synthetic,this is done the same way as for the node datasets.
The only difference being that the process is repeated for every graphin the dataset.
As there are no motifs deﬁned for the Mutagenicity dataset, the ground-truth labels can not be deﬁnedbased on them.
Instead, for this dataset edge labels are used, as provided by the original dataset repository 2 .
AUC score With the explanation mask provided by the PGExplainer and the ground-truths deﬁned as above, the AUCscore can be computed.
However, there are a few important notes to consider when computing the AUC score.
First, forthe node classiﬁcation datasets, the explanation mask is only determined for a 3-hop graph around each node.
This isdone because the GCN model only contains three layers.
Second, only the nodes that are part of a motif are used in the
AUC computation.
This is because there is no real deﬁnition of ground-truth for the nodes outside the motifs.
Thisevaluation design choice is further discussed in Sec. 6.
Third, for the BA-2Motif dataset only a subset of the graphs isused to determine the AUC score, this is done to reduce computation time.
Lastly, for the Mutagenicity dataset only themutagenic graphs have a valid ground-truth interpretation.
Hence, the AUC is determine using only these graphs.
Ofthese four considerations, only the last is mentioned in the original paper.
Comparison The authors compare their method against four baselines; a gradient-based model (GRAD) [9], a graphattention network (ATT) [8] and Gradient [7].
With the exception of the scores presented for the graph-classiﬁcationdatasets, the scores presented are reused from the PGExplainer paper (see Table 4).
In communication with the authors,it was mentioned that the reimplementation of these explainers by the authors had resulted in lackluster results.
For thisreason the decision was made to use the original scores by the original authors.
For our replication of the evaluation we focus our comparison on the GNNExplainer.
This method is the most similarand was a major inspiration for the PGExplainer.
In contrast the the original evaluation, we do perform the comparisonusing our own re-implementation of the GNNExplainer.
Our re-implementation of this method is largely inspired bythe implementation in the PyTorch Geometric library.
The main difference is that our re-implementation is adapted toalso work with graph-classiﬁcation datasets.
This is not possible with the plain PyTorch Geometric implementation.
4.3.2 Qualitative evaluation
[utils/plotting.py]
In order to obtain a visualisation of the chosen sub-graph the system takes as input the ground truth labels and themask provided by the Explainer.
Given the mask, two thresholds are calculated, one for importance to the explanationand one to determine which other elements to plot for the sub-graph.
Then, using these thresholds all nodes thathave an interesting enough weight are selected.
Following this, only nodes that are in a direct sub-graph together thenode-to-be-explained are selected.
When drawing the explanation for the graph classiﬁcation this sub-graph is selectedusing the top- k edges.
The original evaluation sets k to be the number of edges in the deﬁning motif for the synthetic
[Table]
Tree
[Table]
Grid
[Table]
Mutagenicity
[Table]
4.3.3 Efﬁciency evaluation
[evaluation/EfficiencyEvaluation.py]
In the paper, the authors only compare the efﬁciency of their PGExplainer to the GNNExplainer.
Unfortunately, we wereunable to extract the exact method for doing so from both the paper and the provided codebase.
Our implementation istherefore mainly our own design.
We compute the inference time as the average over ten runs.
During each run we measure the times it takes to explainall samples that are also used for the quantitative evaluation.
This time is divided by the number of samples explainedto get the ﬁnal inference time per sample in milliseconds.
Note that, similar to the paper, for the evaluation of the
PGExplainer only the time to explain each sample is considered.
On the other hand, for the GNNExplainer the timerequired to train the explainer is also taken into account because it has to be retrained for each sample.
Results
5.1 Model training
[experiment_models_training.ipynb]
In Tab. 1 the ﬁnal accuracies for all 6 trained models are provided.
Note that these are the accuracies of the modelsthat will be explained by the two explainers, not the explanation accuracy of the explainers themselves.
For most ofthe models, using the conﬁgurations found in the code, we achieve results comparable to the results presented in thepaper.
The two exceptions being the BA-Community and the Mutagenicity models.
Both of these score lower then theiroriginal counterpart.
Logically this difference could be contributed to the difference in the use of batch normalization.
Where the originalmodel in the PGExplainer paper did use batch normalization where we do not.
However, as the results presented in
Tab. 5 show, replication with the original batch normalization yields the same reduced accuracies.
We hypothesise thattherefore the difference might be the result of an undocumented use of weight regularization.
We observed that in theoriginal training script the conﬁguration exist to use L2-weight regularization, but it is not used.
5.1.1 Replicability study
[experiment_replication.ipynb]
Quantitative Quantitatively there is a large difference in the reported AUC scores and what we were able to achieveusing the speciﬁed conﬁgurations for the PGExplainer.
Only for BA-Shapes a AUC equal or higher then the presented
AUC score was observed.
However, BA-Shapes did require some minor modiﬁcations to the conﬁgurations to get it towork.
With the temperature parameter set as originally presented in the code, the evaluation crashed.
Only when thetemperature was changed to the conﬁguration as presented in the paper we were able to run the evaluation.
Similarly,with the conﬁguration as described in the code, the PGExplainer produces the opposite of the expected result forthe BA-2Motifs dataset.
This is reﬂected both quantitatively and qualitatively.
However, it should be noted that thesame drop in AUC score between our implementation and the one originally reported score can also be seen for the
GNNExplainer.
Due to this, the reported improvement of the PGExplainer over the GNNExplainer remains valid.
We believe that the difference seen between the AUC scores originally reported for the two explainers and what weobserved during our reproduction might be the result of the undocumented effect of the entropy/size regularizations andused temperature.
Based on empirical observations we found that the ﬁnal AUC score is highly dependent on thesethree hyperparameters.
A small follow-up ablation study presented in Tab.
5.1.1 conﬁrms this.
Qualitative The replicated qualitative evaluation is very similar to the original results.
PGExplainer is very capableof ﬁnding the motifs in the graphs and highlighting their edges.
The same holds for the GNNExplainer.
[Table] [Table]
[experiment_ablation.ipynb]
The main observed difference is the Mutagenicity dataset.
In our replication, only two edges are darkened in contrast tothe ten edges darkened in the original paper.
However, this difference is created artiﬁcially by a difference in the kvalue reported in the paper and used in the code.
While this difference therefore does not tell us anything about thequality of the explanation, it does show the importance of the k hyperparameter.
This is further discussed in the Sec. 6.
Efﬁciency In terms of efﬁciency, the reimplemention results are consistent with the claims of the authors.
The use ofdifferent frameworks between the original implementation and our reimplementation makes a direct comparison of theresult is ill advised, but the speedup between the PGExplainer and the GNNExplainer is consistent.
Ground truth explanations for Graph Explanations
For the evaluation of the PGExplainer the authors made use of predeﬁned ground-truth explanations.
These explanationsare made possible by the use of synthetic datasets, generated based on the notion of motifs.
In this section we expresssome concerns with regards to the use of motifs for generating ground-truth explanations.
No ground-truth outside motif
In the case of the node classiﬁcation datasets the deﬁnition of the ground-truthexplanation is only valid for a small number of nodes within a graph; those within a motif.
In essence, for nodes outsidethe motifs, the ground-truth explanation is an empty graph—i.e. all surrounding edges have to be excluded from theexplanation to achieve the maximum score.
The same is true for non-mutagenic graphs in the Mutagenicity dataset.
This is incompatible with the PGExplainers approach to determine its explanation.
An empty graph can never producethe same explanation as the original graph, hence it will never be the explanation provided by the PGExplainer.
The authors overcome this issue by excluding all nodes outside the motifs from their quantitative evaluation.
However,this reduces the explanation task of the node classiﬁcation datasets to a much simpler problem.
For nodes outside themotif, the explanation has to be based on the absence instead of presence of edges.
Solving these issues satisfactorilywould require a new deﬁnition for the ground-truths for graph datasets.
For example, in the case of the tree-cycledataset, one could deﬁne the ground-truth of a node outside a motif to be the entire 7-hop subgraph as this would be theminimal number of steps to take before one can conclude that no cycles have been formed.
We, however, believe this tobe outside the scope of this replication.
Qualitative evaluation dependent on knowing size of motif The PGExplainer gives as output a mask that describesfor each edge in the graph the probability of it being important for the models classiﬁcation decision.
To turn this intoa visualizable explanations the top- k edges are selected from each mask, i.e only the k edges that have the highestinﬂuence on the models classiﬁcation decision are considered part of the explanation.
As a result, k is a crucialhyperparameter for obtaining a visual explanation.
If k is set too high, the explanation could contain edges that actuallyonly contribute to the ﬁnal decision marginally.
If k is set too low, the explanation could be missing important parts ofthe graph.
This difference in visual explanation quality was also empirically observed in the difference between theoriginal and our explanations for the Mutagenicity dataset.
As mentioned in the experimental setup of the qualitative evaluation, the authors, and preceding works, set the value ofk in the evaluation based on the amount of edges in the deﬁning motif.
However, this is not a possibility outside ofthe synthetic evaluation datasets.
Hence, for real world applicability of the proposed explanation method a differentapproach has to be found to ﬁnd k .
For this reason, we believe that evaluating the quality of the explanations basedusing k preset to the number of edges in the synthetic dataset is an aspect to reconsider.
In essence, both the k -parameter and the earlier mentioned number of edges selected for the ground-truth can beconsidered as hyperparameters for the evaluation pipeline.
By selecting a speciﬁc value for these parameters theevaluation can become biased towards assigning high credibility to explanations that have a speciﬁc characteristic.
Byperforming an extensive search over these hyperparameters the results of the explanation evaluation can potentially beimproved.
In Sec.
C of the appendix we present a short study on how these hyperparameters can inﬂuence the ﬁnalresults of the evaluation.
Conclusion
In this work, we have presented a replication of the paper Parameterized Explainer for Graph Neural Network .
Thereplication experiments have lead us to a number of conclusions.
First, based on the paper alone, it is difﬁcult toreplicate the presented results.
The main contributing factor is the discrepancy between the provided details in the paperand those in the codebase.
Based on communication with the authors, we conclude that the hyperparameter settingspresented in the paper are oversimpliﬁed.
For the method to work, more hyperparameter tuning is needed then the papersuggests.
This is validated by our ablation study.
Second, even with the provided codebase, replication of the presented results is still arduous.
With the conﬁgurationspulled from the codebase used in our re-evaluation, we still found lackluster results for a number of the datasets.
Weaccredit this problem mainly to the structure of the codebase itself.
The code is overly convoluted with the experimentconﬁgurations being overridden in numerous locations.
Due to this, it is unclear if the conﬁgurations we found in thecodebase are those that generated the results presented in the original paper.
Lastly, as discussed in Sec. 6, we are uncertain if the evaluation based on synthetic datasets as used in the evaluation isvalid.
However, we can not contribute this issue to only the authors’ paper as it is also used in other graph explanationpapers, including the GNNExplainer.
In addition to showing that these issues exists, our extended evaluation presentedin appendix Sec C showed that it is not trivial to solve them based on the current deﬁnition of a ground-truth explanationfor motif graphs.
Rethinking the evaluation for Graph Neural Networks Explainers is therefore important future work.
data and results PGExplainer
B Direct replication of PGExplainer with BatchNorm activated model
Here we present a replication experiment similar to the one presented in our replication work.
However, as discussedin in the main work, the models used in the original paper contained two batch-normalization layers.
These layerswere incorrectly kept in training mode during evaluation.
In the replication results presented here, the same batchnormalization setup was used for the node-classiﬁcation models.
B.1 Results
Model training Tab. 5 shows that the accuracies of the models trained using the batch normalization are very similarto those used in the main paper.
The BA-Community dataset still shows the same issues with overﬁtting as are discussedin the main paper.
Quantitative Quantitatively there is a signiﬁcant difference between the explanations of the PGExplainer for modelstrained with or without batch normalization.
However, the main conclusion based on these results remain the same.
The
[Table] [Table]
[Table]
experiments show that using the conﬁguration provided in the codebase it is not possible to directly replicatethe results presented in the paper.
Qualitative No consistent signiﬁcant difference in the visualized explanations can be observed between the twoexplained models.
Efﬁciency The time required to explain the classiﬁcation decision of a single node in the graph is consistent betweenthe models trained with and without batch normalization.
C Extended replication
In this extend replication we perform a simple experiment considering the issues raised in Sec. 3.
Speciﬁcally, we redothe quantitative evaluation of the synthetic node-classiﬁcation using all test nodes instead of only those located in amotif.
The model used for the explanation and the deﬁnition of the ground truth remains the same.
[Table]
C.1 Results
Quantitatively the PGExplainer scores signiﬁcantly worse in the extended replication than during the original replication(see Tab. 7).
This is a direct result of performing the evaluation over the entire test set instead of only the nodes within amotif.
The ground-truth for nodes outside the motif and the method used by both the GNNExplainer by PGExplainerare simply incompatible.
Nevertheless, the improvement claimed by the authors of the PGExplainer over the GNNExplainer is still visible.
Considering all datasets, the PGExplainer consistently outperforms the GNNExplainer by a signiﬁcant margin.
D Conﬁguration
Conﬁguration ﬁles are used to provide a stable, ﬂexible and reproducible way to run the experiments.
D.1 Model conﬁguration ﬁles
The ﬁrst type of conﬁguration is the model conﬁguration as seen in Fig.
1, which contains (from top to bottom) theparameters required for training a GNN model.
dataset assigns which dataset the model has to train on, paper deﬁneswhich paper the model is build on (either PG, GNN or TAG), lr is the learning rate, epochs is the amount of epochsused for training and clip_max is the parameter to which determines at what point the gradient is clipped.
Additionallythe ﬁle includes early_stopping which deﬁnes the amount of epochs with no improvement that are required to enactearly stopping of training, seed which deﬁnes the seed used for training and eval_enabled which determines whetherthe model uses it’s eval mode.
D.2 Explainer conﬁguration ﬁles
[Figure]
The second type of conﬁguration is the explainer conﬁguration an example of which is shown in Fig.
2, these conﬁguration ﬁles contain all parameters required to train an explainer and perform the the replication experiment using them.
It includes the following parameters: dataset deﬁnes the dataset that the model that is to be explained is trained on.model is the type of model that has to be explainer, explainer is the implementation of the explainer (either PG or
GNN).
The conﬁguration also contains the learning rate and number of training epochs ( lr and epochs respectively).
As well as sample_bias which determines the sample bias, the parameters reg_size and reg_ent that determine thesize loss and entropy loss coefﬁcients respectively, the temperatures, seeds the seeds used for training, eval_enabled if the model uses evaluation mode and thres_snip and thres_min which deﬁne the thresholds for the interesting andsub-graph edges related to the drawing of the result explanations.