From: ICLR.cc/2023/Conference/Program_Chairs
title: Paper Decision
decision: Reject
metareview:_summary,_strengths_and_weaknesses: This paper proposes MotifExplainer, a method that identifies important motifs - recurrent and statistically significant patterns in graphs - to use as explanations for a Graph Neural Network (GNN) prediction task.  It employs an attention-based method to identify the most important motifs as explanations. It encodes a motif embedding of the extracted motifs using a pre-trained GNN. These motif embeddings serve as keys to the graph embedding (query) in the attention mechanism applied to learn attention weights by minimizing the loss between the prediction of the attention model and the original predictions. Empirical studies on both synthetic and real-world datasets demonstrate the effectiveness of the proposed method. However, application for this method on other domain would be limited owing to the lack of the domain knowledge. The reviewers raised concerns over the method’s dependence on the quality of motif extraction rules, high computational complexity, dependence on domain knowledge that limits generalizability, limited technical contribution and novelty, lacking a sufficient description/visualization of how much the extractions based on the rules vary from the ground truth. Additionally, one reviewer pointed to the need that some evaluation (whether quantitative or qualitative) without using those black-box module is essential. Another reviewer pointed to the need to experiment MotifExplainer on different GNN architectures. The authors addressed some of those comments. The AC sympathize with the reviewers on the unaddressed comments. Overall, a more comprehensive revision of the paper addressing all the mentioned issues is still needed to improve the paper and pass the bar for acceptance. 
justification_for_why_not_higher_score: The reviewers raised concerns over the method’s dependence on the quality of motif extraction rules, high computational complexity, dependence on domain knowledge that limits generalizability, limited technical contribution and novelty, lacking a sufficient description/visualization of how much the extractions based on the rules vary from the ground truth. Additionally, one reviewer pointed to the need that some evaluation (whether quantitative or qualitative) without using those black-box module is essential. Another reviewer pointed to the need to experiment MotifExplainer on different GNN architectures. The authors addressed some of those comments. The AC sympathize with the reviewers on the unaddressed comments. Overall, a more comprehensive revision of the paper addressing all the mentioned issues is still needed to improve the paper and pass the bar for acceptance. 
justification_for_why_not_lower_score: N/A

From: ICLR.cc/2023/Conference/Paper1107/Authors
title: Response to Reviewer b3v8
comment: We thank you for your additional comments.

1. We do not think the time complexity of motif extraction is a big problem in practice. For node classification tasks, we extract motifs on a computational graph. For graph classification tasks, most datasets have a relatively small number of nodes. From the efficiency study in section 4.6, our method has much less training and inference time than other competitors.

2. This training part is only for motif selection, and dataset split will not affect method evaluation.

3. We will revise these issues in the final version of our paper.

From: ICLR.cc/2023/Conference/Paper1107/Reviewer_b3v8
title: Acknowledgement of the authors' response
comment: Thank the authors for the response. While some explanations have addressed my questions, some concerns still remain: 
- Based on the given time complexity, one major concern is that the time complexity of motif extraction is high (e.g. $\mathcal{O}(n^3)$), which is prohibitive for large graphs. 
- The other major concern is the dataset split, the authors used the whole dataset for training, which is not reasonable as the test set cannot be touched and leaked during training.
- The motif extraction rule: the authors responded by using the general rule for all datasets in experiments, however, Section 3.2 states that "For molecule datasets, we can use sophisticated decomposition methods like RECAP and BRICS algorithms to extract motifs". Please clarify the discrepancy here and in the revised paper. 
- label $y$ is vaguely defined: predicted labels and ground truth labels are totally two different concepts. The author should make it clear in the paper, for example, in Section 2 and algorithm 1, the author should clearly define Y as the predicted label, instead of generally saying "label".
- The definition of fidelity is still wrong: based on the authors' response, the feature extractor $\Psi$ should be wrapped by the classifier $\xi$ to make the output probability, which is not corrected in the revised paper.

From: ICLR.cc/2023/Conference/Paper1107/Reviewer_ayU3
title: Response
comment: Thank you and I have updated my review.

From: ICLR.cc/2023/Conference/Paper1107/Authors
title: Respond to Reviewer ayU3
comment: Thank you for your kind review.

We added more clarification in sections 3.4/3.5 with red color in the latest version of our paper.

From: ICLR.cc/2023/Conference/Paper1107/Reviewer_ayU3
title: Feedback to author response
comment: Thank you for your efforts in addressing my concerns. Most of my concerns are addressed. But I would expect to see an updated version of the manuscript that better clarifies those points. If I miss any part, please kindly point it out. And I am happy to raise my score with an updated manuscript.

From: ICLR.cc/2023/Conference/Paper1107/Authors
title: Respond to Reviewer M4gL
comment: We thank the reviewer for the useful comments. We will address concerns and questions individually.

1. Many works of literature in other fields (graph feature learning) have shown that cycles and edges can be used as motifs. [1, 2] In the ablation study, there is a misunderstanding here. Our proposed extraction method has been used to generate motifs in all experiments. The results demonstrate the effectiveness of our proposed motif extraction rule.

2. Thank you for pointing out this. There are only two reinforcement learning-based GNN explainers applying attention to their models. [3, 4] We added them to the related work section in the latest version of our paper.

3. To make a fair comparison, we follow the same settings from previous works. [5, 6, 7] Most GNN explainers (including our proposed method) are model-agnostic, which means they do not rely on the backbone models. That's why most of the literature uses vanilla GNN with three layers as a target GNN.

[1]. Bouritsas, Giorgos, et al. "Improving graph neural network expressivity via subgraph isomorphism counting." IEEE Transactions on Pattern Analysis and Machine Intelligence (2022).

[2]. Yu, Zhaoning, and Hongyang Gao. "Molecular Representation Learning via Heterogeneous Motif Graph Neural Networks." International Conference on Machine Learning. PMLR, 2022.

[3]. Shan, Caihua, et al. "Reinforcement learning enhanced explainer for graph neural networks." Advances in Neural Information Processing Systems 34 (2021): 22523-22533.

[4]. Wang, Xiang, et al. "Reinforced Causal Explainer for Graph Neural Networks." IEEE Transactions on Pattern Analysis and Machine Intelligence (2022).

[5]. Lucic, Ana, et al. "Cf-gnnexplainer: Counterfactual explanations for graph neural networks." International Conference on Artificial Intelligence and Statistics. PMLR, 2022.

[6]. Ying, Zhitao, et al. "Gnnexplainer: Generating explanations for graph neural networks." Advances in neural information processing systems 32 (2019).

[7]. Luo, Dongsheng, et al. "Parameterized explainer for graph neural network." Advances in neural information processing systems 33 (2020): 19620-19631.

From: ICLR.cc/2023/Conference/Paper1107/Authors
title: Respond to Reviewer ayU3
comment: We thank the reviewer for the useful comments. We will address concerns and questions individually.

1. For a node classification task, to keep the connectivity between a target node and a motif, we are keeping the shortest path between each node in the motif and the target node in an explanation graph. To reduce the impact of nodes on the path, we set irrelevant nodes' features to zero.

2. We are not using attention scores in the evaluation step, which means no matter whether attention is a black-box model, the evaluation metric can truly represent the expressiveness of our method. And attention is only a tool for selecting important motifs. Any other methods that can select motifs can replace attention in our model. We claim that whether an explanation is better or not should depend on the evaluation, and in sections 4.2/4.3, the experimental results have demonstrated the effectiveness of our model.

3. The main novelty of our work is how to conduct GNN explanation based on motifs. Attention only serves as a tool to identify important motifs. Any other tools that can identify relevances between two feature vectors can be applied in our model. In addition, since all motifs in the dataset share the same query, the explanations can provide a global understanding of the trained GNN. Directly calculating the contribution of each motif in an instance may not have this advantage.

From: ICLR.cc/2023/Conference/Paper1107/Authors
title: Respond to Reviewer EDg3 (Part 2)
comment: 6. We completely understand your concern about our motif-based explainer. Here, we want to demonstrate the "human explainable" of graphs from a downstream perspective. In particular, the motif-based explanations will enable broader and more effective applications of GNN explainers. When explaining a GNN model, the first objective is identifying the most relevant inputs leading to a specific prediction. We have empirically shown that motif-based explanations lead to better quantitative and qualitative performances. At the same time, the identified relevant motifs will be more useful for downstream applications. For example, fragment-based drug discovery (FBDD) is a powerful method to develop potent small-molecule compounds starting from fragments [6]. FBDD is based on fragment libraries, including fragments or motifs relevant to the target property. Such libraries usually depend on domain experts. However, by using a motif-based GNN explainer, we can identify relevant fragments or motifs. The motifs are easy to operate when growing drug-like lead compounds. In practice, some fragment libraries only include 800 fragments [7], which means they care about repeated and relevant fragments instead of free-form fragments. In this case, the motif explanations generated by our explainer can be easily used in FBDD. In contrast, the free-form explanations, such as four edge house structure, will be hard to apply in such kinds of downstream applications. 

6. In our experimental study, we also study the importance of motifs in the ablation study section (4.5). In particular, we build an attnExplainer that uses edges as basic explanation units and apply the same attention model to select relevant edges as explanations. The only difference between our motifExplainer and attnExplainer is motifExplainer uses motifs as basic units while attnExplainer uses edges. The results (table 3 and table 6 in Appendix E) show that using motifs as basic explanation units is better than using edges. Also, the experimental results in Table 1 demonstrate that our motif-based method is better than edge/nodes-based baselines.

[1]. Yuan, Hao, et al. "Explainability in graph neural networks: A taxonomic survey." IEEE Transactions on Pattern Analysis and Machine Intelligence (2022).

[2]. Ying, Zhitao, et al. "Gnnexplainer: Generating explanations for graph neural networks." Advances in neural information processing systems 32 (2019).

[3]. Luo, Dongsheng, et al. "Parameterized explainer for graph neural network." Advances in neural information processing systems 33 (2020): 19620-19631.

[4]. Yuan, Hao, et al. "On explainability of graph neural networks via subgraph explorations." International Conference on Machine Learning. PMLR, 2021.

[5]. Wang, Xiang, et al. "Towards multi-grained explainability for graph neural networks." Advances in Neural Information Processing Systems 34 (2021): 18446-18458.

[6]. Li, Qingxin. ``Application of fragment-based drug discovery to versatile targets.'' Frontiers in molecular biosciences 7 (2020): 180.

[7]. Sabbah, M., Mendes, V., Vistal, R. G., Dias, D. M. G., Záhorszká, M., Mikušová, K., et al. (2020). Fragment-based design of mycobacterium tuberculosis InhA inhibitors. J. Med. Chem. 63, 4749–4761. doi: 10.1021/acs.jmedchem.0c00007

From: ICLR.cc/2023/Conference/Paper1107/Authors
title: Respond to Reviewer EDg3 (Part 1)
comment: We thank the reviewer for the useful comments. We will address concerns and questions individually.

1. Thank you for pointing this out. There is a misunderstanding here. For real-world datasets, we do not have ground truth. For the MUTAG dataset, we can leverage some domain knowledge to analyze the generated explanations. NH2 and NO2 groups can be seen as ground truth, while N-H and N-O bonds are regarded as motifs. However, for any other datasets, we do not have that domain knowledge. That is why we need the Fidelity metric to evaluate an explanation. In addition, we want to mention that our method identifies important basic motif units and is capable of composing a new, meaningful substructure as an explanation.

2. There is a misunderstanding here. Our MotifExplainer is not similar to GraphLime. GraphLime studies the selection of different node features for node classification tasks and uses selected node features as an explanation of an original GNN prediction. They ignore graph structures such as nodes, edges, and motifs, which is different from our proposed method. In addition, GraphLime is a kind of surrogate method, and our MotifExplainer is a perturbation-based method. [1]

3. There is a misunderstanding here. Our proposed method does not require K to fix an upper bound of the number of selected nodes. To make a fair comparison, we set a K for each baseline to control the searching space. Following previous works [2,3], we choose K=5 for two synthetic datasets. For real-world datasets, we control K to make sure all baselines have similar Sparsity. Due to space limitations, we reported more results with different Sparsity in Appendix E. And from the results, our method can consistently outperform other baselines.

4. Thank you for pointing this out. There is a misunderstanding here. Following previous works [2, 3], we use the Accuracy metric to evaluate models on synthesis datasets (BA-2Motif \& BA-Shape) with ground truth. We use the Fidelity metric on other real-world datasets. We have added some explanation in the title to avoid confusion.

5. Our method does not need a K to make an explanation. For real-world datasets, the baselines are setting different K values to reach a similar Sparsity level. Following previous works [2, 3, 4, 5], we compare our method with other baselines under similar Sparsity levels.

From: ICLR.cc/2023/Conference/Paper1107/Authors
title: Respond to Reviewer U3mX
comment: We thank the reviewer for the useful comments. We will address concerns and questions individually.

1. We would like to unpack the reviewer’s concerns into three main points. First, we propose a general motif extraction rule in Appendix B for datasets that do not have sophisticated extraction methods. Second, we claim that regarding meaningful motifs as a basic unit is a better way to explain the model, as they carry additional knowledge from already existing evidence. This is exactly the contribution and novelty of our proposed method. Finally, there is a misunderstanding here. Our method identifies important basic motif units and is capable of composing a new, meaningful substructure as an explanation. This has been demonstrated in Section 4.2. In the experiment on the MUTAG dataset, we only extract a $N-O$ bond as a motif, and our explainer can accurately identify the $NO_2$ group as an explanation by combing two $N-O$ bonds. Thus, our method can give novel insights and observations about the given graph.

2. We want to mention that the main novelty of our work is how to conduct GNN explanations based on motifs. Attention only serves as a tool to identify important motifs. Any other tools that can identify relevances between two feature vectors can be applied in our model. Also, node classification and graph classification tasks have different computational graphs, which means motif extraction and motif embedding generation are different in these tasks. It is hard to clearly introduce them together.

3. To keep the completeness of a motif, we keep the same edge in multiple motifs. These duplicate edges will not affect the results because we only select important motifs. In the evaluation step, there will be no existing duplicate edges in an explanation.

4. The complexity of the proposed extraction rule depends on the cycle basis generation algorithm. Currently, the complexity of the cycle basis generation algorithm in our algorithm is $O(n^3)$ [1]. Thus, the complexity of our extraction method is $O(n^3)$. We have already included motif extraction time in the study of computational cost. We report the actual time for extracting motifs on five real-world datasets below. We can see that our motif extraction rule is fast on all five datasets.

 | Dataset | MUTAG | NCI1 | PROTEINS | PTC | IMDB |
 |---------|-------|------|----------|-----|------|
 | Time(s) | 103   | 90   | 19       | 0.9 | 33   |

5. For a node classification task, to keep the connectivity between a target node and a motif, we keep the shortest path between each motif node and the target node in an explanation graph. To reduce the impact of nodes on the path, we set irrelevant nodes' features to zero.

6. We initial irrelevant nodes with zero features.

7. For reproducibility, we have already included codes in the supplementary material.

[1]. Paton, Keith. "An algorithm for finding a fundamental set of cycles of a graph." Communications of the ACM 12.9 (1969): 514-518.

From: ICLR.cc/2023/Conference/Paper1107/Authors
title: Respond to Reviewer b3v8
comment: We thank the reviewer for the useful comments. We will address concerns and questions individually.

1. There is a misunderstanding here. We have already used the proposed general motif extraction rule in the experiments. The results demonstrate that our proposed extraction method works for all fields of datasets.

2. The complexity of the proposed motif extraction rule depends on the complexity of the cycle basis generation algorithm. Currently, the algorithm applied is $O(n^3)$ [1]. Thus, the complexity of our extraction method is $O(n^3)$. The actual time for extracting motifs on five real-world datasets is like below.

| Dataset | MUTAG | NCI1 | PROTEINS | PTC | IMDB |
|---------|-------|------|----------|-----|------|
| Time(s) | 103   | 90   | 19       | 0.9 | 33   |

3. There is a misunderstanding here. In figure 1, y is the predicted label of an original graph, which can be seen as a ground truth label of the graph.

4. The size of motif lists for five real-world datasets are like below. We want to mention that a motif may appear multiple times in a graph. We could not get the specific number of the candidate sets of SubgraphX. However, our method needs less time to generate and select subgraphs/motifs in terms of efficiency studies.

 | Dataset | MUTAG | NCI1 | PROTEINS | PTC | IMDB |
 |---------|-------|------|----------|-----|------|
 |   Size  | 148   | 296   | 1584    | 55  | 377  |

5. In our reported performance, we are using the whole dataset for training. We tested different data splits (0.8/0.2, 0.9/0.1) in the experiments. The results are quite similar, and using the whole dataset for training can give us a slightly better result.

6. It is based on probability, and subscript $y_i$ means we choose the probability that the corresponding label is the ground truth label $y_i$. You can refer to equation (4) in [2].

7. Thank you for pointing out those typos. We have revised it in the latest version of our paper.

[1]. Paton, Keith. "An algorithm for finding a fundamental set of cycles of a graph." Communications of the ACM 12.9 (1969): 514-518.

[2]. Yuan, Hao, et al. "Explainability in graph neural networks: A taxonomic survey." IEEE Transactions on Pattern Analysis and Machine Intelligence (2022).

From: ICLR.cc/2023/Conference/Paper1107/Reviewer_b3v8
confidence: 4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
summary_of_the_paper: This paper for the first time adopts motif extraction as a pre-process step to generate the candidate set for explanation, and employs an attention-based method to identify the most important motifs as explanations.

strength_and_weaknesses: **Strength**
- Using motifs as explanations is a new and motivating idea
- The proposed algorithm is simple and effective
- The writing is easy to follow

**Weaknesses**
- The method heavily relies on the quality of motif extraction rules, whose complexity should also be discussed
- Some details need to be explained as commented in my summary

clarity,_quality,_novelty_and_reproducibility: The writing and presentation is in general clear and easy to follow. The novelty lies in the using motifs to provide explanations, which triggers a simple (with limited technical contribution) yet effective method. The codes are provided for reproducibility.

summary_of_the_review: This paper provides a new way to explain GNNs via motifs. The method is simple, practical and effective, while the idea is straight-forward with limited technical contribution. I also have the following questions.
- The method requires a motif extraction step, which heavily relies on domain knowledge. As a fair comparison, what if directly applying the general motif extraction algorithm for all datasets, including the molecule datasets?
- The complexity of the motif extraction algorithm should be discussed, especially when the graph is large.
- There is a contradiction in the method description: $y$ is the predicted label based on Figure 1, but in Algorithm 1 it represents the ground truth label of the graph. Which one is adopted in the actual design?
- How large is the motif list for different datasets, compared with the candidate set of SubgraphX?
- How are the datasets split for training and testing?
- The fidelity metric defined in Eq. (10) is unclear: the output of the feature extractor $\Psi$ should be the embedding, then what is the meaning of the subscript $y_i$? Presumably, it should be based on the prediction accuracy or probability (e.g. output of the classifier $\xi$) as in [1].
- The reported evaluation metric in Table 1 should be explained in the title to avoid confusion: for BA-2Motif and BA-Shape datasets, the metric is accuracy.
- In section 2, “donate” -> “denote”

[1] Hao Yuan, Haiyang Yu, Shurui Gui, and Shuiwang Ji. Explainability in graph neural networks: A taxonomic survey. arXiv preprint arXiv:2012.15445, 2020b.


correctness: 3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.
technical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
empirical_novelty_and_significance: 3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.
flag_for_ethics_review: NO.
recommendation: 6: marginally above the acceptance threshold

From: ICLR.cc/2023/Conference/Paper1107/Reviewer_U3mX
confidence: 3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
summary_of_the_paper: This paper aims to explain predictions of graph neural networks (GNNs). Existing GNN explanation methods identify important edges, nodes, or arbitrary substructures but fail to consider recurrent and interpretable substructures. This work proposes MotifExplainer to explain GNNs by identifying important motifs. Empirical studies on both synthetic and real-world datasets demonstrate the effectiveness of the proposed method.
strength_and_weaknesses: Strengths
1. Using the attention mechanism for motif-based explanation seems like an efficient and reasonable approach.
2. Diverse experiments are done to demonstrate the superiority of the proposed approach.
3. This paper is generally written well, and there is not much confusion in understanding it.

Weakesses
1. The proposed approach requires domain-specific motifs as its inputs. This makes the approach less generalizable, and its improved accuracy as the result of adding domain-specific knowledge into the algorithm, rather than the result of technical improvements. One important goal of GNN explanation is to find new, meaningful substructures from the graph which play a key role in the prediction, which give novel insights and observations about the given graph. This work does not have such an advantage.
2. The technical contribution is limited. Using attention for GNN explanation is a reasonable approach, but is not novel. This paper presents the approaches for node and graph classification in separate subsections, but they look very similar and can be presented together. Other parts of the proposed approach, such as motif extraction and embedding, also look trivial.
3. Many parts of the proposed approach are not clearly presented, even though they are the core parts of technical contributions. See the questions below.

Questions
1. Section 3.2: What happens if the same edge is contained in multiple motifs?
2. Section 3.2: What is the complexity and actual time for extracting motifs?
3. Section 3.3: How do we keep edges from the target node to the motif?
4. Section 3.3: How do we mask features of irrelevant nodes?
clarity,_quality,_novelty_and_reproducibility: - Clarity: This paper is generally written well, except that the technical details of the proposed approach are missing.
- Quality: The technical contributions seem to be limited.
- Novelty: This paper solves an interesting problem which is to consider domain-specific motifs in GNN explanation.
- Reproducibility: Code is not given, but sufficient information for reproducibility is presented.

summary_of_the_review: It is a reasonable approach to use known (or domain-specific) motifs to better explain the decisions of GNNs. However, I think the technical contributions of this paper are limited in two ways: a) the proposed approach does not work without pre-defined motifs, which is common in real-world cases, and b) the proposed approach seems to be a combination of existing techniques.
correctness: 3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.
technical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
empirical_novelty_and_significance: 3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.
flag_for_ethics_review: NO.
recommendation: 5: marginally below the acceptance threshold

From: ICLR.cc/2023/Conference/Paper1107/Reviewer_EDg3
confidence: 5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.
summary_of_the_paper: This paper proposes MotifExplainer, a method that identifies important motifs - recurrent and statistically significant patterns in graphs - to use as explanations for a Graph Neural Network (GNN) prediction task. The method first extracts motifs in a graph using domain-specific motif extraction rules (specified in the paper). It encodes a motif embedding of the extracted motifs using a pre-trained GNN. These motif embeddings serve as keys to the graph embedding (query) in the attention mechanism applied to learn attention weights by minimizing the loss between the prediction of the attention model and the original predictions.  Experimental results on seven datasets show that MotifExplainer can highlight relevant motifs in the graphs (quantitatively and qualitatively). 

strength_and_weaknesses: Strengths:
- The presented idea is interesting and well-explained. A simpler framework - motif extraction+attention - is trained to mimic the performance of a complicated GNN model. The learned attention weights highlight the relevant extracted motif. 

- The paper provides motif extraction rules for different application domains.

- The paper presents qualitative and quantitative results on synthetic and real-world datasets and reports state-of-the-art performance. 

- The reported computational time for the proposed method is lower than existing models.
 
Weaknesses:
The main weakness of the paper is in the experimental design, as some of the choices are not well defined. 
 
- First, given the importance of the motif extractor scheme for this work, the paper needs to provide a sufficient description (example visualizations) of how much the extractions based on the rules vary from the ground truth. Specifically, the line “note that neither NH2 nor NO2 is explicitly included in our motif extraction rules” is useful. However, how similar/different are these motifs to the construction, like a single input example?  
 
- While the included baselines are state-of-the-art methods, the presented framework seems to have some ideological similarities to GraphLIME [1] method. Is there a reason this was not included in the comparisons?
 
- All the methods (proposed and baselines) have hyperparameters that they can be quite sensitive to. Was sufficient hyperparameter tuning performed to select the best explanation? For example, why K=5 was chosen? If not, is that fair?
 
- My understanding is that fidelity scores should be low (less difference in prediction change when removing unimportant features). Why are the higher scores in Table 1 (for BA datasets) highlighted in bold?  

Minor points:

- The values for K are not specified in Table 6,7,8 in the appendix
 
- (This point  is on a broader level) The line - “a fundamental criterion for explanations is that they must be human explainable.” While I agree with this statement, it is unclear what "human explainable" means outside the context of image and text. If a GNN model can make an accurate prediction using 4 edges of the house motif instead of 5 - do we necessarily want the explainer method to highlight the full motif with 5 edges? I can see the possibility of a disconnect between what the model is learning and what the human expects the model to learn. Maybe the paper could clarify this.

clarity,_quality,_novelty_and_reproducibility: The motivation, explanation, and illustration of MotifExplainer components (Figure 1) are clear.
 
The experiment setup could be improved to support the claims of the paper better
 
The paper is the first to apply the attention mechanism to explain GNNs from a motif-level perspective and one of few methods to consider subgraphs to explain GNNs.
 
The description of the model is clear, but the paper lacks hyperparameter tuning and settings

summary_of_the_review: MotifExplainer uses domain-specific motif extraction rules and attention to identify importation motifs. Changing the motif extraction rules accordingly allows for easy application of the method to various domains. The experimental design requires more extensive descriptions (especially about the extracted motifs and design choices) to support the claim that MotifExplainer is a state-of-the-art explanation method. 
correctness: 2: Several of the paper’s claims are incorrect or not well-supported.
technical_novelty_and_significance: 3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.
empirical_novelty_and_significance: 3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.
flag_for_ethics_review: NO.
recommendation: 3: reject, not good enough

From: ICLR.cc/2023/Conference/Paper1107/Reviewer_ayU3
confidence: 4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
summary_of_the_paper: This paper studies explainability of graph neural networks (GNNs). The authors are particularly interested in using motifs to generate the explanantion, since motifs are more meaningful and statistically significant in many graph mining tasks. Based on this motivation, the authors propose MotifExplainer, which first extracts motifs with domain knowledge and then leverages attention mechanism to measure the contribution of each motif. And the motifs with the highest attention scores are used as the proxy to find the most important motifs. Experimental results show that the proposed MotifExplainer not only outperforms all baseline methods quantitatively and qualitatively, but also is the fastest method during inference.
strength_and_weaknesses: Strengths:
- Motif is more meaningful in many graph mining tasks, thus the motivation is clear and reasonable.
- The paper is overall well-written and easy to follow.
- Both quantitative and qualitative analysis demonstrate the efficacy of the proposed model.

Concerns/Questions:
- For a node classification task, suppose that the target node is 'far away' from the motif and is not directly connected to any node in the motif. How should we keep the connectivity between the target node and the motif?
- The goal of explainability is to open the black box of deep learning models. More importantly, there was a hot debate in NLP domain on whether we should trust attention as a proxy of explanation. I assume such problem could exist in GNN as well. As such, how should we trust the explanation generated by another black-box model (attention layer here)? 
- Why should we use attention score rather than directly using the contribution of each motif to find important subgraphs (i.e., the loss $f(y, y')$)?


========================
The author have addressed most of my concerns. I have raised my score. But in general, I still think some evaluation (whether quantitative or qualitative) without using those black-box module is essential.
clarity,_quality,_novelty_and_reproducibility: The paper is overall well-written and clear. The motivation is clear and the proposed method is novel.
summary_of_the_review: The paper is well motivated and easy to follow. My major concern on the technical side is the use of attention mechanism, which is a black box in its nature and controversial to be used as a proxy for finding explanation in NLP tasks. 
correctness: 3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.
technical_novelty_and_significance: 4: The contributions are significant, and do not exist in prior works.
empirical_novelty_and_significance: 3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.
flag_for_ethics_review: NO.
recommendation: 6: marginally above the acceptance threshold

From: ICLR.cc/2023/Conference/Paper1107/Reviewer_M4gL
confidence: 3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
summary_of_the_paper: This paper presented an explanation model on motif of the graph. The motif of choice is generated with domain knowledge. Following by an attention model, the attention weights is utilized for the explanation for the GNN from the motif. Compared with state of the art, the method showed high performance. The human readability of the motif is also a plus of the paper. However, application for this method on other domain would be limited owing to the lock of the domain knowledge.
strength_and_weaknesses: Strength:
1. This is the first method to introduce attention mechanism to explain the importance of the motif. It does showed certain level of improvement compared with previous methods.
2. The input of domain knowledge do boost the performance increase for the explanation.
3. They also introduce different scores to cover the explanation, for which their method showed good performance.  

Weakness:
1. Introducing domain knowledge generated motif would be great but also limited the applicability of this methods. It also introduced certain bias in the performance evaluation as other methods do not have the domain knowledge input. Even though this paper provided a way to generate motif. But only counting the cycle or edge, does not fit my impression of "motif" and do limited the power of the method. The ablation study also illustrate the important of the domain knowledge input. For that, I think it significantly hurt the novelty of this paper.
2. The method could also introduce some background of the attention mechanism in neural network explanation.
3. This model only performed the test on vanilla GNN with three layers. I think more experiment should be conducted. For example, different GNN architecture. Different hyperparameters like the number of layers, the width of the GNN. I would guess the attention mechanism would have high variance if the architecture is changed. 



 

 
clarity,_quality,_novelty_and_reproducibility: It is clear written. The novelty of the paper is limited by the generalization to other domains without domain knowledge. I would also recommend more experiments for the GNN of interest, i.e., hyperparameters, model architectures. 
summary_of_the_review: This paper presented the motif explanation and showed empirical performance increase over state of the art models. But some limitation also applied for the method to generalize to other domain. 
correctness: 3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.
technical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
empirical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
flag_for_ethics_review: NO.
recommendation: 5: marginally below the acceptance threshold