Paper: SGOOD: Substructure-enhanced Graph-Level Out-of-Distribution Detection (ICLR-2024)

Reviewer comment (Novelty):
The motivation to improve graph-level OOD detection by encoding more substructure information into graph representations is unclear.

Author response:
The motivation of encoding task-agnostic substructures to improve graph-level OOD detection is empirically justified in Table 1 of the paper. The intuition is that substructure differences commonly exist in real-world ID and OOD graphs, and naturally, if the learned representations can encode substructures, which means better differentiation between ID and OOD graphs, the performance of graph-level OOD detection could be improved.

We agree that Table 1 itself is not self-contained, due to the lack of space in introduction, and we have revised Section 1 in the latest version uploaded. Specifically, please refer to Table 2 in Section 4 for ID and OOD graph statistics. Table 1 shows the percentage of OOD test graphs containing substructures never appeared in ID training graphs in each dataset. These percentage values are high, higher than 44% in 4/6 datasets, indicating that many OOD graphs contain substructures rarely appear in ID graphs. This observation motivates our design to encode substructures to improve graph-level OOD detection. Note that the only purpose of this statistical finding is to motivate our idea, and it is not used in the training, since OOD graphs are only available at test time.

Reviewer comment (Theory):
The notion that encoding more substructure information into graph representations will enhance graph-level OOD detection faces skepticism. In practice, theoretically more powerful GNNs often under-perform their 1-WL equivalent counterparts across various graph datasets [1]. This is due to the fact that, in cases where node attributes can function as supplements to structural information, nearly all graphs can be differentiated by 1-WL equivalent GNNs. Substructures do not exist in isolation, and are accompanied by a lot of attribute information. Furthermore, these concerns are verified by the results presented in Table 7. Specifically, more powerful GNNs like NGNN and GNN-AK+ fail to outperform 1-WL equivalent GNNs SAG, TopK, and DiffPool in the graph-level OOD detection task.

Author response:
To answer **W2**, we echo your insightful comment in **W5**: "In [2], authors proposed that encoding the task-agnostic (e.g., graph classification task-agnostic) information into representations can improve the OOD detection task." Existing GNNs like NGNN and GNN-AK+ learns task-specific substructures. As mentioned in Section 1, these methods are trained with a focus on classification-related structures. *On the contrary, our method SGOOD preserves graph classification task-agnostic substructures*, so that the generated representations by SGOOD can better distinguish ID and OOD graphs at test time. The input substructures used in SGOOD are not restricted by ID graph labels, and these substructures are extracted by existing methods as a pre-step. This explains the performance improvements by SGOOD over NGNN and GNN-AK+ in Table 7. In order to make the paper complete, the analysis in Section 3.5 just serves as a complementary justification for SGOOD from the perspective of 1&2-WL test.

Reviewer comment (Theory):
This paper lacks a clear definition of the graph distribution, and it does not explore the factors contributing to the distribution differences between ID and OOD graphs. It places excessive emphasis on the influence of substructures in graph-level OOD detection while neglecting the discussion of node attributes. Two graphs with identical structures but distinct node features may exhibit entirely different distributions.

Author response:
Note that different datasets may have different ID and OOD graph distributions. Moreover, the distribution differences (i.e., distribution shifts) between ID and OOD graphs also vary on different datasets. A method dedicated to one distribution shift may not be generalizable to another kind of distribution shift. Therefore, by following the literature in OOD detection, we do not restrict a proposed method to a specific type of graph distributions or a specific type of distribution shifts. As shown in Table 2, for protein networks (ENZYMES) and social networks (IMDB-M, IMDB-B, REDDIT-12K), graph distributions are class distributions [3], and the distribution shifts of OOD graphs are unseen classes, i.e., a test graph is OOD if it belongs to an unseen class. For molecular graphs (BACE, BBBP, DrugOOD, HIV) in Table 2, we follow [4] and regard graph distributions as either scaffold distributions or protein target distributions. It means that OOD molecular graphs are the graphs with their scaffold or protein targets different from ID ones. More details can be found in Appendix B.1.

Then, as shown in Table 1, regardless of the types of distribution shifts, substructure differences between OOD and ID graphs commonly exist on various real datasets, and thus, we propose SGOOD that encodes substructures for graph-level OOD detection. We do not restrict SGOOD to a specific type of distribution shift. We agree that it is promising to explore the factors contributing to the distribution differences, and leave it as future work.

Moreover, we kindly clarify that SGOOD encodes node attributes together with graph structure using GNNs, as shown in Eq. (1). As a result, two graphs with identical structures, but distinct node features can also be differentiated by SGOOD. In this work, we focus on substructure patterns instead of node attributes.

Reviewer comment (Novelty):
The paper does not explicitly delineate the specific contributions of the proposed method, SGOOD, to the graph-level OOD detection task. Given the existence of many theoretically more powerful GNNs, it remains unclear why SGOOD is better than those GNNs in the graph-level OOD detection task. SGOOD appears to resemble a new GNN with powerful expressiveness rather than a specialized GNN that can identify OOD graphs.

Author response:
Then for **W4**, the main contributions of SGOOD are *how to effectively encode task-agnostic substructures into the generated representations* to detect OOD graphs, different from GNNs like NGNN and GNN-AK+ that are to learn task-specific substructures. To achieve this, as shown in Figure 1, we first design a super graph of substructures, which models the relationships between task-agnostic substructures in a graph, and then generate substructure-enhanced graph representations via two-level graph encoding (Section 3.1); we then design substructure-preserving graph augmentations to enrich training data, while keeping task-agnostic substructures themselves intact (Section 3.2); lastly we design proper objective functions in Section 3.3. These techniques in SGOOD are designed with the purpose to encode task-agonistic substructures into representations for better graph-level OOD detection.

We have revised Section 1 and 3 accordingly in the latest version of our paper that has been uploaded.

Reviewer comment (Theory):
Author wrote: "For augmentations, intuitively, if more information about training ID data is preserved, it is easier to distinguish unseen OOD data. The substructure-preserving graph augmentations are designed to achieve this." Please provide further explanation for “more information”. What we need to do is to embed all the information related to the substructure into the graph representation? In [2], authors proposed that encoding the task-agnostic (e.g., graph classification task-agnostic) information into representations can improve the OOD detection task.

Author response:
We thank the reviewer for the insightful comment by mentioning [2]. The substructures used in our method SGOOD are indeed *task-agnostic*, as they are detected by existing methods, before the training of SGOOD. SGOOD encodes these substructures into the generated representations to improve graph-level OOD detection task.

Augmentation techniques are usually designed to enrich training data to train models. Therefore, the "more information" here is to just intuitively indicate that the proposed augmentation techniques can help generate more substructure-preserving training samples, and subsequently, we can sufficiently train our method to generate representations that can better distinguish OOD from ID graphs, which is validated in the ablation study in Table 4.

Reviewer comment (Presentation):
Table 1 lacks clarity, making it difficult for readers to realize the ID and OOD graphs used in statistics, and these statistical findings rely on prior knowledge.

Author response:
We agree that Table 1 itself is not self-contained, due to the lack of space in introduction, and we have revised Section 1 in the latest version uploaded. Specifically, please refer to Table 2 in Section 4 for ID and OOD graph statistics. Table 1 shows the percentage of OOD test graphs containing substructures never appeared in ID training graphs in each dataset. These percentage values are high, higher than 44% in 4/6 datasets, indicating that many OOD graphs contain substructures rarely appear in ID graphs. This observation motivates our design to encode substructures to improve graph-level OOD detection. Note that the only purpose of this statistical finding is to motivate our idea, and it is not used in the training, since OOD graphs are only available at test time.

Reviewer comment (Experiments):
GNNsafe appears to be primarily designed for node-level OOD detection. How can it be implemented at the graph-level?

Author response:
GNNSafe is primarily designed for node-level OOD detection. Therefore, we compare with a modified version of GNNSafe, as stated in Appendix B.3. Specifically, in Section 3.2 of the GNNSafe paper, in a single graph, GNNSafe propagates node energy scores that are obtained by training GNN on *node labels*, which however are not available in our graph-level setting where only *graph-level labels* are available. Therefore, we compared with a version of GNNSafe without node energy score propagation.

Reviewer comment (Experiments):
OCGIN, OCGTL, and GLocalKD are predominantly designed for graph anomaly detection, and their use as comparison algorithms may not be entirely appropriate for graph-level OOD detection.

Author response:
We follow previous studies on graph-level OOD detection [5, 6], and included the comparison to these methods, in order to comprehensively evaluate the performance of our method and existing methods.

Reviewer comment (Presentation):
Figure 1 does not effectively convey how SGOOD is specifically tailored for the graph-level OOD detection task.

Author response:
Thank you for the suggestion. We have updated Figure 1 and its caption in the latest version of our paper.

Reviewer comment (Novelty):
The proposed substructure learning on graphs is related to identifying and learning causally invariant substructures, which has been studied in some previous works [1-3].

Author response:
We clarify that we do not learn causally invariant substructures. The substructures used in SGOOD are *task-agnostic*, while causally invariant substructures in existing studies are learned with specific tasks. (i) As stated in Section 3.1 (last paragraph on Page 3), our method SGOOD is orthogonal to existing subgraph detection methods, and it is not our focus on how to identify substructures.  (ii) As shown in Figure 1, SGOOD is not relevant to learning causally invariant substructures. We use substructures to enhance the generated representations for graph-level OOD detection. (iii) As shown in Table 6, SGOOD can work with different substructures identified by different methods, and improve performance compared with the base method without substructures.

Reviewer comment (Theory):
As for Substructure-Preserving Graph Augmentations, although it preserves substructures, it might change the semantics of graphs.

Author response:
We highlight that (i) for the purpose of enriching the input graph data, augmentation techniques, including ours, will change the semantics of graphs, but (ii) there is no need to worry about this in SGOOD, since in the training of SGOOD for loss $\mathcal{L}_{CE}$ in Eq. (5), all original input graphs are used *without* semantic changes. Moreover, as shown in Table 4 of the paper, the proposed augmentation techniques indeed bring performance improvements.

Reviewer comment (Theory):
How can the proposed SGOOD ensure semantically meaningful substructures extracted by predefined methods?

Author response:
We clarify that it is not our focus on how to extract semantically meaningful substructures, as stated in Section 3.1. The substructures are task-agnostic and extracted as a pre-step. Our main goal is how to encode the substructures into effective representations for graph OOD detection, as illustrated in Figure 1. Our design is motivated by the observation in Table 1 that ID and OOD graphs often have different task-agnostic substructures. As validated in Table 6, SGOOD with different substructures by different methods can always improve performance, compared with the base method without substructures.

Reviewer comment (Experiments):
Why not use other learning-based techniques like hypergraph learning, graph pooling or causal learning to extract substructures?

Author response:
(i) In Table 7 of the paper, we have compared with graph pooling methods (SAG, TopK, DiffPool), which is outperformed for graph-level OOD detection. (ii) The focus of SGOOD is to learn expressive representations to encode task-agnostic substructures to distinguish OOD graphs, while it is not our focus on how to extract substructures, as stated in Section 3.1, and it is a different topic from our method. We agree that it is possible to further investigate the impact of learning techniques as future work.

Reviewer comment (Experiments):
How is the model sensitive to different substructures as prior information? And how does this impact different tasks and datasets?

Author response:
**Q1:** How is the model sensitive to different substructures as prior information? And how does this impact different tasks and datasets?  
**Response:**  
As stated in Section 3.1, our framework SGOOD is orthogonal to existing subgraph detection methods. The substructures used in SGOOD are task-agnostic. As reported in Table 6 of the paper, compared with the base version without substructures (w.o. substructures), SGOOD can always improve OOD detection performance when adopting different substructures identified by different methods (Modularity, Graclus, LP, BRICS). This demonstrates that SGOOD is not sensitive to specific substructures as prior knowledge. Moreover, it is a natural observation that on different datasets, the performance may vary, but as reported in Table 3 of the paper, SGOOD consistently achieves superior performance under various metrics, including AUROC, AUPR, and FRP95, across 8 real datasets.

Reviewer comment (Experiments):
The authors mentioned that GNNSafe [1], which is the state-of-the-art model for out-of-distribution detection on graphs, cannot be directly compared, can it be stated more clear why GNNSafe is not comparable with the methods in the experiment?

Author response:
**Q3:** The authors mentioned that GNNSafe [2], which is the state-of-the-art model for out-of-distribution detection on graphs, cannot be directly compared, can it be stated more clear why GNNSafe is not comparable with the methods in the experiment?  
**Response:**  
We clarify that (i) GNNSafe is compared in experiments, as reported in Table 3, and (ii) the compared GNNSafe is a modified version, since GNNSafe itself is for node-level OOD detection, but not for graph level. Specifically, in Section 3.2 of the GNNSafe paper [2], in a single graph, GNNSafe propagates node energy scores that are obtained by training GNN on *node labels*, which however are not available in our graph-level setting where only *graph-level labels* are available. Therefore, we compared with a modified version of GNNSafe without such node energy score propagation in Table 3 of our paper.

Reviewer comment (Experiments):
The experimental datasets already used are small. How does the model perform on large datasets? What is the computation cost compared with others?

Author response:
**Q4:** The experimental datasets already used are small. How does the model perform on large datasets? What is the computation cost compared with others?  
**Response:**  
First, we want to explain that, compared with the existing papers on graph OOD detection [3,4], we are already using many datasets with sufficiently large sizes in experiments, such as HIV with in total *36192* graphs and REDDIT with 9622 graphs in *11* classes, as shown in Table 2 of the paper. 

Second, the training and inference time of our method SGOOD and baselines have been reported in Table 11 of Appendix C in the submission. Compared with existing methods, SGOOD achieves better or comparable efficiency. Considering together the effectiveness reported in Table 3, we conclude that SGOOD has superior accuracy for graph-level OOD detection, while being reasonably efficient.