Paper: Score Propagation as a Catalyst for Graph Out-of-distribution Detection: A Theoretical and Empirical Study (ICLR-2024)

Reviewer comment (Experiments):
However, the experiment was all conducted on small-scale of data, which limits the ability to fully validate the proposed approach and conclusions. More experiments on large-scale real world data would strength the claims.

Author response:
> **Q1. Experiments on large-scale real world data.**

We are glad to share with you that we have conducted experiments on large datasets in paper, i.e. arxiv-year, snap-patents, and wiki, which are three large-scale real world datasets reflecting real-world scenarios. Specifically, Arxiv-year has node and edge counts in the order of hundreds of thousands and millions, respectively. Snap-patents features millions of nodes and tens of millions of edges, while wiki, being a large dense graph, boasts millions of nodes and billions of edges. The detailed statistics of these datasets [1] are provided in the following table. We believe these datasets sufficiently meet the requirements of real-world applications. If the reviewer has an interest in other large datasets, feel free to share with us and we would be more than willing to conduct additional experiments on these datasets to further check the effectiveness of our proposed method.

|Dataset | #Nodes | #Edges |
|:-------- |--------:|--------:|
| arXiv-year | 169,343 | 1,166,243 |
|snap-patents |**2,923,922** |13,975,788 |
|wiki|**1,925,342**|**303,434,860**|

[1] Derek Lim, Felix Hohne, Xiuyu Li, Sijia Linda Huang, Vaishnavi Gupta, Omkar Bhalerao and Ser-Nam Lim. Large Scale Learning on Non-Homophilous Graphs: New Benchmarks and Strong Simple Methods. NeurIPS 2021.

Reviewer comment (Evaluation):
The time complexity of GRASP compared to baselines is not mentioned in the paper. A thorough accounting of computation/memory demands compared to baselines is important as it relates to practical deployment.

Author response:
> **Q2. The time complexity of GRASP and a thorough accounting of computation/memory demands compared to baselines to account for practical deployment.**

Great question! We first present the conclusion and then provide the support from both mathmatical and empirical points. 

**Takeway:** Our method's computational footprint in terms of a single propagation’s runtime and memory usage is:
- Time Complexity: $O(N+|\mathcal{E}|+n)$,
- Memory Complexity: $O(N+|\mathcal{E}|+n)$,

where $N$ is node count of the entire graph $\mathcal{G}$ with $|\mathcal{E}|$ edges and subset $G$ has $n$ nodes. Hence, our algorithm has a linear complexity and this makes our method applicable to large-scale networks.

**Justification:** While our algorithm introduces the fully connected matrix $E$, we adeptly transform the propagation formula to eliminate direct matrix computation between $E$ and the OOD score vector $\mathbf{g}$, resulting in an efficient linear complexity. This can be  confirmed by the computational complexity  analysis presented below. 
- **Computational complexity analysis.** As described by Equation (5) in our paper, given a raw OOD scoring vector $\hat{\mathbf{g}} \in \mathbb{R}^{N}$, the propagated scoring vector using augmentated adjacency matrix after propagation for $k$ times is given by: 
 \begin{align*}
     \mathbf{g} \_{GRASP} &= (\bar{A} \_+)^k \hat{\mathbf{g} } 
     = ({D}^{-1} \_+A \_+)^k \hat{\mathbf{g}} = ({D}^{-1} \_+(A+E))^k \hat{\mathbf{g}} \\
     &= ({D}^{-1} \_+A \_+)^{k-1} \cdot (D \_+^{-1}\boxed{A\hat{\mathbf{g}}}+D \_+^{-1}\boxed{E\hat{\mathbf{g}}})   \\
     &= (\text{run $k-1$ times ...}) \\
\end{align*}
In the above equation, ($a$) $E\hat{\mathbf{g}}$ can be computed with time/space complexity $O(n)$ by a simple summation operation ($\mathbf{g} \_i$ in $G$ will be replaced by $\sum \_{i\in G} \mathbf{g} \_i$ ) to get rid of the matrix multiplication; ($b$) $A\hat{\mathbf{g}}$ can be computed in $O(|\mathcal{E}|)$ with sparse matrix multiplication; ($c$) $D \_+^{-1}$ is an $O(N)$ operation since it is scaling over all elements in the vector. In all, the time/memory complexity of GRASP is $O(N+|\mathcal{E}|+n)$.

Reviewer comment (Theory):
The theoretical analysis assumes edges follow a Bernoulli distribution. How sensitive are the results to this assumption?

Author response:
> **Q3. "Why does the theoretical analysis assume that the edges follow a Bernoulli distribution?"**

As we are dealing with discrete graphs, where edges either exist or do not, the adjacency matrix values are binary, taking on either 0 or 1. This naturally aligns with the Bernoulli distribution, which is frequently employed in graph structure learning, as exemplified in several pertinent references in [1,2,3].

[1] Luca Franceschi, Mathias Niepert, Massimiliano Pontil and Xiao He. Learning Discrete Structures for Graph Neural Networks. ICML 2019.

[2] Pantelis Elinas, Edwin V. Bonilla and Louis C. Tiao. Variational Inference for Graph Convolutional Networks in the Absence of Graph Data and Adversarial Settings. NeurIPS 2020.

[3] Tong Zhao, Yozen Liu, Leonardo Neves, Oliver Woodford, Meng Jiang and Neil Shah. Data Augmentation for Graph Neural Networks. AAAI 2021.

Reviewer comment (Experiments):
How do different propagation mechanisms, like higher-order diffusion, impact the findings?

Author response:
> **Q4. "The impact of other propagation mechanisms, like higher-order diffusion."**

Excellent suggestions! We have conducted an analysis of the hyper-parameter $k$ (order of propagation) and its impact on the AUROC, averaged across all datasets using a GCN backbone. Our findings indicate that OOD detection performance benefits from propagation orders within a reasonable range. However, excessive propagation (greater than 10) may be detrimental.


| |  |  |  |  | |
|:-------- |:--------:|:--------:|:--------:|:--------:| :--------:|
|$k$|2|4|8|10|16|
|AUROC|79.38 |80.87 |82.45|80.53 |71.19 |



In addition, we have investigated various classical propagation mechanisms, including Personalized PageRank (PPR) [1], Heat Kernel Diffusion (GraphHeat) [2], Graph Diffusion Convolution (GDC) [3], Mixing Higher-Order Propagation (MixHop) [4], and Generalized PageRank (GPR) [5]. The results (**AUROC** in percentages) demonstrate that GRASP emerges as the most effective propagation strategy in our study.

| |cora| amazon| coauthor| chameleon| 
|:-------- |:--------:|:--------:|:--------:|:--------:|        
|PPR|82.68 |83.34 |85.18 |52.79 |
|GraphHeat	|92.76 	|76.99 |	70.53 |	58.48 |	
|GDC|89.48 |92.33 |96.52 |54.47 |
|MixHop|79.80 |82.68 |86.04 |55.03 |
|GPR|78.70 |81.57 |81.84 |52.86 |
|**GRASP**|**94.65** |**96.76** |**97.94** |**67.97** |


| |squirrel| arxiv-year| snap-patents| wiki |
|:-------- |:--------:|:--------:|:--------:|:--------:|    
|PPR|48.17 |    56.87 |46.70 |34.83 |
|GraphHeat	|45.63 |38.40 |	41.00 |OOM |
|GDC|48.82  | OOM|OOM|OOM  |     
|MixHop|48.66 | 34.41 |28.93 |39.22 |
|GPR|49.64 | 34.30 |29.66 |36.98 |
|**GRASP**|**54.93** | **74.66** |**67.36** |**65.56** |



[1] Johannes Gasteiger, Aleksandar Bojchevski, and Stephan Günnemann. Predict then Propagate: Graph Neural Networks Meet Personalized PageRank. ICLR 2019.

[2] Bingbing Xu , Huawei Shen , Qi Cao , Keting Cen and Xueqi Cheng. Graph Convolutional Networks using Heat Kernel for Semi-supervised Learning. IJCAI 2019.

[3] Johannes Gasteiger, Stefan Weißenberger and Stephan Günnemann. Diffusion Improves Graph Learning. NeurIPS 2019.

[4] Sami Abu-El-Haija, Bryan Perozzi, Amol Kapoor, Nazanin Alipourfard, Kristina Lerman, Hrayr Harutyunyan, Greg Ver Steeg, and Aram Galstyan. MixHop: Higher-Order Graph Convolutional Architectures via Sparsified Neighborhood Mixing. ICML 2019.

[5] Eli Chien, Jianhao Peng, Pan Li, and Olgica Milenkovic. Adaptive universal generalized pagerank graph neural network. ICLR 2021.

Reviewer comment (Theory):
What is the time complexity of GRASP compared to baselines?

Author response:
> **Q2. The time complexity of GRASP and a thorough accounting of computation/memory demands compared to baselines to account for practical deployment.**

Great question! We first present the conclusion and then provide the support from both mathmatical and empirical points. 

**Takeway:** Our method's computational footprint in terms of a single propagation’s runtime and memory usage is:
- Time Complexity: $O(N+|\mathcal{E}|+n)$,
- Memory Complexity: $O(N+|\mathcal{E}|+n)$,

where $N$ is node count of the entire graph $\mathcal{G}$ with $|\mathcal{E}|$ edges and subset $G$ has $n$ nodes. Hence, our algorithm has a linear complexity and this makes our method applicable to large-scale networks.

**Justification:** While our algorithm introduces the fully connected matrix $E$, we adeptly transform the propagation formula to eliminate direct matrix computation between $E$ and the OOD score vector $\mathbf{g}$, resulting in an efficient linear complexity. This can be  confirmed by the computational complexity  analysis presented below. 
- **Computational complexity analysis.** As described by Equation (5) in our paper, given a raw OOD scoring vector $\hat{\mathbf{g}} \in \mathbb{R}^{N}$, the propagated scoring vector using augmentated adjacency matrix after propagation for $k$ times is given by: 
 \begin{align*}
     \mathbf{g} \_{GRASP} &= (\bar{A} \_+)^k \hat{\mathbf{g} } 
     = ({D}^{-1} \_+A \_+)^k \hat{\mathbf{g}} = ({D}^{-1} \_+(A+E))^k \hat{\mathbf{g}} \\
     &= ({D}^{-1} \_+A \_+)^{k-1} \cdot (D \_+^{-1}\boxed{A\hat{\mathbf{g}}}+D \_+^{-1}\boxed{E\hat{\mathbf{g}}})   \\
     &= (\text{run $k-1$ times ...}) \\
\end{align*}
In the above equation, ($a$) $E\hat{\mathbf{g}}$ can be computed with time/space complexity $O(n)$ by a simple summation operation ($\mathbf{g} \_i$ in $G$ will be replaced by $\sum \_{i\in G} \mathbf{g} \_i$ ) to get rid of the matrix multiplication; ($b$) $A\hat{\mathbf{g}}$ can be computed in $O(|\mathcal{E}|)$ with sparse matrix multiplication; ($c$) $D \_+^{-1}$ is an $O(N)$ operation since it is scaling over all elements in the vector. In all, the time/memory complexity of GRASP is $O(N+|\mathcal{E}|+n)$.

Reviewer comment (Theory):
The proposed method is developed based the assumption that intra-edges dominate the graph. However, the challenge in OOD detection arises from the unknown pattern of OOD nodes. This strong assumption makes the analysis less insightful and constrains the practicality of the proposed method.

Author response:
> **1. "Assumption of the intra-edge domination."**

We respectfully point out that the reviewer's interpretation of "*The proposed method is developed based on the assumption that intra-edges dominate the graph*" is incorrect. The intra-edge-domination assumption is only required for the *naive propagation method* as proved in Theorem 3.2. One of the primary objectives of our algorithm is to address the hard problem of graph node ood detection in heterophilic scenarios, **where intra-edges do not dominate the graph.**

For instance, on three particularly challenging heterophily datasets—Chameleon, arXiv-year, and snap-patents, the application of naive propagation method leads to a significant performance decline. In contrast, our approach not only reversed this trend but also demonstrates substantial improvement compared to pre-propagation performance. Refer to the table below for detailed results, where the performance values  are evaluated on backbone GCN. From the table we can see that our method yields a maxmium performance increase of up to **32.5** percentage points on these challenging heterophilic datasets (AUROC on arXiv-year for MSP). This robustly underscores the effectiveness of our proposed approach.

| | Chameleon| |arXiv-year| |snap-patents ||
|:-------- |:--------| :--------|:--------|:--------|:--------|:--------|
||AUROC	|FPR95	|AUROC		|FPR95		|AUROC		|FPR95	|
|MSP|59.91|90.87| 43.35|95.60 | 51.42|93.10| 
|MSP+prop|42.82|98.17 |  35.30|100.0  | 27.25|100.0  | 
|MSP+**GRASP**|**69.57**|**72.77** | **75.87**|**85.78** | **67.80**|**74.91** | 
|Energy	|59.68|94.98|47.58|95.00 |46.93|93.85 |
|Energy+prop|49.65|97.74 | 35.40|100.0   |  27.26|100.0 | 
|Energy+**GRASP**|**67.97**|**74.54**| **74.66**|**86.70** | **67.36**|**74.60** |

Reviewer comment (Experiments):
The experiments are conducted exclusively with one type of OOD nodes (Label Leave-Out), which limits the comprehensive evaluation of the proposed method under different distribution shifts.

Author response:
> **2. "How does the proposed method perform when faced with structural manipulation and feature interpolation as addressed in GNNSafe?"**

Thanks for raising this point! We adopt the "label leave-out" setting since it aligns closely with practical scenarios where OOD data comes from a different class. The other two settings (structural manipulation and feature interpolation) construct OOD data through manual perturbation, which significantly differs from real-world OOD datas. Even if a method performs well on artificially synthesized OOD data, it does not necessarily reflect its real performance in real-world scenarios. Therefore, we exclusively adopt the "label leave-out" setting.


As the reviewer suggest, we also conduct experiments on the two different types of OOD (structural manipulation and feature interpolation) proposed by GNNSafe, and the results (AUROC) indicate that our method still performs well in these different constructed OOD scenarios.

|			|cora	|		|Amazon	|		|coauthor|		|Average|	  |
|:-------- |:--------| :--------|:--------|:--------|:--------|:--------|:--------|:--------|
||S		|F 		|S		|F 		|S		|F 		|S		|F    |
|MSP			|70.90 	|85.39 	|98.27 	|97.31 	|95.30 	|97.05 	|88.16 	|93.25|
|ODIN		|49.92 	|49.88 	|93.24 	|81.15 	|52.14 	|51.54 	|65.10 	|60.86|
|Mahalanobis	|46.68 	|49.93 	|71.69 	|76.50 	|80.46 	|93.23 	|66.28 	|73.22|
|Energy		|71.73 	|86.15 	|98.51 	|97.87 	|96.18 	|97.88 	|88.81 	|93.97|
|GKDE		|68.61 	|82.79 	|76.39 	|58.96 	|65.87 	|80.69 	|70.29 	|74.15|
|GPN			|77.47 	|85.88 	|97.17 	|87.91 	|34.67 	|72.56 	|69.77 	|82.12|
|GNNSafe		|87.52 	|93.44 	|99.58 	|98.55 	|99.60 	|99.64 	|95.57 	|97.21|
|**GRASP**		|**95.29** 	|**98.93** 	|**98.50** 	|**98.45** 	|**99.99** 	|**99.80** 	|**97.93** 	|**99.06**|

Reviewer comment (Evaluation):
The selection of Sid/Sood is based on the pre-computed OOD scores (using MSP), rendering the proposed method ineffective when MSP fails, as observed in dataset Squirrel in Table 2.

Author response:
> **3. "The proposed method may be ineffective when MSP fails, as observed in dataset Squirrel in Table 2."**

We would like to share a differing viewpoint from the reviewer.  The effectiveness of our method, GRASP, is evidenced through two key observations:

1. As shown in Table 2, GRASP significantly surpasses MSP by a margin of **6.76** in AUROC on the Squirrel dataset using the GCN model. This improvement underscores the benefits of score propagation augmented by our method. **Notably, this enhancement is also observed in scenarios where MSP alone is ineffective**.

2. On the other hand, the efficacy of our strategy is primarily measured by the extent to which it enhances Out-of-Distribution (OOD) detection performance beyond the initial OOD scores prior to propagation. Among traditional OOD detection methods, the KNN scores achieved the highest AUROC on the Squirrel dataset. However, when our propagation technique is applied to KNN, there is a notable improvement in its pre-propagation performance, as demonstrated in the table below. Furthermore, across all datasets, applying our method to KNN consistently improves performance in terms of both AUROC and FPR95. This consistently better performance across various metrics highlights the robustness of our approach. It is important to note that this improvement is observed even in cases where MSP underperforms.

    ||cora	|		|amazon		|			|coauthor	|		|chameleon	|		|squirrel	|      |
    |:--------|:--------|:--------|:--------|:--------|:--------|:--------|:--------|:--------|:--------|:--------|
    ||AUROC	|FPR95	|AUROC		|FPR95		|AUROC		|FPR95	|AUROC		|FPR95	|AUROC |FPR95 |
    |KNN		|81.24	|72.29	|86.01		|60.61		|91.34		|47.99	|62.09		|93.43	|56.74	|94.42 |
    |KNN+**GRASP**|**88.23**|**33.16**|**95.30**|**21.43**|**97.57**|**8.86** |**74.44**		|**62.29**	|**58.57**		|**87.96** |


We are also more than happy to provide more discussion to address the concerns regarding the "MSP reliance" as follows:
1.  As proved in Theorem 4.2, the conditions required for our method to work well are easy to satisfy. The chosen node set G only needs to satisfy the condition $|\mathcal{E} \_{G\leftrightarrow S \_{ID}}|>|\mathcal{E} \_{G \leftrightarrow S \_{OOD}}|$. 
2.  Our method selectively considers data points at the two ends of the score distribution. This selection minimizes the error in selecting $S \_{ID}$ and $S \_{OOD}$.

Reviewer comment (Theory):
According to Theorem 3.2, the propagation is deemed effective only when intra-edges dominate. Why does it seem to work well in heterophily dataset Chameleon?

Author response:
> **4. "Why naive propagration seems to work well in heterophily dataset Chameleon?"**

We are happy to provide more clarification here.  As illustrated in Table 3, on the Chameleon dataset, both MSP and Energy exhibit **degraded** performance after naive propagation (evaluated using FPR95, where lower values indicate better results). Notably, our proposed method successfully reverses this trend, significantly improving performance. Table 3 provides a clear demonstration of this enhancement.

Reviewer comment (Evaluation):
Why does the improvement of the proposed method over GNNSafe appear to be marginal in homophily datasets like Amazon and Coauthor, while the improvement is more significant in the two heterophily datasets in Table 2?

Author response:
> **5. "Why does the improvement of the proposed method over GNNSafe appear to be marginal in homophily datasets, while the improvement is more significant in the heterophily datasets?"**

We respectfully disagree with the reviewer's observation. Firstly, our method's improvement over GNNSafe in homophily datasets is not marginal, which can be verified by the results in the following table (Averaged on the three homophily datasets Cora, Amazon and Coauthor). From the results we can see that our method outperforms GNNSafe by **7.10** and **1.62** on FPR and AUROC respectively. Secondly, the more pronounced improvement on heterophily datasets aligns with our theoretical derivation, since the augmentation strategy aims to improve the ratio of "intra-edges" after propagation. Notably, if inter-edges dominate, traditional propagation methods would lead to performance degradation, while our approach successfully reverses this trend, as evidenced by certain results in Table 3.
|		|FPR95|AUROC|
|:--------|:--------|:--------|
|GNNSafe|22.25|94.83	|
|**GRASP**|**15.15**|**96.45**	|

Reviewer comment (Experiments):
How does the proposed method perform when faced with different types of OOD, such as structural manipulation and feature interpolation as addressed in GNNSafe?

Author response:
> **2. "How does the proposed method perform when faced with structural manipulation and feature interpolation as addressed in GNNSafe?"**

Thanks for raising this point! We adopt the "label leave-out" setting since it aligns closely with practical scenarios where OOD data comes from a different class. The other two settings (structural manipulation and feature interpolation) construct OOD data through manual perturbation, which significantly differs from real-world OOD datas. Even if a method performs well on artificially synthesized OOD data, it does not necessarily reflect its real performance in real-world scenarios. Therefore, we exclusively adopt the "label leave-out" setting.


As the reviewer suggest, we also conduct experiments on the two different types of OOD (structural manipulation and feature interpolation) proposed by GNNSafe, and the results (AUROC) indicate that our method still performs well in these different constructed OOD scenarios.

|			|cora	|		|Amazon	|		|coauthor|		|Average|	  |
|:-------- |:--------| :--------|:--------|:--------|:--------|:--------|:--------|:--------|
||S		|F 		|S		|F 		|S		|F 		|S		|F    |
|MSP			|70.90 	|85.39 	|98.27 	|97.31 	|95.30 	|97.05 	|88.16 	|93.25|
|ODIN		|49.92 	|49.88 	|93.24 	|81.15 	|52.14 	|51.54 	|65.10 	|60.86|
|Mahalanobis	|46.68 	|49.93 	|71.69 	|76.50 	|80.46 	|93.23 	|66.28 	|73.22|
|Energy		|71.73 	|86.15 	|98.51 	|97.87 	|96.18 	|97.88 	|88.81 	|93.97|
|GKDE		|68.61 	|82.79 	|76.39 	|58.96 	|65.87 	|80.69 	|70.29 	|74.15|
|GPN			|77.47 	|85.88 	|97.17 	|87.91 	|34.67 	|72.56 	|69.77 	|82.12|
|GNNSafe		|87.52 	|93.44 	|99.58 	|98.55 	|99.60 	|99.64 	|95.57 	|97.21|
|**GRASP**		|**95.29** 	|**98.93** 	|**98.50** 	|**98.45** 	|**99.99** 	|**99.80** 	|**97.93** 	|**99.06**|

Reviewer comment (Experiments):
Is the proposed method sensitive to hyperparameter k, $\alpha$, and $\beta$? How to choose the $\alpha$ and $\beta$ when MSP’s performance varies in different datasets?

Author response:
> **6. "Hyperparameter sensitive analysis and how to choose $\alpha$ and $\beta$ when MSP’s performance varies in different datasets"**

Great suggestions! We conduct a sensitivity analysis of all the hyper-parameters on all the datasets and the averaged results (**AUROC** in percentages) on model GCN with Energy are displayed in the following table. The performance comparison in the table for each hyper-parameter is reported by fixing other hyper-parameters. From the results we can see that within the range of chosen hyperparameter values, our proposed method's performance does not vary significantly, which constantly outperforms baselines by a large margin.


| |  |  |  |  | |
|:-------- |:--------:|:--------:|:--------:|:--------:| :--------:|
| $\alpha$ | 1 | 2.5 | 5 | 10 |15 |
|AUROC |81.23 |81.24 |82.45|80.83 |80.35 |
|$\beta$|10|40|50|60|90|
|AUROC|77.78 |81.22 |82.45|81.79  |80.12 |
|$k$|2|4|8|10|16|
|AUROC|79.38 |80.87 |82.45|80.53 |71.19 |

Reviewer comment (Theory):
The data augmentation relies on a robust base model. During the graph augmentation, ID nodes and OOD nodes are sampled based on the base model's OOD prediction scores, which may lead to potential error propagation.

Author response:
We thank the reviewer for the insightful questions! Below we address each of your comments in detail.

> **Q1. "The base model's OOD prediction scores may lead to potential error propagation"**

We are more than happy to provide more discussion to address the concerns as follows:

1. We would like to highlight that our augmentation approach will not incorporate error-edges connecting ID and OOD, since we add edges in the training dataset and the augmentation is performed **exclusively** on in-distribution (ID) data. 
2. As proved in Theorem 4.2, the conditions required for our method to work well are easy to satisfy. The chosen node set G only needs to satisfy the condition $|\mathcal{E} \_{G\leftrightarrow S \_{ID}}|>|\mathcal{E} \_{G \leftrightarrow S \_{OOD}}|$. This condition, compared to directly augmenting edges on the test set, allows for a stronger degree of tolerance to errors.
3. Our method selectively considers data points at the two ends of the Maximum Softmax Probability (MSP) distribution. This selection minimizes the error in selecting $S \_{ID}$ and $S \_{OOD}$. 

For reviewer's interest, we are happy to provide comparison with an alternative approach -- augmenting edges directly on the test data. Such strategy would lead to severe error propagation, which is known as "confirmation bias". To validate this point, we conduct experiments comparing the effectiveness of "directly augmenting edges on test data" versus "our proposed method of augmenting edges in the training dataset." From the results, we can see that our method has achieved an improvement of nearly **10** percentage points in AUROC compared to the approach of directly adding edges on test data, which supports our assertion.

(Cells' values represent AUROC in percentages on backbone GCN with Energy Score.)

| Method| Cora | Amazon | Coauthor | Chameleon | Squirrel | Average |
|:-------- |:--------:|:--------:|:--------:|:--------:| :--------:| :--------:|
|Before Augmetation (Energy Score)|89.48|92.33|96.52|59.68|45.06|76.61|
|Augmentation on Test (with confirmation bias)|85.90|87.99|94.72|48.32|46.96|72.78|
|Augmentation on Train (**Ours**)|**94.65**|**96.76**|**97.94**|**67.97**|**54.93**|**82.45**|

Reviewer comment (Experiments):
Analysis of some key hyper-parameters is missing. It appears that the two hyper-parameters \alpha and \beta in the proposed method significantly affect the sampling results, and I would like to know the impact of different \alpha and \beta values on the results.

Author response:
> **Q2. "Analysis of two hyper-parameters $\alpha$ and $\beta$ is missing"**

Great suggestions! We conduct sensitivity analysis of hyper-parameters $\alpha$ and $\beta$ on all the datasets and the averaged results (**AUROC** in percentages) on model GCN with Energy are displayed in the following table. The performance comparison in the table for each hyper-parameter is reported by fixing other hyper-parameters. From the results we can see that within the range of chosen hyperparameter values, our proposed method's performance does not vary significantly, which constantly outperforms baselines by a large margin.



|          |       |       |       |       |       |
|:-------- |:-----:|:-----:|:-----:|:-----:| -----:|
| $\alpha$ |   1   |  2.5  |   5   |  10   |    15 |
| AUROC    | 81.23 | 81.24 | 82.45 | 80.83 | 80.35 |
| $\beta$  |  10   |  40   |  50   |  60   |    90 |
| AUROC    | 77.78 | 81.22 | 82.45 | 81.79 | 80.12 |

Reviewer comment (Experiments):
According to the authors, the number of intra-edges and inter-edges has a significant impact on OOD detection. What are the respective proportions of these two types of edges in different datasets? And what are their proportions after graph augmentation?

Author response:
> **Q3. "The respective proportions of intra-edges and inter-edges before and after graph augmentation"**

Great suggestions! We provide proportions of intra-edges and inter-edges after two rounds of propagations in each dataset  before and after graph augmentation respectively in the following table. From the results we can see that the proportions of intra-edges increase after graph augmentation, which lead to improved ood performance, validating the correctness of our theory. 

|Dataset| intra-edges ratio(Before Aug) | inter-edges ratio(Before Aug) | intra-edges ratio(After Aug) | inter-edges ratio(After Aug) | 
|:--------|:--------:|:--------:|:--------:|:--------:|
|cora		|0.8155 	|0.1845 	|**0.9188** 	|0.0812| 
|amazon-photo|0.6405 	|0.3595 	|**0.7264** 	|0.2736| 
|coauthor-cs	|0.8105 	|0.1895 	|**0.8690** 	|0.1310| 
|chameleon	|0.5082 	|0.4918 	|**0.5274** 	|0.4726| 
|squirrel	|0.4972 	|0.5028 	|**0.5167** 	|0.4833|