Paper: Class-Imbalanced Graph Learning without Class Rebalancing (ICLR-2024)

Reviewer comment (Theory):
The augmentation method introduced in this paper could potentially escalate the computational complexity, presenting challenges for deployment in large-scale scenarios. It might be beneficial to either highlight this as a potential limitation or to delve into a theoretical analysis addressing its implications in expansive applications.

Author response:
## (W1) Computational complexity

**TL;DR: ToBE introduces light computational overhead in practice. We addressed this concern in the "Experiment - Complexity Analysis" section and provided a comprehensive analysis of ToBE's time and space complexity. We will move it to the methodology section to improve the readability.**


In short, ToBE$_0$ and ToBE$_1$ exhibits $\mathcal{O}(|\mathcal{V}|C)$ and $\mathcal{O}(|\mathcal{E}|C)$ space/computation complexity, where $|\mathcal{V}|$ is the number of nodes, $|\mathcal{E}|$ is the number of edges, and $C$ is the number of classes, which is typically small in practice. Moreover, since all operations can be executed in parallel in matrix form, the time complexity of ToBE$_0$/ToBE$_1$ can be further reduced to $\mathcal{O}(\frac{|\mathcal{V}|C}{D})$ / $\mathcal{O}(\frac{|\mathcal{E}|C}{D})$, where D is the number of available computing units, often large for modern GPUs.

We provide computational efficiency results of ToBE on real-world graph data in Table 3. Additionally, in Appendix D.2, we discuss strategies for further accelerating ToBE in practical applications.

Reviewer comment (Experiments):
The manuscript remains silent on the performance of the proposed methodology in multi-class classification environments, as well as its adaptability to tasks other than node classification. Exploring these facets could provide a more comprehensive view of its applicability.

Author response:
## (W3) Multi-classification and potential future directions

**TL;DR: Our work does indeed focus on multi-class classification scenarios (Appendix C.1). We will incorporate a more in-depth discussions on limitations and possible future directions.**

**Regarding multi-classification**: We want to clarify that our work does indeed focus on multi-class classification scenarios. For further clarification, please refer to Appendix C.1, where the number of classes for each dataset used in the experiments is detailed (Table 4). Additionally, we provide a visualization of class distributions across different settings in Figure 8.

**Regarding tasks beyond node classification**: 
Thank you for your valuable feedback on potential areas for further exploration. We acknowledge the importance of extending ToBE to other imbalanced graph learning tasks. For instance, imbalanced edge prediction is a less-explored yet crucial direction (e.g., for knowledge graph completion). This involves imbalance at both the edge level (presence/absence) and the node/entity level (some entities having more relationships). Adapting the existing CIGL framework to address these challenges is non-trivial. We will incorporate a more in-depth discussion of these possibilities in the manuscript.

Reviewer comment (Evaluation):
The proposed method seems to be meaningless at times. As shown in Table, sometimes the original methods (such as APPNP and GPRGNN) can beat either +ToBE0 or +ToBE1. Moreover, as shown in Tables 7 and 8, lots of baselines (Vanilla, Reweight, ReNode, and GSMOTE) can sometimes beat either +ToBE0 or +ToBE1.

Author response:
## (W1, Q1) Regarding the effectiveness of ToBE

**TL;DR: Empirically, ToBE is effective in over 98% of the cases (103/105) across all performance metrics (Table 7/8/9). The rare cases that ToBE does not work well are caused by oversmoothing from the multi-step message-passing in APPNP/GPRGNN, and we show that such senario can be prevented by simply using smaller propagaion step in the base model (APPNP/GPRGNN).**

### **Why is the proposed method useful?**

We note that for a same model, Tables 7 and 8 report Balanced Accuracy and Macro-F1, respectively. Evaluating the effectiveness of ToBE requires a comprehensive consideration of both aspects.

1. The scenario described in Q1 ("beat both ToBE$_0$ and ToBE$_1$") occurs in only **2 out of 105 experimental conditions** (3 datasets x 7 baselines x 5 GNN backbones). Specifically, 
   - For BAcc: on the PubMed dataset [APPNP, APPNP+ReNode];
   - For Macro-F1: on the Cora dataset [GPRGNN+Reweight, GPRGNN+ReNode].
2. **Jointly considering BAcc and Macro-F1, no baseline can outperform both ToBE$_0$ and ToBE$_1$**, e.g.,
   - For the PubMed dataset [APPNP, APPNP+ReNode]: Both ToBE$_0$ and ToBE$_1$ boosted their Macro-F1;
   - Similarly, for the Cora dataset [GPRGNN+Reweight, GPRGNN+ReNode]: Both ToBE$_0$ and ToBE$_1$ boosted their BAcc.
3. Our work provides one of the most comprehensive experimental setups in existing literature, covering 2 imbalance types, 4 imbalance ratios, combinations of the proposed method with 5 GNN architectures and 7 baseline methods. We believe that maintaining effectiveness in **over 98% of the cases (103/105) across all performance metrics** in such a comprehensive experimental setup demonstrates the usefulness of ToBE.

### **Case study on APPNP and GPRGNN.**

***The reason behind***: We now conduct a deeper analysis on the undesired cases we discussed above. As can be observed (and suggested in W1), these cases are related to APPNP and GPRGNN. We note that **both APPNP and GPRGNN involve multi-step message-passing** based on Personalized PageRank (PPR).  Additionally, due to the introduction of virtual edges, **ToBE-enhanced graphs exhibit improved connectivity**, potentially leading to **over-smoothing issues** when combined with these models. From this perspective, we also observed that:
1. Reweight and ReNode only reweight training nodes, so the issue persists.
2. Oversampling methods like Resample, GSMOTE, and GENS introduce additional nodes, alleviating over-smoothing. Therefore, ToBE performs better when combined with these methods (as can be observed in Tables 7 and 8).

***Empirical validation***: Furthermore, we conducted experiments to validate the above issues. As described in Appendix C.2, following the original papers' settings, we set the number of propagation steps K for APPNP/GPRGNN to 10. However, when combined with ToBE, we should reduce K to alleviate over-smoothing. We tested K=10/5/3/1 on the cases where ToBE failed, and the best results in each column are highlighted in bold. Detailed results are listed in the tables below.

We can observe that APPNP/GPRGNN typically achieve the best results with K=10. However, **when combined with ToBE, smaller values of K can cope with the increased graph connectivity, avoiding over-smoothing and outperforming the best results of the original models**. We will include a discussion of over-smoothing in the appendix and point out the caveats of using ToBE in conjunction with multi-step message-passing GNNs.

**PubMed - APPNP - Vanilla**

| Metric      | BAcc           |                |                | F1             |                |                |
| ----------- | -------------- | -------------- | -------------- | -------------- | -------------- | -------------- |
| #Prop. Step | Base           | +ToBE$_0$      | +ToBE$_1$      | Base           | +ToBE$_0$      | +ToBE$_1$      |
| k=10        | **59.30**±0.50 | 55.62±0.31     | 57.82±0.29     | 44.50±0.21     | 44.67±0.07     | 44.59±0.06     |
| k=5         | 59.07±0.22     | 58.56±0.14     | 59.00±0.51     | 44.70±0.08     | 45.69±0.49     | 44.68±0.15     |
| k=3         | 59.12±0.09     | 59.85±0.26     | 60.11±0.31     | **45.18±0.15** | 45.76±0.56     | 46.36±1.38     |
| k=1         | 58.97±0.11     | **59.99**±0.17 | **62.91**±1.91 | 45.01±0.08     | **49.09**±1.42 | **49.40**±4.38 |

Reviewer comment (Writing):
Some grammatical errors, like 1) groundtruth labels → “ground-truth”; 2) Coauthor networks → “co-author”; and 3) “Fig. 5 compares” → “Figure”.

Author response:
## (W2) Paper writing

We appreciate your meticulous review. We have fixed the errors and will carefully check and revise the entire manuscript.

**Thank you once again for your feedback. We hope we have addressed our concerns, and we welcome any follow-up question or discussion.**

Reviewer comment (Evaluation):
As shown in Tables 7 and 8, why can lots of baselines sometimes beat both ToBE0 and ToBE1? As such, why is the proposed method useful?

Author response:
## (W1, Q1) Regarding the effectiveness of ToBE

**TL;DR: Empirically, ToBE is effective in over 98% of the cases (103/105) across all performance metrics (Table 7/8/9). The rare cases that ToBE does not work well are caused by oversmoothing from the multi-step message-passing in APPNP/GPRGNN, and we show that such senario can be prevented by simply using smaller propagaion step in the base model (APPNP/GPRGNN).**

### **Why is the proposed method useful?**

We note that for a same model, Tables 7 and 8 report Balanced Accuracy and Macro-F1, respectively. Evaluating the effectiveness of ToBE requires a comprehensive consideration of both aspects.

1. The scenario described in Q1 ("beat both ToBE$_0$ and ToBE$_1$") occurs in only **2 out of 105 experimental conditions** (3 datasets x 7 baselines x 5 GNN backbones). Specifically, 
   - For BAcc: on the PubMed dataset [APPNP, APPNP+ReNode];
   - For Macro-F1: on the Cora dataset [GPRGNN+Reweight, GPRGNN+ReNode].
2. **Jointly considering BAcc and Macro-F1, no baseline can outperform both ToBE$_0$ and ToBE$_1$**, e.g.,
   - For the PubMed dataset [APPNP, APPNP+ReNode]: Both ToBE$_0$ and ToBE$_1$ boosted their Macro-F1;
   - Similarly, for the Cora dataset [GPRGNN+Reweight, GPRGNN+ReNode]: Both ToBE$_0$ and ToBE$_1$ boosted their BAcc.
3. Our work provides one of the most comprehensive experimental setups in existing literature, covering 2 imbalance types, 4 imbalance ratios, combinations of the proposed method with 5 GNN architectures and 7 baseline methods. We believe that maintaining effectiveness in **over 98% of the cases (103/105) across all performance metrics** in such a comprehensive experimental setup demonstrates the usefulness of ToBE.

### **Case study on APPNP and GPRGNN.**

***The reason behind***: We now conduct a deeper analysis on the undesired cases we discussed above. As can be observed (and suggested in W1), these cases are related to APPNP and GPRGNN. We note that **both APPNP and GPRGNN involve multi-step message-passing** based on Personalized PageRank (PPR).  Additionally, due to the introduction of virtual edges, **ToBE-enhanced graphs exhibit improved connectivity**, potentially leading to **over-smoothing issues** when combined with these models. From this perspective, we also observed that:
1. Reweight and ReNode only reweight training nodes, so the issue persists.
2. Oversampling methods like Resample, GSMOTE, and GENS introduce additional nodes, alleviating over-smoothing. Therefore, ToBE performs better when combined with these methods (as can be observed in Tables 7 and 8).

***Empirical validation***: Furthermore, we conducted experiments to validate the above issues. As described in Appendix C.2, following the original papers' settings, we set the number of propagation steps K for APPNP/GPRGNN to 10. However, when combined with ToBE, we should reduce K to alleviate over-smoothing. We tested K=10/5/3/1 on the cases where ToBE failed, and the best results in each column are highlighted in bold. Detailed results are listed in the tables below.

We can observe that APPNP/GPRGNN typically achieve the best results with K=10. However, **when combined with ToBE, smaller values of K can cope with the increased graph connectivity, avoiding over-smoothing and outperforming the best results of the original models**. We will include a discussion of over-smoothing in the appendix and point out the caveats of using ToBE in conjunction with multi-step message-passing GNNs.

**PubMed - APPNP - Vanilla**

| Metric      | BAcc           |                |                | F1             |                |                |
| ----------- | -------------- | -------------- | -------------- | -------------- | -------------- | -------------- |
| #Prop. Step | Base           | +ToBE$_0$      | +ToBE$_1$      | Base           | +ToBE$_0$      | +ToBE$_1$      |
| k=10        | **59.30**±0.50 | 55.62±0.31     | 57.82±0.29     | 44.50±0.21     | 44.67±0.07     | 44.59±0.06     |
| k=5         | 59.07±0.22     | 58.56±0.14     | 59.00±0.51     | 44.70±0.08     | 45.69±0.49     | 44.68±0.15     |
| k=3         | 59.12±0.09     | 59.85±0.26     | 60.11±0.31     | **45.18±0.15** | 45.76±0.56     | 46.36±1.38     |
| k=1         | 58.97±0.11     | **59.99**±0.17 | **62.91**±1.91 | 45.01±0.08     | **49.09**±1.42 | **49.40**±4.38 |

Reviewer comment (Theory):
“Similar analysis can extend to k ≥ 3.” — have you ever proved this?

Author response:
## (Q3) Theoretical analysis for k>3

As mentioned in the paper, our analysis can extend to $k\\ge3$, but the results will look messier. To demonstrate that, we provide a sketch analysis for general $k$ using the same ideas for $k=2$.

Note that the results of both Theorems 1 & 2 depend only on the distributions of $H_{ij}$. Thus, as long as we can calculate the distributions of $H_{ij}$, Theorems 1 & 2 follows. Indeed, the distributions of $H_{ij}$ for general $k$ can still be calculated via the BFS tree counting approach in our proof, i.e.,
$$
\\begin{aligned}
\\mathbb P\\{H_{11}=s\\}={}&\\sum_{a_1+\\cdots+a_k=s}\\binom{n_1-1}{a_1,\\dots,a_k,n_1-1-s}p^{a_1}\\bigg(\\prod_{t=2}^k(1-p)^{a_t(1+a_1+\\cdots+a_{t-2})}(1-(1-p)^{a_{t-1}})^{a_t}\\bigg)(1-p)^{(n_1-1-s)(1+s-a_k)}\\\\
\\sim{}&\\sum_{a_1+\\cdots+a_k=s}\\frac{n_1^s}{a_1!\\cdots a_k!}p^{a_1}\\bigg(\\prod_{t=2}^k(a_{t-1}p)^{a_t}\\bigg)(1-p)^{n_1(1+s-a_k)}\\\\
\\to{}&\\mathrm e^{-\\beta_{11}}\\sum_{a_1+\\cdots+a_k=s}\\frac{(\\beta_{11}\\mathrm e^{-\\beta_{11}})^{a_1}}{a_1!}\\bigg(\\prod_{t=2}^{k-1}\\frac{(a_{t-1}\\beta_{11}\\mathrm e^{-\\beta_{11}})^{a_t}}{a_t!}\\bigg)\\frac{(a_{k-1}\\beta_{11})^{a_k}}{a_k!},\\\\
\\mathbb P\\{H_{12}=s\\}={}&\\sum_{a_1+\\cdots+a_k=s}\\binom{n_2}{a_1,\\dots,a_k,n_2-s}q^{a_1}\\bigg(\\prod_{t=2}^k(1-q)^{a_t}(1-p)^{a_t(a_1+\\cdots+a_{t-2})}(1-(1-p)^{a_{t-1}})^{a_t}\\bigg)(1-q)^{n_2-s}(1-p)^{(n_2-s)(s-a_k)}\\\\
\\sim{}&\\sum_{a_1+\\cdots+a_k=s}\\frac{n_2^s}{a_1!\\cdots a_k!}q^{a_1}\\bigg(\\prod_{t=2}^k(a_{t-1}p)^{a_t}\\bigg)(1-q)^{n_2}(1-p)^{n_2(s-a_k)}\\\\
\\to{}&\\mathrm e^{-\\beta_{12}}\\sum_{a_1+\\cdots+a_k=s}\\frac{(\\beta_{12}\\mathrm e^{-\\beta_{22}})^{a_1}}{a_1!}\\bigg(\\prod_{t=2}^{k-1}\\frac{(a_{t-1}\\beta_{22}\\mathrm e^{-\\beta_{22}})^{a_t}}{a_t!}\\bigg)\\frac{(a_{k-1}\\beta_{22})^{a_k}}{a_k!}.
\\end{aligned}
$$
Regarding Theorem 1, note that for any $j=1,\\dots,k$,
$$
\\mathrm e^{-\\beta_{11}}\\sum_{a_1=0}^\\infty\\cdots\\sum_{a_k=0}^\\infty a_j \\frac{(\\beta_{11}\\mathrm e^{-\\beta_{11}})^{a_1}}{a_1!}\\bigg(\\prod_{t=2}^{k-1}\\frac{(a_{t-1}\\beta_{11}\\mathrm e^{-\\beta_{11}})^{a_t}}{a_t!}\\bigg)\\frac{(a_{k-1}\\beta_{11})^{a_k}}{a_k!}=\\beta_{11}^j.
$$
Thus,
$$
\\begin{aligned}
&\\lim_{n\\to\\infty}\\mathbb E[H_{11}]=\\sum_{s=0}^\\infty s\\cdot\\mathrm e^{-\\beta_{11}}\\sum_{a_1+\\cdots+a_k=s}\\frac{(\\beta_{11}\\mathrm e^{-\\beta_{11}})^{a_1}}{a_1!}\\bigg(\\prod_{t=2}^{k-1}\\frac{(a_{t-1}\\beta_{11}\\mathrm e^{-\\beta_{11}})^{a_t}}{a_t!}\\bigg)\\frac{(a_{k-1}\\beta_{11})^{a_k}}{a_k!}\\\\
={}&\\mathrm e^{-\\beta_{11}}\\sum_{a_1=0}^\\infty\\cdots\\sum_{a_k=0}^\\infty(a_1+\\cdots+a_k)\\frac{(\\beta_{11}\\mathrm e^{-\\beta_{11}})^{a_1}}{a_1!}\\bigg(\\prod_{t=2}^{k-1}\\frac{(a_{t-1}\\beta_{11}\\mathrm e^{-\\beta_{11}})^{a_t}}{a_t!}\\bigg)\\frac{(a_{k-1}\\beta_{11})^{a_k}}{a_k!}\\\\
={}&\\sum_{j=1}^k\\mathrm e^{-\\beta_{11}}\\sum_{a_1=0}^\\infty\\cdots\\sum_{a_k=0}^\\infty a_j\\frac{(\\beta_{11}\\mathrm e^{-\\beta_{11}})^{a_1}}{a_1!}\\bigg(\\prod_{t=2}^{k-1}\\frac{(a_{t-1}\\beta_{11}\\mathrm e^{-\\beta_{11}})^{a_t}}{a_t!}\\bigg)\\frac{(a_{k-1}\\beta_{11})^{a_k}}{a_k!}\\\\
={}&\\sum_{j=1}^k\\beta_{11}^j.
\\end{aligned}
$$
Similarly,
$$
\\begin{aligned}
\\lim_{n\\to\\infty}\\mathbb E[H_{22}]&=\\sum_{j=1}^k\\beta_{22}^{j},\\\\
\\lim_{n\\to\\infty}\\mathbb E[H_{12}]&=\\sum_{j=1}^k\\beta_{12}\\beta_{22}^{j-1},\\\\
\\lim_{n\\to\\infty}\\mathbb E[H_{21}]&=\\sum_{j=1}^k\\beta_{21}\\beta_{11}^{j-1}.
\\end{aligned}
$$
Hence,
$$
\\lim_{n\\to\\infty}\\frac{\\alpha_1}{\\alpha_2}=\\lim_{n\\to\\infty}\\frac{\\mathbb E[H_{12}]/\\mathbb E[H_{11}]}{\\mathbb E[H_{21}]/\\mathbb E[H_{22}]}=\\frac{\\sum_{j=1}^k\\beta_{12}\\beta_{22}^{j-1}\\big/\\sum_{j=1}^k\\beta_{11}^j}{\\sum_{j=1}^k\\beta_{21}\\beta_{11}^{j-1}\\big/\\sum_{j=1}^k\\beta_{22}^j}=\\bigg(\\rho\\cdot\\frac{\\sum_{j=1}^k(\\rho\\beta)^{j-1}}{\\sum_{j=1}^k\\beta^{j-1}}\\bigg)^2.
$$
Regarding Theorem 2, similar analysis applies, but the results get much messier as $k$ grows. For example, for $k=3$, one of the key quantities in the proof of Theorem 2 is
$$
\\lim_{n\\to\\infty}\\mathbb E[(1-r_1^{\\text L})^{H_{11}}]=\\mathrm e^{-\\left(1-(1-r_1^{\\text L})\\beta_{11}\\mathrm e^{-\\left(1-(1-r_1^{\\text L})\\beta_{11}\\mathrm e^{-r_1^{\\text L}\\beta_{11}}\\right)\\beta_{11}}\\right)\\beta_{11}}.
$$
In general, it has $k$ nested exponentiations, which looks messy. Meanwhile, if we calculate the closed-form expression of $\\lim_{n\\to\\infty}\\frac{\\delta_1}{\\delta_2}$ numerically, then the result for any $k$ will be almost the same as that of $k=2$, i.e.,
$$
\\lim_{n\\to\\infty}\\frac{\\delta_1}{\\delta_2}\\approx\\frac{1-r_1^{\\text L}}{1-r_2^{\\text L}}\\mathrm e^{(\\rho-1)\\beta}.
$$
That is why we previously chose $k=2$ as a representative case.

Reviewer comment (Presentation):
In Table 3, why does the “Node” column only have one type of results?

Author response:
## (Q4) The "Node" column in Table 3

As described in Section 3.3, the process of synthesizing virtual nodes for both ToBE$_0$ and ToBE$_1$ is identical: **each class generates one virtual node**. Hence, the "Node" column in Table 3 only presents a single type of result.

Reviewer comment (Evaluation):
The use of "relative improvement rate" may not be the most comprehensive measure to evaluate the model's performance, as some base models may inherently perform poorly in addressing highly imbalanced class semi-supervised vertex classification tasks.

Author response:
**(W4) On the usage of "relative improvement rate"**

**The "relative improvement rate" reported in our main results (Table 1) pertains to combinations of base model + CIGL method, rather than the base models themselves that may inherently perform poorly.** This ensures an objective reflection of the performance enhancement attributable to the ToBE method.

> For example, the relative gain for **GCN+ReNode+ToBE** is calculated relative to the performance of **GCN+ReNode**, rather than **GCN** itself (which may perform poorly by itself in CIGL task, as you suggested).

**In Table 2, the "relative improvement rate" was reported for both the best CIGL method and ToBE.** This is can fairly and objectively shows the performance gain brought about by the best CIGL method and ToBE. To better address your concern, we will include a more intuitive metric "performance gain" (Perf(A+ToBE) - Perf(A)), facilitating a clearer understanding of the performance advantage conferred by ToBE. This modification should enhances the transparency and interpretability of our results.

Reviewer comment (Theory):
The details of how this module collaborates with other base models are not adequately explained, lacking formulas and clear visual representations.

Author response:
**(Q3.1, W5) How ToBE works with other CIGL methods**
Thank you for bringing this to our attention. We will include a detailed discussion in the paper.

**In each training epoch, ToBE augments the original graph $\mathbf{G}$ based on the current model $\mathbf{F}$ and returns the augmented graph $\mathbf{G}^*$. Other CIGL methods then operate on $\mathbf{G}^*$ for subsequent steps**, specifically:
- Loss function engineering methods (Reweight and ReNode) perform the loss computing and backpropagation based on $\mathbf{G}^*$.
- Data augmentation methods (Resampling, SMOTE, GSMOTE, GENS) will conduct further class-balancing operations on $\mathbf{G}^*$, i.e., they will generate new minority nodes based on $\mathbf{G}^*$.

Reviewer comment (Experiments):
How does the effectiveness of this module compare to post-processing modules of other class imbalance graph learning methods?

Author response:
**(W1, Q1) ToBE is not a post-processing module:** We would like to highlight that **ToBE works during the training phase**, thus should be considered as an **in-processing** technique. Specifically,  in each training step, ToBE takes the current model $F$ and original graph $\mathbf{G}$ as input and output augmented graph $\mathbf{G}^*$. Please refer to Algorithm 1 in Appendix B for more details. 

**(Q2) We tested ToBE with multiple base models combined with data balancing techniques:**
As ToBE performs augmentation on the topology structure, it is orthogonal to (and thus can be combined with) other data balancing techniques: they can operate on the augmented graph $\mathbf{G}^*$ for subsequent steps. In the paper we have conducted a comprehensive comparison with relevant data balancing methods, including approaches handling class imbalance by synthesizing new samples (e.g., GraphSMOTE [1], GraphENS [2]) and addressing topological imbalance through modifying loss functions (e.g., ReNode [3]).

> **For example, in Table 1, the row GCN-ReNode corresponds to GCN base model with ReNode [3] data balancing**. The column "Base" report the performance of GCN+ReNode, and the columns "+ToBE$_0$"/"+ToBE$_1$" report the performance after futher incorporating with ToBE. Results show that ToBE boosted the balanced accuracy of GCN+ReNode from 66.60 to 71.37/71.84. Please also refer to Table 7/8/9 in Appendix E for more detailed results.


**(Q1, Q2) On potential counterparts:** Other than data augmentation and loss function engineering, existing CIGL methods primarily focus on adversarial generation [4,5], pseudo-labeling [6,7], model refinement [8, 9], and margin adjustment [10], while very few post-processing methods (e.g., data balancing during prediction) exists. We have carefully looked through existing literature and recent surveys [11,12], but faced challenges in finding works that apply *residual propagation* **(Q1)** or *test-time data balancing* **(Q2)** for class-imbalanced node classification. **We would greatly appreciate it if you could provide specific references, and we are more than willing to incorporate relevant experiments and analyses based on your suggestions.**

Reviewer comment (Experiments):
How does the performance of this module compare to the base model combined with data balancing during prediction?

Author response:
**(W2, W3) Test on balanced data**
**We note that CIGL inherently addresses the challenges posed by imbalanced datasets, thus this line of research (e.g., [1-10]) does not consider balanced data as an useful testbed. We provide additional discussions and experiments to illustrate the reasons behind.**

**CIGL degeneratess into normal graph learning on perfectly balanced data**: When data is perfectly balanced (all classes have an equal number of samples), existing CIGL methods tend to degenerate into conventional graph learning processes. For instance, since there are no minority/majority class distinctions, oversampling methods like GraphSMOTE and GraphENS would generate no nodes, while reweighting methods would assign equal weights to each class, reducing them to standard graph learning. Becasue of this, CIGL research works do not conduct experiments on balanced data as this is considered beyond the problem scope.

**Experiments on balanced datasets:** Please see the Table below. It can be observed that on balanced datasets, similar to other CIGL techniques, our method exhibits minimal impact on model training and performance remains consistent. This is attributed to (1) the smaller uncertainty of the base model and (2) the absence of imbalance-aware risk calibration.

| Metric  | BAcc         |              |              | F1           |              |              |
| ------- | ------------ | ------------ | ------------ | ------------ | ------------ | ------------ |
| Dataset | Cora         | CiteSeer     | PubMed       | Cora         | CiteSeer     | PubMed       |
| Vanilla GCN     | 81.67±0.51 | 65.28±0.54 | 77.94±0.27 | 78.89±0.54 | 64.85±0.52 | 76.01±0.14 |
| GCN+ToBE_0  | 81.88±0.35 | 65.24±0.39 | 77.97±0.15 | 79.56±0.38 | 64.88±0.32 | 76.78±0.36 |
| GCN+ToBE_1  | 81.90±0.56 | 65.66±0.37 | 77.89±0.32 | 78.97±0.88 | 65.25±0.37 | 76.61±0.27 |

Reviewer comment (Theory):
What are the specific formulas and graphical representations illustrating the collaboration between this module and the base model? Can it be independently developed as a foundational model rather than a post-processing module?

Author response:
**(Q3.1, W5) How ToBE works with other CIGL methods**
Thank you for bringing this to our attention. We will include a detailed discussion in the paper.

**In each training epoch, ToBE augments the original graph $\mathbf{G}$ based on the current model $\mathbf{F}$ and returns the augmented graph $\mathbf{G}^*$. Other CIGL methods then operate on $\mathbf{G}^*$ for subsequent steps**, specifically:
- Loss function engineering methods (Reweight and ReNode) perform the loss computing and backpropagation based on $\mathbf{G}^*$.
- Data augmentation methods (Resampling, SMOTE, GSMOTE, GENS) will conduct further class-balancing operations on $\mathbf{G}^*$, i.e., they will generate new minority nodes based on $\mathbf{G}^*$.

## (Q3.2) Can it be independently developed as a foundational model rather than a post-processing module?
**ToBE is a model-agnostic in-processing module that can be boost any foundation GNN model (e.g., GCN, GAT, GraphSAGE, APPNP, GPRGNN, etc). It is not a post-processing module nor a foundation GNN model itself.**

As described in the paper, ToBE is an in-processing data augmentation technique. It dynamically augments the data during training, with no dependence on the base model used. Our extensive experiments confirmed the versatility of ToBE on various GNN backbones.

Reviewer comment (Experiments):
Can this module be applied to other tasks and demonstrate effectiveness? How does it perform in terms of inductive capabilities?

Author response:
**(Q4.1, W6) ToBE's scope and potential in other tasks**
We appreciate your valuable feedback regarding potential avenues for further research. 

**We note that the scope of this study aligns with the existing CIGL research** (e.g., [1] in WSDM'21, [2] in ICLR'22, [10] in ICML'22), **and it has widespread application in real-world scenarios**, e.g., outlier detection, fraudulent user detection, and social spammer detection [12].

**We believe that our work may benefit tasks in other dedicated lines of research, but such extension is non-trivial.** For example, imbalanced edge prediction is an important task especially in domains like knowledge graph completion. However, extending existing CIGL concepts to address this task is non-trivial as it involves multi-facet imbalance: e.g., node-level (quantity diffenrent between different types of entities), edge-level (relation existence/non-existence), and degree-level (certain entities having more relationships). We recognize this as an important future direction and will discuss more possible future directions in the paper.

## (Q4.2) How does ToBE perform in terms of inductive capabilities?

**TL;DR: ToBE retifies minority misclassifications, and thus help GNN to learn better representations for minority classes even on unseen graphs. We conducted further experiments and confirm that ToBE can improve GNN's inductive learning capabilities.**

This is an exciting problem. Following your suggestion, we test ToBE's incductive capabilities on the [protein-protein interaction networks (PPI)](https://pytorch-geometric.readthedocs.io/en/latest/generated/torch_geometric.datasets.PPI.html#torch_geometric.datasets.PPI). The task is to predict protein functions on **unseen proteins (graphs)**. The PPI dataset contains multiple distinct protein function targets, forming binary classification tasks with varying class imbalance ratios. This setup serves as a robust testbed for ToBE's capabilities in addressing class imbalance on unseen graphs. We show the results in the table below (PPI-X denotes using the X-th protein function as the target):

**Balance Accuracy on PPI**:
| Task   | PPI-1 (IR=2.054)   | PPI-2 (IR=3.051)   | PPI-3 (IR=3.835)   | PPI-4 (IR=3.010)   | PPI-5 (IR=6.963)   |
| ------ | ------------------ | ------------------ | ------------------ | ------------------ | ------------------ |
| GCN    | 73.52±1.26         | 72.06±3.81         | 69.79±2.82         | 74.61±3.16         | 71.70±3.85         |
| GCN+ToBE_0 | **74.89**±1.64 (+1.36) | 75.47±2.34 (+3.41) | 70.17±3.20 (+0.38) | **77.64**±2.76 (+3.03) | 74.98±3.98 (+3.28) |
| GCN+ToBE_1 | 73.97±1.44 (+0.44) | **76.56**±2.82 (+4.49) | **73.43**±3.14 (+3.63) | 76.34±3.13 (+1.73) | **75.12**±3.74 (+3.42) |

**Macro F1 score on PPI**:
| Task   | PPI-1 (IR=2.054)   | PPI-2 (IR=3.051)   | PPI-3 (IR=3.835)   | PPI-4 (IR=3.010)   | PPI-5 (IR=6.963)   |
| ------ | ------------------ | ------------------ | ------------------ | ------------------ | ------------------ |
| GCN    | 73.85±0.34         | 74.30±3.22         | 71.84±1.64         | 75.33±1.77         | 73.18±1.92         |
| GCN+ToBE_0 | **75.23**±0.80 (+1.39) | 77.07±1.07 (+2.77) | 72.73±2.24 (+0.89) | **76.85**±1.34 (+1.51) | **76.33**±1.49 (+3.15) |
| GCN+ToBE_1 | 74.86±0.91 (+1.01) | **77.58**±1.54 (+3.28) | **74.90**±1.62 (+3.06) | 76.48±1.40 (+1.15) | 76.28±1.76 (+3.11) |

For each task, we use the first 5 graphs for training, 5 for validation, and test the model on the remaining 10 graphs. Results are averaged over 5 runs with different random seeds. The results show that **ToBE can help GNN to better generalize to unseen graphs**, especially for tasks with high class imbalance ratios (e.g., PPI-5). We attribute this to TOBE's ability to retify minority misclassifications, and thus help GNN to learn better representations for minority classes. A more detailed discussion will be included in the paper.

Reviewer comment (Theory):
I find the theoretical analysis has nothing to do with the model design. In the theoretical analysis, this work only analyzes the relation between the imbalance ratio and the severity of AMP and DMP. However, in the model design, this work directly uses model prediction uncertainty to estimate nodes’ risk of being misclassified and simply claims that this risk of being misclassified is due to AMP/DMP, which is not verified in the theoretical part. From my view, there is not any relationship between these two parts. Even without the theoretical part, the model design part looks self-contained.

Author response:
## (W2) From theoretical results to algorithm design

**TL;DR: Armed with the theoretical results, we investigated AMP/DMP on real-world graphs and found that the impact of AMP/DMP is concentrated on a small fraction of “critical” nodes. This provides us a way to surrogate the infeasible \rho (imbalance ratio) manipulation, and inspired the development of ToBE. Our experimental results confirmed ToBE's ability in alleviating both AMP and DMP (e.g., Figure 6).**

Thank you for your careful and thoughtful review of our paper. As discussed in **Section 2 - "A closer look at AMP & DMP in practice"**, while the imbalance ratio ($\rho$) determines the severity of bias introduced by AMP/DMP, directly altering $\rho$ is not feasible (oversampling methods synthesize new samples based on existing observations, which is fundamentally different from sampling from the underlying distribution). Consequently, we further investigated the distribution of AMP/DMP on real-world graphs and found that the impact of AMP/DMP is concentrated on a small fraction of “critical” nodes. This insight inspired the development of ToBE, and experimental results demonstrate its efficient alleviation of AMP and DMP (e.g., Figure 6).

> In this work, we aimed for a lightweight and broadly applicable algorithm. We acknowledge the need for a more in-depth exploration of ToBE's properties and also more informed algorithm design in future work. Your feedback is greatly appreciated and will consider it in refining our approach and conducting more detailed analyses in subsequent research.

Reviewer comment (Evaluation):
In Figure 2(b), it seems that there is still a large discrepancy between the performance of the minority class and the majority class, even if the AMP/DMP score is the same. That indicates that there are some other factors that influence the performance discrepancy. I would expect the authors to further clarify this.

Author response:
## (W4) The other factor behind performance discrepancy

Great question! **TL;DR: The discrepancy between the two curves in Figure 2(b) stems from the poor representation of minority classes in the feature space, which is caused by the "class imbalance" itself.**

Consider a graph with features $\mathbf{X}$ and adjacency matrix $\mathbf{A}$. While AMP/DMP are factors related to the topological structure ($\mathbf{A}$), class imbalance also significantly influences the representation quality of different classes in the feature space ($\mathbf{X}$).  Therefore, a model learn from $\mathbf{X}$ (i.e., traditional i.i.d. imbalance learning) will inherently exhibit poorer generalizability for minority classes [5], and thus minority classes are more sensitive to noises [6, 7]. Additionally, recent research suggests that GNN can be viewed as a Graph Signal Denoising [8, 9] that denoise $\mathbf{X}$ with a low-pass filter based on $\mathbf{A}$. From this perspective, we see that AMP introduces additional noise from dissimilar nodes, and DMP leads to less efficient label propagation/denoising for a specific node. Both factors hinders the graph denoising process, and thus minority class nodes (which are more sensitive to noise) are more prone to the influence of AMP/DMP (i.e., experiencing more performance degradation at the same AMP/DMP level).

> In essence, Figure 2(a) illustrates the proportion of high AMP/DMP nodes for majority/minority classes, while Figure 2(b) demonstrates the sensitivity of majority/minority classes to AMP/DMP. The discussions were initially included, but were omitted due to space constraints. We will take this discussion back in the paper to better illustrate the results.

Reviewer comment (Experiments):
The existing baselines lack comprehensiveness, and there is a lack of thorough comparison with recent approaches, such as [1, 2, 3, 4, 5].

[1] Imgcl: Revisiting graph contrastive learning on imbalanced node classification. AAAI 2023

[2] Balanced neighbor exploration for semi-supervised node classification on imbalanced graph data. Information Sciences 2023

[3] Graphmixup: Improving class-imbalanced node classification by reinforcement mixup and self-supervised context prediction. ECML-PKDD 2022

[4] Imbalanced node classification beyond homophilic assumption. IJCAI 2023

[5] Tam: Topology-aware margin loss for class-imbalanced node classification. ICML 2022

Author response:
## (W5) Comparison with latest baselines

Thank you for the suggestions. **We have included more results comparing ToBE with the suggested CIGL baselines.**

Given the need in our experiments to test ToBE with a comprehensive combination of existing CIGL techniques and GNN backbones, we chose to include model-agnostic and efficient CIGL methods for a thorough evaluation.

Nevertheless, we thoroughly investigated the literature you provided and implemented [BNE](https://github.com/ZonghaiZhu/BNE) [11], [GraphMixup](https://github.com/LirongWu/GraphMixup) [12], and [TAM](https://github.com/Jaeyun-Song/TAM) [14] (the other two papers [10][13] did not provide source code). To ensure a fair comparison, we implemented these methods in a unified experimental framework, employing the same data splitting strategy as in our paper and reporting the averages over five independent runs (i.e., ensuring comparability with the results reported in our paper). The results for Imbalance Ratio=10/20 are as follows:

**Imbalance Ratio = 10**

|Metric|BAcc|||F1|||Disp|||
|-|-|-|-|-|-|-|-|-|-|
|Dataset|Cora|CiteSeer|PubMed|Cora|CiteSeer|PubMed|Cora|CiteSeer|PubMed|
|GCN|0.619|0.376|0.642|0.607|0.281|0.551|0.275|0.299|0.347|
|BNE|0.651|0.515|0.654|0.674|0.497|0.613|0.204|0.155|0.245|
|TAM|0.620|0.381|0.671|0.610|0.290|0.607|0.296|0.324|0.308|
|GraphMixup|0.641|0.508|OOM|0.627|0.473|OOM|0.245|0.252|OOM|
|ToBE_0|0.655|0.527|**0.686**|0.633|0.516|**0.672**|0.213|**0.139**|**0.092**|
|ToBE_1|**0.698**|**0.554**|0.676|**0.687**|**0.550**|0.644|**0.185**|**0.139**|0.218|

**Imbalance Ratio = 20**

|Metric|BAcc|||F1|||Disp|||
|-|-|-|-|-|-|-|-|-|-|-|
|Dataset|Cora|CiteSeer|PubMed|Cora|CiteSeer|PubMed|Cora|CiteSeer|PubMed|
|GCN|0.619|0.376|0.642|0.607|0.281|0.551|0.275|0.299|0.347|
|BNE|0.651|0.515|0.654|0.674|0.497|0.613|0.204|0.155|0.245|
|TAM|0.620|0.381|0.671|0.610|0.290|0.607|0.296|0.324|0.308|
|GraphMixup|0.641|0.508|OOM|0.627|0.473|OOM|0.245|0.252|OOM|
|ToBE_0|0.655|0.527|**0.686**|0.633|0.516|**0.672**|0.213|**0.139**|**0.092**|
|ToBE_1|**0.698**|**0.554**|0.676|**0.687**|**0.550**|0.644|**0.185**|**0.139**|0.218|

It can be observed that ToBE still exhibits advantage compared to these methods. 
We also note that GraphMixup uses the dense adjacency matrix for computation, which leads to out-of-memory (OOM) issues on the PubMed dataset (with Tesla V100 32GB GPU).
We will include relevant discussions in the paper.

> We would like to emphasize that due to the properties of these methods, we did not combine ToBE with them but conducted a direct comparison. BNE preprocesses graph data to obtain independent features and directly uses MLP for prediction, making it incompatible with ToBE and existing GNNs. GraphMixup and TAM rely on invariant graph topology, and the code implementation for GraphMixup does not support the addition of virtual nodes to the original graph.