Paper: Understanding Community Bias Amplification in Graph Representation Learning (ICLR-2024)

Reviewer comment (Theory):
I do not agree with the authors claim on where the bias comes from. The datasets for illustrative examples in Fig.1 contain the independent components. Oono and Suzuki (2020) and its earlier work [1] shed a light on the phenomenon over-smoothing -- if we stack layers, the stacked adjacency matrix is dominant by the eigenspace associated with the largest eigenvalues. In the Cora and Citeseer cases, that space is a set of indicate vectors of independent components. Also, the claim is this worsens the performance, which is understandable, since the only this eigenspace of the independent components are too simplified as an underlying structure. However, the underlying graph structure does not necessarily correspond to the classes, while of course graph structure and the classes are loosely related. If they are, we observe somewhat comparative performance only using the graph of Cora and Citeseer, but we do not observe such performance by conducting for example the simple spectral clustering on graph. Thus, the community amplification bias of the underlying graphs is not the primal reason why we observe the unfairness of Fig. 1. Instead, I believe that the bias is more nuanced -- hope to see what is the dominant.

Author response:
**W1.1: I do not agree with the authors claim on where the bias comes from. The datasets for illustrative examples in Fig.1 contain the independent components.**

In our analysis, we always assume the graph is connected. Different communities in our analysis are not disconnected, but are clusters in a connected graph. Although real-world graph datasets may consist of multiple components, there is always a main component (the largest connected subgraph) and this largest connected subgraph contains the vast majority of nodes. Our analysis interprets the reason why community bias amplification occurs on this single component. This conclusion also holds for graphs with multiple connected components, as each component can be analyzed in the same way independently. 

**W1.2: Oono and Suzuki (2020) and its earlier work [1] shed a light on the phenomenon over-smoothing -- if we stack layers, the stacked adjacency matrix is dominant by the eigenspace associated with the largest eigenvalues. In the Cora and Citeseer cases, that space is a set of indicate vectors of independent components. Also, the claim is this worsens the performance, which is understandable, since the only this eigenspace of the independent components are too simplified as an underlying structure.**

As clarified above, our analysis is for a single connected graph. So the comments here actually misinterprets our theoretical analysis. Moreover, we have emphasized in the submission that existing spectral analysis on over-smoothing, which assumes the number of layers goes to infinity, is not suitable in our setting. Here, we would like to clarify more on this issue.

**Distinct from oversmoothing.** We want to emphasize that what we are studying are the phenomena that occur **when oversmoothing does not happen**. Oversmoothing is a phenomenon that is only observed when there are enough stacked adjacency matrices present. In our theoretical analysis, we are considering more realistic shallow GCN encoders. This is also in line with reality, as in common GCL models, GCN encoders usually only have few layers of graph convolution, at which point oversmoothing does not occur. At a high level, our justification for bias amplification is that when oversmoothing has not occurred, the level of smoothness at different regions of the graph could differ significantly due to disparities in local structures. Overall, the reviewer's hypothesis that community bias amplification could be caused by oversmoothing is not valid.

**W1.3: If they are, we observe somewhat comparative performance only using the graph of Cora and Citeseer, but we do not observe such performance by conducting for example the simple spectral clustering on graph. Thus, the community amplification bias of the underlying graphs is not the primal reason why we observe the unfairness of Fig. 1. Instead, I believe that the bias is more nuanced -- hope to see what is the dominant.**


These GCL models are usually applied to homophily graphs, in which intra-class edges significantly outnumber inter-class edges, hence the graph structure and classes are strongly correlated. 

What we want to express through Figure 1 is that the performance gap $\Delta_1$ of the GCL model is significantly higher than the performance gap $\Delta_0$ of MLP. We are studying the reasons why $\Delta_1 > \Delta_0$, and how to reduce this community bias amplification, but not the main reason for $\Delta_0$.

Reviewer comment (Theory):
Also, even if the community bias were primal reason, the argument of Eq. (3) is weak since they only compare the value of the eigenvalues of two clusters. Also, how the example of Appendix A reflects the Cora and Citeceer dataset? Do we observe such things in the real datasets? How do you argue that?

Author response:
**W2:Also, even if the community bias were primal reason, the argument of Eq. (3) is weak since they only compare the value of the eigenvalues of two clusters. Also, how the example of Appendix A reflects the Cora and Citeceer dataset? Do we observe such things in the real datasets? How do you argue that?**

After recognizing the community bias **amplification** in GCLs, we summarize the evidences for this phenomenon as three folds:

(1) Eq 3 shows that a upper bound of the representation density shrinks with a rate that depends on the **second** largest eigenvalue. We would like to point out that the rationale on two clusters can also be extended to multiple clusters with one-vs-the-rest (OvR) multiclass strategy. Secondly, The comparison of eigenvalues is not ``weak" due to the structural properties they imply [1], and it is widely used in the analysis of GNNs [2-5].

(2) **For random graphs:** In appendix A, we show the proof for bias amplification on the CSBM. Random graphs like CSBM is widely adopted for GNN analysis (e.g., [6-8]), and it is very suitable for illustrating bias amplification. 

(3) **Emprically:** We would like to highlight Table 2 in our original submission. In Table 2, we show the standard deviation of community density, computed on the node representations produced by various GCL models. It is clear that, for the embeddings of baseline models, **densities differ a lot among communities.** This is a direct evidence of community bias on two real datasets, Cora and Citeceer. Furthermore, it shows that our method RGCCL effectively mitigates this issue.

Reviewer comment (Theory):
How do you defend that the community amplification bias of the underlying graph is the primal reason we observe an unfairness between classes? As stated in the weakness section, I feel like there exists some gap between them.

Author response:
**Q1: How do you defend that the community amplification bias of the underlying graph is the primal reason we observe an unfairness between classes? As stated in the weakness section, I feel like there exists some gap between them.**

In summary, we discover a phenomenon called community bias amplification: After applying a **shallow** GNN, the representation densities of communities differs a lot due to their distinct structure (which is validated both theoretically and empirically, c.f., our response to W2). Different representation densities are proved to be unfair in Proposition 1 and Figure 3. The reviewer's concern on oversmoothing and eigenvalues are carefully addressed in our response to W1.2 and W2, respectively: (1) oversmoothing will not occur in **shallow** GNN; (2) the **second** largest eigenvalue contains rich information of structural properties, thus comparison on it is not ''weak'' and is widely adopted in the literature.

Reviewer comment (Theory):
One of the research questions has not been answered in a good way, i.e., why community bias amplification exists in existing GCL method? Although some theoretical analyses have been provided in this paper, they are based on general (and simplified) graphs and it is not clear why there is such bias in GCL methods.

Author response:
**W1\&Q1: One of the research questions has not been answered in a good way, i.e., why community bias amplification exists in existing GCL method? Although some theoretical analyses have been provided in this paper, they are based on general (and simplified) graphs and it is not clear why there is such bias in GCL methods.**

Thanks for the comments. Yes, our analysis is for embeddings produced by general GCN encoders, which provides insights on why community bias amplification exists in typical GCN encoders. Since most mainstream and competitive GCL models all use GCN and its variants as the encoder; thus current GCL models suffers community bias amplification. 

Likewise, supervised GNN as long as GCN is used as the encoder, community bias amplification will occur. The reason why we only focus on GCL is that supervised GNNs is also provided with label information at training time, and thus various techniques designed for handling class imbalance in supervised GNNs can be effectively applied, e.g., TAM [1] and GraphENS [2]. On the contrary, since there is no label information during the training process for unsupervised GCL, it is more difficult to mitigate the imbalance of embeddings from GCN encoders. And our paper is the first work that explicitly identifies this problem in GCL and provides an effective solution.

Reviewer comment (Evaluation):
How to quantitatively measure the (community) bias amplification is not clear in the experiments. More analysis should be conducted to better illustrate why the proposed method is able to mitigate the issue of bias amplification. For example, some measures [1] can be used for the quantitative analysis.

Author response:
**W2\&Q2: It is possible to give some quantitative analysis on how the proposed method can mitigate the issue of community bias amplification rather than simply comparing the performance between different classes/communities?**

Thanks for the question, and we agree that simply comparing the performance is not enough. We would like to highlight **Table 2 in our original submission.** In this table, we computed representation densities for each communities, and reported the average and standard deviation of these densities among communities. A lower standard deviation means that different communities are with similar degree of concentration, which is more fair for classification as illustrated in Figure 3 and Proposition 1 in our original submission. The results in Table 2 quantitatively shows that the proposed method RGCCL can mitigate the issue of community bias amplification (which is also proved by Lemma 4.1 in our original submission).



Since the metrics in [3] are primarily about fairness and causal inference, they are not applicable to our task. Furthermore, suggested by reviewer K8c3, we conducted experiments using the  Matthew's coefficient to measure bias. We present the results of representative GCL models and our RGCCL regarding the Matthew's coefficient in Table 1. The results also demonstrate the effectiveness of RGCCL in mitigating the amplification of community bias.

Table 1: Matthew’s coefficient for RGCCL and baselines.

|         | Cora         | CiteSeer     | PubMed       |
| ------- | ------------ | ------------ | ------------ |
| DGI     | 73.0$\pm$1.5 | 64.5$\pm$1.2 | 57.8$\pm$4.3 |
| GRACE   | 67.9$\pm$1.6 | 60.0$\pm$2.0 | 62.3$\pm$3.8 |
| CCA-SSG | 74.5$\pm$1.6 | 64.6$\pm$1.4 | 64.0$\pm$4.1 |
| GGD     | 77.0$\pm$1.6 | 64.9$\pm$1.2 | 63.7$\pm$4.1 |
| GRADE   | 75.7$\pm$1.5 | 62.1$\pm$1.3 | 58.1$\pm$3.4 |
| RGCCL   | 78.9$\pm$0.9 | 66.3$\pm$0.8 | 65.6$\pm$4.2 |

Reviewer comment (Theory):
What are the answers to the first research question, i.e., why community bias amplification exists in existing GCL method?

Author response:
**W1\&Q1: One of the research questions has not been answered in a good way, i.e., why community bias amplification exists in existing GCL method? Although some theoretical analyses have been provided in this paper, they are based on general (and simplified) graphs and it is not clear why there is such bias in GCL methods.**

Thanks for the comments. Yes, our analysis is for embeddings produced by general GCN encoders, which provides insights on why community bias amplification exists in typical GCN encoders. Since most mainstream and competitive GCL models all use GCN and its variants as the encoder; thus current GCL models suffers community bias amplification. 

Likewise, supervised GNN as long as GCN is used as the encoder, community bias amplification will occur. The reason why we only focus on GCL is that supervised GNNs is also provided with label information at training time, and thus various techniques designed for handling class imbalance in supervised GNNs can be effectively applied, e.g., TAM [1] and GraphENS [2]. On the contrary, since there is no label information during the training process for unsupervised GCL, it is more difficult to mitigate the imbalance of embeddings from GCN encoders. And our paper is the first work that explicitly identifies this problem in GCL and provides an effective solution.

Reviewer comment (Evaluation):
It is possible to give some quantitative analysis on how the proposed method can mitigate the issue of community bias amplification rather than simply comparing the performance between different classes/communities?

Author response:
**W2\&Q2: It is possible to give some quantitative analysis on how the proposed method can mitigate the issue of community bias amplification rather than simply comparing the performance between different classes/communities?**

Thanks for the question, and we agree that simply comparing the performance is not enough. We would like to highlight **Table 2 in our original submission.** In this table, we computed representation densities for each communities, and reported the average and standard deviation of these densities among communities. A lower standard deviation means that different communities are with similar degree of concentration, which is more fair for classification as illustrated in Figure 3 and Proposition 1 in our original submission. The results in Table 2 quantitatively shows that the proposed method RGCCL can mitigate the issue of community bias amplification (which is also proved by Lemma 4.1 in our original submission).



Since the metrics in [3] are primarily about fairness and causal inference, they are not applicable to our task. Furthermore, suggested by reviewer K8c3, we conducted experiments using the  Matthew's coefficient to measure bias. We present the results of representative GCL models and our RGCCL regarding the Matthew's coefficient in Table 1. The results also demonstrate the effectiveness of RGCCL in mitigating the amplification of community bias.

Table 1: Matthew’s coefficient for RGCCL and baselines.

|         | Cora         | CiteSeer     | PubMed       |
| ------- | ------------ | ------------ | ------------ |
| DGI     | 73.0$\pm$1.5 | 64.5$\pm$1.2 | 57.8$\pm$4.3 |
| GRACE   | 67.9$\pm$1.6 | 60.0$\pm$2.0 | 62.3$\pm$3.8 |
| CCA-SSG | 74.5$\pm$1.6 | 64.6$\pm$1.4 | 64.0$\pm$4.1 |
| GGD     | 77.0$\pm$1.6 | 64.9$\pm$1.2 | 63.7$\pm$4.1 |
| GRADE   | 75.7$\pm$1.5 | 62.1$\pm$1.3 | 58.1$\pm$3.4 |
| RGCCL   | 78.9$\pm$0.9 | 66.3$\pm$0.8 | 65.6$\pm$4.2 |

Reviewer comment (Theory):
Discussion on extending this key idea to general GRL. I asked this question because the theoretical analysis is general enough on any GRL methods.

Author response:
**Q3:Discussion on extending this key idea to general GRL. I asked this question because the theoretical analysis is general enough on any GRL methods.**



As we answered in Q1, for general GRLs, such as supervised GCN, GAT models, etc., since we can use label information during the training process, we can refer to methods like TAM [1], GraphENS [2], etc., to mitigate the community bias amplification. As for classic unsupervised general GRLs like Node2vec [4] and SDNE [5], since they do not use GCN encoders, community bias amplification does not exist.