Uncovering the Structural Fairness in Graph Contrastive Learning
Abstract
Recent studies show that graph convolutional network (GCN) often performs worse for low-degree nodes, exhibiting the so-called structural unfairness for graphs with long-tailed degree distributions prevalent in the real world.
Graph contrastive learning (GCL), which marries the power of GCN and contrastive learning, has emerged as a promising self-supervised approach for learning node representations.
How does GCL behave in terms of structural fairness?
Surprisingly, we find that representations obtained by GCL methods are already fairer to degree bias than those learned by GCN.
We theoretically show that this fairness stems from intra-community concentration and inter-community scatter properties of GCL, resulting in a much clear community structure to drive low-degree nodes away from the community boundary.
Based on our theoretical analysis, we further devise a novel graph augmentation method, called GRAph contrastive learning for DEgree bias (GRADE), which applies different strategies to low- and high-degree nodes.
Extensive experiments on various benchmarks and evaluation protocols validate the effectiveness of the proposed method.
Introduction
Despite their strong expressive power in graph representation learning, recent studies reveal that the performance of vanilla graph convolutional network (GCN) [17] exhibits a structural unfairness [25, 16] which is primarily beneficial to high-degree nodes (head nodes) but biased against low-degree nodes (tail nodes).
Such a performance disparity is alarming and causes a performance bottleneck, given that node degrees of real-world graphs often follow a long-tailed power-law distribution [2].
Graph contrastive learning (GCL) [30, 24, 22, 38] has been a promising paradigm in graph domain, which integrates the power of GCN and contrastive leaning [13, 5, 10].
In a nutshell, these methods typically construct multiple views via stochastic augmentations of the input, then optimize the GCN encoder by contrasting positive samples against negative ones.
Inheriting advantages of contrastive learning, GCL relieves graph representation learning from human annotations, and displays state-ofthe-art performance in a variety of tasks [35, 12, 39, 36].
Will GCL present the same structure unfairness as GCN ?
For this purpose, we conduct experiments to surprisingly find out that GCL methods are better at maintaining structural fairness where a smaller performance gap exists between tail nodes and head nodes than that of GCN.
This finding suggests that GCL has the potential to mitigate structural unfairness.
Based on this finding, a natural and fundamental question arises: why is graph contrastive learning fairer to degree bias?
A wellinformed answer can yield profound insights into solutions to this important problem, and deepen our understanding of the mechanism of GCL.
[Figure]
Intuitively
[Figure]
graph augmentation provides an opportunity for tail nodes to generate more withincommunity edges, making their representations closer to those with the same community via the contrastive framework.
These refined representations drive tail nodes away from the community boundary.
Theoretically, we prove that node representations learned by GCL conform to a clearer community structure by Intra-community Concentration Theorem 1 and Inter-community Scatter
Theorem 2.
These theorems are relevant to two crucial components in GCL.
One is the alignment of positive pairs, which is exactly the optimization objective.
The other is the pre-defined graph augmentation, determining the concentration of augmented representations.
Based on the analysis, we establish the relation between graph augmentation and representation concentration, implying that a well-designed graph augmentation can promote structural fairness by concentrating augmented representations.
To take a step further, we propose a GRAph contrastive learning for DEgree bias (GRADE) based on a novel graph augmentation.
To make augmented representations more concentrated within communities, GRADE enlarges limited neighbors of tail nodes to contain more nodes within the same community, where the ego network of the tail node interpolates with that of the sampled similar node.
As for head nodes, GRADE purifies their ego networks by removing neighbors from different communities.
Extensive experiments on various benchmark datasets and several evaluation protocols validate the effectiveness of GRADE.
In summary, our contributions are four-fold:
• We are the first to discover that GCL methods exhibit more structural fairness than GCN, which has a smaller performance disparity between tail nodes and head nodes.
This discovery inspires a new path for alleviating structural unfairness based on contrastive learning.
• We theoretically validate the reason for structural fairness in GCL is that GCL stimulates intracommunity concentration and inter-community scatter.
Therefore, tail nodes are farther away from the community boundary for better classification.
• Based on theoretical insights, we propose a method GRADE to further improve structural fairness by enriching the neighborhood of tail nodes while purifying neighbors of head nodes.
• Comprehensive experiments demonstrate that our GRADE outperforms baselines on multiple benchmark datasets and enhances the fairness to degree bias.
Exploring the Behavior of Graph Contrastive Learning on Degree Bias
Real-world graphs in many domains follow a long-tailed distribution in node degrees, i.e., a significant fraction of nodes are tail nodes with small degrees.
It is well known that GCN often performs worse accuracy for tail nodes.
Here we aim to study how GCL behaves under degree bias.
Experimental Setup We take two representative GCL algorithms DGI [30] and GraphCL [35] as examples to analyze the performance disparity under degree bias.
Specifically, we train DGI and
GraphCL on four benchmarks Cora [17], Citeseer [17], Photo [23] and Computer [23], and leverage early stopping based on the training loss.
To compare with GCN, our linear evaluation protocol deploys the semi-supervised split [17], where 20 labeled nodes per class form training set and test set composes of randomly sampled 1000 nodes with degrees less than 50.
GCN follows the standard training paradigm [17] with the above train-test split.
Further implementation details and chosen hyper-parameters are deferred to Appendix A.
Results To illustrate the fairness to degree bias, we group nodes of the same degree and calculate the average accuracy of these degree groups separately shown in Figure 1.
To further visualize how the model balances the performance between tail nodes and head nodes, we fit these scatters with linear regression.
If the slope of the regression line is flat, the model is fair to degree bias.
More results on Photo and Computer datasets can be seen in Appendix A. From the figure, we can find that the average accuracy for tail nodes of GCL methods DGI and GraphCL is higher than that of
GCN, and the slope of the regression line is also smaller.
These interesting observations suggest that label-independent GCL methods are actually fairer than GCN under degree bias.
Analysis on the Structural Fairness of Graph Contrastive Learning
Based on the above observations, a natural and fundamental question arises: where does this structural fairness stem from?
We first define some preliminary notations, then provide a theoretical analysis to explain this question.
3.1 Preliminary Notations
Let G =
( V , E , X ) be a graph, where V is the set of N nodes { v 1 , v 2 , · · · , v N } ,
E ⊆
V × V is the set of edges, X
=
[ x 1 , x 2 , · · · , x N ] ∈ R N × B represents the node feature matrix and x i is the feature vector of node v i .
The edges can be represented by an adjacency matrix A ∈ { 0 , 1 } N × N , where
A ij = 1 iff ( v i , v j ) ∈ E .
Given unlabeled training nodes, each node belongs to one of K latent communities C 1 , C 2 , · · · , C K .
Assuming the augmentation set T consisting of all transformations on topology, the set of potential positive samples generated from ego network G i of node v i is denoted as T ( G i ) .
The goal of GCL is to learn a GCN encoder f such that positive pairs are closely aligned while negative pairs are far apart.
Here we focus on topological augmentation and single-layer GCN,
[Equation]
where ˜ L i is the i -row of transition matrix ˜ L = ˜ D − 1 ˜ A ,
˜ A
=
A
+ I is self-looped adjacency matrix and
˜ D ii
=
(cid:80)
j ˜
A ij is degree matrix.
We consider a community indicator F f
[Equation]
where µ k = E v
i ∈
C k E ˆ G
i ∈T ( G i )
[ f ( ˆ G i )] is the community center, and ∥ · ∥ stands for l 2 -norm.
To quantify the performance of F f , the error can be formulated as
[Equation]
With the above definitions, we denote S ε
= { v i ∈ ∪ Kk =1 C k : ∀ ˆ G 1 i , ˆ G 2 i ∈ T ( G i ) , ∥ f
( ˆ G 1 i ) − f ( ˆ G 2 i ) ∥ ≤ ε } as the set of nodes with ε -close representations among graph augmentations.
3.2 Theoretical Analysis
We assume the nonlinear transformation has M -Lipschitz continuity, i.e., ∥ f ( G i ) − f ( G j )
∥
=
∥ ReLU (˜ L i XW ) − ReLU (˜ L j XW )
∥
≤ M ∥
˜ L i X −
˜ L j X ∥ , and graph augmentations are uniformly sampled with m augmented edges P [ ˆ G i
=
T ( G i )]
= 1 /C ( N − 1 , m ) .
Let there be a ball of radius βm such that for any augmentation ∥
˜ L i X − ˆ L i X ∥ 2 ≤ βm , where ˆ L is the augmented transition.
Theorem 1 Intra-community Concentration.
Let pre-transformation representations ˜ LX be subGaussian random variable with variance σ 2 .
For all nodes v i ∈ S ε , if ε 2 ≤ βm 6 M 2 κ , their representations f
( G i ) fit sub-Gaussian distribution with variance σ 2 f,ε ≤ 1 κ σ 2 with κ ≥ 1 where κ is a coefficient that reflects the degree of concentration.
This theorem builds a relation between the intra-community concentration of final representations and the alignment of positive pairs in S ε .
Specifically, intra-commmunity concentration requires smaller ε .
By decreasing the distance between positive pairs, GCL fits the requirement.
Next, we demonstrate that GCL also maintains the property of inter-community scatter for community assignment.
For a given augmentation set T , we first define the augmentation distance between two nodes as the minimum distance between their pre-transformation representations,
[Equation]
where ˆ A
i is the i -row of augmented adjacency matrix ˆ A , and ˆ d i is the augmented node degree.
Based on the augmentation distance, we further introduce the definition of ( α, γ, ˆ d ) -augmentation to measure the concentration of pre-transformation representations.
Definition 1 ( α, γ, ˆ d )-Augmentation.
The augmentation set T is a ( α, γ, ˆ d ) -augmentation, if for each community C k , there exists a subset C 0 k ⊂ C k such that the following two conditions hold
P [ v i ∈ C 0 k ] ≥ α P [ v i ∈
C
k ]
where
α ∈ (0 , 1] , 2.
sup v i ,v j ∈ C 0 k d T ( v i , v j ) ≤ γ ( B ˆ d k min ) 12 where γ ∈ (0 , 1] ,
where ˆ d k min =
min v i ∈ C 0 k , ˆ G i ∈T ( G i )
ˆ d i , and B is the feature dimension.
Larger α and smaller γ ( B/ ˆ d k min ) 12 indicate pre-transformation representations are more concentrated.
We assume that the representation is normalized by ∥ f ( G i ) ∥
=
r and let p k = P [ v i ∈
C k ] .
Then we simultaneously bound the inter-community distance and the error of the community indicator.
Lemma
For a ( α, γ, ˆ d ) -augmentation with subset C 0 k of each community C k , if nodes belonging to ( C 01 ∪ · · · ∪ C 0 K ) ∩ S ε can be correctly assigned by the community indicator F f , then the error of all nodes can be bounded by (1 − α )
+ R ε , where R ε =
P [ S ε ] is the proportion of complement.
The above lemma presents a sufficient condition to guarantee the performance of the community indicator.
Then we need to explore when nodes in ( C 01 ∪ · · · ∪ C 0 K ) ∩ S ε can be correctly assigned by F f .
Lemma
For a ( α, γ, ˆ d ) -augmentation and each ℓ ∈ [ K ] , if
[Equation]
holds for all k ̸ = ℓ , then every node v i ∈ C 0 ℓ ∩ S ε can be correctly assigned by the community indicator
F f , where ρ ℓ ( α, δ, ε ) =
2(1 − α )
+ 2 R ε p ℓ + α ( Mγ √ B
r √ ˆ d ℓ min + 2 εr ) and ∆ µ = 1 − min k ∈
[ K ] ∥ µ k ∥ 2 /r 2 .
Combining Lemma 1 and 2, we can obtain the Inter-community Scatter Theorem as follows.
Theorem 2 Inter-community Scatter.
For a ( α, γ, ˆ d ) -augmentation, if
[Equation]
holds for any pair of ( ℓ, k ) with ℓ ̸
= k , then the error of the community indicator F f can be bounded by (1 − α )
+ R ε , where ρ max ( α, γ, ˆ d,
[Equation]
= 2(
[Equation]
+
[Equation]
and ∆ µ = 1 − min k ∈ [ K ] ∥ µ k ∥ 2 /r 2 .
For the better assignment, the RHS of Equation 27 should approach r 2 , implying that smaller ρ max ( α, γ, ˆ d, ε ) is required.
We further bound R ε by the alignment objective of contrastive loss.
Theorem
The term R ε is upper bounded by
[Equation]
We direct the readers to Appendix B for proof of all the above lemmas and theorems.
The combination of Theorem 2 and Theorem 3 indicates that the inter-community distance and the error of community indicator are controlled by two key factors.
The alignment of positive pairs.
Good alignment enables small E ˆ G 1 i , ˆ G 2 i ∈T ( G i ) ∥
f ( ˆ G 1 i ) − f
( ˆ G 2 i ) ∥ , resulting in small R ε . 2)
The concentration of augmented representations, where sharper concentration implies larger α in Definition 1.
Small R ε and large α directly decrease the error bound of community indicator, and provide small ρ max ( α, γ, ˆ d, ε ) for inter-community scatter.
It is worth mentioning that the first factor is the contrastive objective in GCL, reflecting the reason for structural fairness in the GCL framework.
While the second factor depends on the graph augmentation.
Thus, we are motivated to propose a graph augmentation designed for further concentrating augmented representations.
GRADE Methodology
In this section, we present our novel GRADE framework dedicated to degree bias in detail, starting with the special graph augmentation, followed by the objective of GCL.
4.1 Graph Augmentation
We generate two augmentations ˆ G 1 and ˆ G 2 by simultaneously corrupting the original feature and topology to construct diverse contexts [36, 26].
We denote node representations in these two augmentations as H = f
( ˆ G 1 ) and O = f
( ˆ G 2 ) .
Topology Augmentation To obtain more concentrated augmented representations, we aim to increase intra-community edges while decreasing inter-community edges.
Due to the different structural properties of tail nodes and head nodes, we design different topology augmentation strategies for them shown in Figure 2.
In order to expand the neighborhood of tail nodes to include more same-community nodes, we interpolate the ego network of the anchor tail node v tail with that of a sampled similar node v sample .
To prevent injecting many different-community neighbors, we regulate the interpolation ratio depending on the similarity between v tail and v sample .
For head nodes, we purify their neighborhood by similarity-based sampling to remove inter-community edges.
Formally, we build the similarity matrix S between node pairs based on cosine similarity of their representations, S ij = sim( h i , h j ) for i ̸
= j and S ii = 0 otherwise.
For any tail node v tail , we sample a node v sample from the multimodal distribution Multi( s tail ) , where s tail is the row vector of S corresponding to v tail .
Then we construct a new similarity-aware neighborhood for v tail by interpolating between neighbor distributions of v tail and v target .
Here, the neighbor distribution for node v is defined as p ( u | v ) = 1 / |N ( v ) | if u ∈ N ( v ) and p ( u | v ) = 0 otherwise.
To avoid the detrimental connectivity, the similarity sim( h tail , h sample ) is used as the interpolation ratio ϕ ,
[Equation]
The interpolation ratio ϕ decreases as the similarity between v tail and v sample decreases, and we guarantee (1 − ϕ ) to be at least 0.5 to preserve the original neighborhood.
Given the neighbor distribution p sample ( u | v tail ) , we sample neighbors from it without replacement.
The number of neighbors is sampled from degree distribution except tail nodes to keep degree statistics.
#$% !
&#'(%) (a) Interpolation T a il N o d e s
H ea d N od e s (b) Purification
Merge (a) Interpolation      0 1 2 3 4 *(,|! "
#$% )
*(,|! &#'(%) )
Comb.
0 1 2 3 4 * &#'(%) (,|! "
#$% ) (b) Purification ! /)
#      0 1 2 3 4 * &$' (,|! /)#0 )
Sample
For each head node v head , we define the similarity distribution for purification.
Specifically, the similarity distribution for node v is p sim ( u | v ) =
sim( h u , h v ) if u ∈ N ( v ) and p ( u | v ) = 0 otherwise.
Based on the similarity distribution p sim ( u | v head ) , we sample d head (1 − p edr ) neighbors without replacement, where p edr is the edge drop rate.
Through this sampling, edges of dissimilar nodes tend to be removed, thereby retaining effective neighborhood information.
Feature Augmentation We randomly sample a mask vector m ∈ { 0 , 1 } B to hide a fraction of dimensions in node feature.
Each element in mask m is sampled from a Bernoulli distribution
Ber(1 − p fdr )
, where the hyperparameter p fdr is the feature drop rate.
Thus, the augmented node feature ˆ X is computed by
[Equation]
In our implementation, we set a threshold ζ to distinguish tail nodes and head nodes.
The same hyperparameters p fdr and p edr are used to generate augmentations
ˆ G 1 and ˆ G 2 .
4.2 Optimization Objective
We employ a contrastive objective [38] on obtained node representations of two graph augmentations.
For node v i , node representations h i
and o i from different graph augmentations form the positive pair, and node representations of other nodes in two graph augmentations are regarded as negative pairs.
Therefore, we define the pairwise objective for each positive pair ( h i , o i ) as
[Equation]
where τ is a temperature parameter.
The critic θ ( h , o ) is defined as sim( g ( h ) , g ( o )) , where the projection g is a two-layer multilayer perceptron (MLP) to enhance the expression power [5].
The overall objective to be maximized is the average of all positive pairs,
[Equation]
Cora
Citeseer
Photo
[Table]
Experiments
Datasets For a comprehensive comparison, we use four real-world datasets to evaluate the performance of node classification and the fairness to degree bias.
Specifically, we choose two categories of datasets: 1) citation networks including Cora [17] and Citeseer [17], 2) social networks Photo [23] and Computer [23] from Amazon.
The statistics of these datasets are summarized in Appendix C.
Evaluation Protocol We compare GRADE with state-of-the-art GCL models DGI [30],
GraphCL [35], GRACE [38], MVGRL [12] and CCA-SSG [36]
[Table]
and semi-supervised baseline
GCN [17] for reference.
For GCL models, we follow the linear evaluation scheme introduced in [30], where each model is firstly trained in an unsupervised manner and node representations are subsequently fed into a simple logistic regression classifier.
We adopt two universally accepted splits for full evaluation: 1) semi-supervised split [30, 35] that 20 labeled nodes per class are for training and 1000 nodes are for testing, 2) supervised split [38, 36] that 1000 nodes are for testing and the rest of nodes form the training set.
It is worth noting that 1000 nodes in the test set are randomly sampled with degrees less than 50 to provide an appropriate degree range for analysis.
GCN is trained by the original paradigm [17] with the above train-test split.
We refer readers of interest to Appendix C on details of experiments, including implementation and hyperparameters.
5.1 Main Results and Analysis
Node Classification We train each model for 10 independent trials with different seeds, and report mean and standard deviation results in Table 1.
We observe that the proposed GRADE outperforms all baselines in most cases.
The improvement of GRADE is more pronounced on Cora and Citeseer datasets, where average node degrees are around 3 and a large number of tail nodes exist.
Additionally, we find that GCL models tend to outperform GCN in the semi-supervised split, suggesting that GCN may benefit more from end-to-end training with more supervision.
To verify that GRADE improves the classification performance of tail nodes and also retains the performance of head nodes, we divide test nodes of Cora into tail nodes and head nodes based on the threshold ζ .
We draw average accuracy w.r.t.
degree of GRADE and competitive baselines as violin plots in Figure 3.
The subsequent experiments default to the supervised split if not specified.
As expected, GRADE achieves obvious performance gain regardless of tail nodes or head nodes.
Fairness Analysis
In order to quantitatively analyze the fairness to degree bias, we define the group mean as the mean of degree-specific average accuracy while the bias is the variance.
Mathematically,
Avg .
Acc .
( k ) =
E [ { Acc( v i ) , ∀ node v i such that d i = k } ] ,
G.Mean = E [ { Avg .
[Table] [Table]
Based on these metrics, evaluation results are shown in Table 2.
It can be seen that GRADE reduces the bias across all datasets and maintain the highest group mean.
Moreover, graph contrasting learning models have a smaller bias compared to GCN, conforming to our previous study in Section 2.
Visualization To demonstrate that GRADE pulls same-community node representations more concentrated, we visualize node representations of GRADE and competitive baselines on the Cora dataset in Figure 5.
Particularly, we zoom into one specific community colored blue.
Graph contrastive learning baselines always depict more crisp boundaries than GCN, where blue nodes are still scattered in space.
In GRADE, they are finally clustered together, illustrating that the proposed augmentation design exerts a vital part.
5.2 Ablation Study and Hyperparameter Sensitivity
Ablation Study of Sampling Recall that GRADE augments tail nodes by sampling nodes for interpolation, and also sample edges to remove in the purification of head nodes.
We alter the augmentation by fixing to the most similar node (without random interpolation) or top- d head (1 − p edr ) similar nodes (without random deleting) to validate the effectiveness of these sampling processes.
Results of the ablation study on Cora and Citeseer datasets are reported in Table 3.
We can observe that GRADE is consistently better than the remaining variants.
Such a phenomenon implies that reasonable randomness provides more diverse node contexts to contrast out essential features.
Effect of Threshold We investigate the impact of threshold ζ used to split tail nodes and head nodes on classification performance.
Figure 4 (a) shows the test Micro-F1 w.r.t.
different ζ on
Cora dataset.
The performance benefits from an applicable selection of ζ .
When ζ is too small, the neighbor sparsity of tail nodes cannot be mitigated; if it is too large, noise is injected.
Effect of Drop Rate We perform sensitivity analysis on feature drop rate d fdr and edge drop rate d edr which control the generation of graph augmentations.
We vary these hyperparameters fromto 0.5 in node classification on the Cora dataset.
The results are shown in Figure 4 (b).
It can be observed that the performance is relatively poor when the hyperparameter d fdr is too large and d edr is too small.
We infer that if the d fdr is too large, the original graph is heavily undermined to contain useful information; and if the d edr is too small, the large number of neighbors makes the perturbation of different augmentations too insignificant to achieve the purpose of contrast.
[Figure] [Figure]
[Figure] [Figure]
[Table]
Related Work
Graph Neural Networks Graph neural networks (GNNs) [18, 32, 4, 31] can be generally divided into spectral methods and spatial methods.
Specifically, spectral methods learn node representations based on graph spectral theory.
[3] first proposes a spectral graph-based extension of convolutional networks
[Table]
and GCN [17] simplifies ChebNet [6] by the first-order approximation.
Spatial methods directly define graph convolution in the spatial domain.
GraphSAGE [11] learns aggregators by sampling and aggregating neighbors
[Table]
and GAT [29] assigns different edge weights during aggregation.
We refer readers to recent surveys [34, 37] for a more comprehensive review.
On the other hand, some literature [25, 33, 21] shows that there is a structural unfairness between tail nodes and head nodes in
GNNs.
Existing explorations [20, 16] focus on supervised settings, ignoring the great potential of self-supervised learning on this problem.
Graph Contrastive Learning Being popular in self-supervised visual representation learning [28, 7, 27, 13, 5, 10, 15], contrastive learning obtains discriminative representations by contrasting positive and negative samples.
Inspired by the local-global mutual information maximization viewpoint [14],
DGI [30] and InfoGraph [24] first marry the power of GNNs and contrastive learning.
Following them, MVGRL [12] introduces the node diffusion to the graph contrastive framework.
GRACE [38],
GCA [39] and GraphCL [35] learn node representations by treating other nodes as negative samples, while BGRL [26] proposes a negative-sample-free model.
CCA-SSG [36] optimizes a feature-level objective other than instance-level discrimination.
There are several surveys [1, 19] summarizing recent advances in graph contrastive learning.
Despite their remarkable achievements, there is no graph contrastive learning targeting the fairness of degree bias.
Conclusion
In this paper, we bring to light the prospect of GCL to alleviate structural unfairness for node representation learning.
We discover that node representations obtained by GCL methods are fairer to degree bias than those learned by GCN, and explore the underlying cause of this phenomenon.
Based on our theoretical analysis, we further propose a novel GCL model targeting degree bias.
Limitations and Broader Impact A limitation of GRADE is its heuristic design, therefore an interesting direction for future work is to extend GRADE into learnable graph augmentation.
Our work investigates the structural fairness in GCL for the first time and points out the great potential of
GCL for this problem.
Considering that most real-world graphs follow the long-tail distribution, the studied problem is practical and important.
Additionally, our work deepens the understanding of the learning mechanism of GCL, and may inspire more future research on structural fairness.
[-Lemma 
-]
[-Lemma 
-]
[-Theorem 
-]