Co-Modality Graph Contrastive Learning for Imbalanced Node Classification
Abstract
Graph contrastive learning (GCL), leveraging graph augmentations to convert graphs into different views and further train graph neural networks (GNNs), has achieved considerable success on graph benchmark datasets.
Yet, there are still some gaps in directly applying existing GCL methods to real-world data.
First, handcrafted graph augmentations require trials and errors, but still can not yield consistent performance on multiple tasks.
Second, most real-world graph data present class-imbalanced distribution but existing GCL methods are not immune to data imbalance.
Therefore, this work proposes to explicitly tackle these challenges, via a principled framework called C
o- M odality G raph C ontrastive L earning ( CM-GCL ) to automatically generate contrastive pairs and further learn balanced representation over unlabeled data.
Specifically, we design inter-modality GCL to automatically generate contrastive pairs (e.g., node-text) based on rich node content.
Inspired by the fact that minority samples can be “forgotten” by pruning deep neural networks, we naturally extend network pruning to our GCL framework for mining minority nodes.
Based on this, we co-train two pruned encoders (e.g., GNN and text encoder) in different modalities by pushing the corresponding node-text pairs together and the irrelevant node-text pairs away.
Meanwhile, we propose intra-modality GCL by co-training non-pruned GNN and pruned GNN, to ensure node embeddings with similar attribute features stay closed.
Last, we fine-tune the
GNN encoder on downstream class-imbalanced node classification tasks.
Extensive experiments demonstrate that our model significantly outperforms state-of-the-art baseline models and learns more balanced representations on real-world graphs.
Introduction
Contrastive learning, aiming to contrast instance pairs generated from unlabeled data to train better representation models, has attracted considerable attention.
Inspired by the consistent success of contrastive learning in computer vision and natural language processing [3, 2, 26], an increasing number of works have started to investigate the great potential of contrastive learning on graphs [44, 52, 38].
Most existing graph contrastive learning (GCL) models leverage different graph transformation methods to augment graph data into different views, and further train graph neural networks (GNNs) by maximizing the similarity between positive pairs while minimizing the similarity between negative pairs.
In particular, PyGCL [52] summarizes nine categories of graph augmentations from differ- ent levels (e.g., node dropping, edge perturbation, and graph diffusion), and implements different augmentation combinations to transform graph into different views for contrastive learning.
Yet, directly applying existing GCL models [28, 27, 31, 44] from the controlled benchmark graph datasets to uncontrolled real-word graph datasets still has some gaps.
First, notwithstanding suitable handcrafted augmentation methods need professional knowledge as well as trials and errors, the consistent excellent performance on multiple tasks can not be guaranteed [52].
Second, node classes in most real-world graphs present imbalanced distribution (e.g., Zipf long-tail distribution).
For instance, the amount of illicit activities (e.g., malicious code/repository [43, 10, 7, 42], illicit drug/crimeware trading [30, 47]) on the Internet and social networks is much less than that of normal activities [33].
To tackle these challenges, we aim to design a novel GCL model that automatically generates contrastive pairs without too much effort, but also can generalize well in real-world graphs with the class imbalance problem.
Accordingly, inspired by existing cross-modality contrastive learning methods in computer vision and
NLP [32, 53], we naturally hypothesize that node embeddings in graph and the corresponding content embeddings (e.g., text embedding) in different modalities will be closer if we train GNN encoder and the content encoder (e.g., text encoder) simultaneously during contrastive learning.
In this way, we can automatically generate contrastive pairs, e.g., (node-text), based on the natural node content, which is called inter-modality GCL.
However, recent studies [14, 41] demonstrate that contrastive learning can alleviate the data imbalance problem but fails to be fully immune to the data imbalance issue.
Therefore, motivated by the fact that minority data may be easily “forgotten” by pruning deep neural networks [13], we extend the ad-hoc compression tool, network pruning, to GCL framework and further discover minority samples not yet well represented by encoders during co-modality GCL.
To summarize, we propose a novel framework called co-modality graph contrastive learning ( CMGCL ) to automatically generate contrastive pairs from unlabeled data and further learn more balanced node representations.
As illustrated in Figure 1, we first prune the graph encoder and content encoder in inter-modality GCL to uncover more minority samples, and further co-train two encoders via optimizing the inter-modality contrastive loss.
Besides, we propose to ensure that node embeddings with similar attribute features should also stay closed.
Therefore, we propose intra-modality GCL that generates contrastive pairs based on the similarities among node attribute features and further co-trains non-pruned graph encoder and pruned graph encoder.
By the aforementioned steps, CM-GCL can boost minority nodes weights in contrastive loss and lead to implicit loss re-balancing.
Finally, we fine-tune the pre-trained GNN encoder on the class-imbalanced node classification tasks.
To conclude, our work makes the following contributions:
• As handcrafted graph augmentations are not efficient on real-world graph datasets, we devise a co-modality framework to automatically generate contrastive pairs.
To the best of our knowledge, this is the first work on GCL that co-trains encoders in different modalities to facilitate node representation learning.
• To address the class imbalance issue on graphs, our model extends network pruning to prune encoders in co-modality GCL for discovering minority samples and further learning more balanced node representations.
• Extensive experiments on multiple real-world graph datasets show the effectiveness of our model by comparing CM-GCL with state-of-the-art methods.
Related Work
Graph Contrastive Learning.
Most GCL models [31, 44, 38, 50, 45, 39] leverage different types of data transformations to augment graph into different views and further train an encoder by discriminating positive pairs and negative pairs generated from unlabeled data.
Specifically, GraphCL [44] designs four types of graph augmentations to generate contrastive pairs in different views for contrastive learning.
GCC [31] implements the random walk for each node to sample subgraphs as augmentation.
In the heterogeneous domain, HeCo [38] employs cross-view (schema view and meta-path view) contrastive learning to train an encoder.
These handcrafted data augmentations require trials and errors, and sufficient domain knowledge, but still cannot yield consistent excellent performance on multiple tasks [44].
Hence, we naturally propose to automatically generate contrastive pairs for graph data.
Inspired by CLIP [32], a cross-modality contrastive learning model on image and text data, we devise cross-modality GCL that leverages the rich node content to automatically generate useful contrastive pairs without resorting to handcrafting or expert knowledge.
Data Imbalance in Self-supervised Learning.
Data imbalance is very common in real-world applications (e.g., anomaly detection [5, 6]).
Most existing methods against data imbalance focus on supervised learning (e.g., re-sampling [1, 24], re-weighting [16, 35], and GraphSMOTE [51]).
Recent works [41, 15, 14] start to explore the benefits of the balanced feature space from selfsupervised learning.
Yang and Xu [41] first studied the label bias in self-supervised learning and concluded that classifiers that are pre-trained in a self-supervised manner consistently outperform their corresponding supervised baselines.
Kang et al.
[15] found that contrastive learning can present a more balanced feature space over imbalanced datasets by comparison with supervised learning.
Based on this, SDCLR [14] validates that contrastive learning is not fully immune to data imbalance.
All aforementioned works study the imbalance problem of image.
Motivated by these works, we devise an algorithm that can well mitigate the data imbalance in our proposed co-modality GCL framework.
Network Pruning.
Network pruning [19, 22, 20] has been considered as a popular compression tool for DNN models.
In recent years, some works [8, 13, 14] have started to explore the deeper connection with DNN memorization and generalization.
Specifically, Frankle and Carbin [8] found that pruning a dense DNN can identify “significant subnetwork” whose testing accuracy is comparable to the one in the original network.
Hooker et al.
[13] thought a small data subset, i.e., data belonging to ambiguous classes and minority classes, is more likely to be impacted by the introduction of sparsity via network pruning.
Based on the conclusion, Jiang et al.
[14] constructed a self-competitor via network pruning to study the imbalanced image in contrastive learning.
In this work, we introduce network pruning in co-modality contrastive learning to uncover minority nodes and further gain the more balanced representation.
Preliminary
Graph Neural Networks.
Given a graph G =
( V , E , X ) , where V is the set of different types of nodes, E
⊆
V × V is the set of edges, and X is the attribute feature set.
We aim to learn the node representation by considering both structure and node attribute features.
Current GNNs (e.g.,
GCN [18], GAT [37], GraphSAGE [9], and HetGNN [46]) follow a messaging-passing framework and has gained consistent performance on various graph data.
In this paper, we choose GCN as the graph encoder to learn the node embedding
h iG ∈ R d G corresponding to node v i .
In particular, GCN is formulated as H l +
= σ ( (cid:101)
A H l W l ) , where H l +1 denotes the node representations at l + 1 layer, (cid:101)
A is a symmetric normalization of A with self-loop, W l is the weight matrix at l -th layer, and σ is the activation function.
We use f G to denote the graph encoder and h G to denote the node embedding.
Besides GCN, we also discuss GAT and GraphSAGE as the graph encoder in experiments.
Text Pre-Trained Model.
In most real-world graphs, nodes contain rich text information.
Most existing graph representation learning methods [18, 9, 49, 48, 40] extract the keyword from text and convert the rich text content into sparse bag-of-words feature vector, and further utilize the extracted features to facilitate the graph representation learning.
Instead of directly utilizing the bag-of-words feature vectors as the attribute features for nodes, we propose to fine-tune the pre-trained language model on the handy unlabeled dataset within the graph.
In this work, we leverage the pre-trained transformer language model (e.g., DistilBERT [34], a distilled version of BERT) as the text encoder and further fine-tune the text encoder with co-modality contrastive loss.
The structure of DistilBERT we implemented is a base size with a 6-layer 512-wide model along with 8 attention heads.
Text sequence is bracketed with [SOS] and [EOS] tokens and we take the activation of the highest layer of transformer at [EOS] token as text representation h T ∈ R d T .
In this paper, f T denotes the text encoder.
After fine-tuning f T via optimizing co-modality contrastive loss, we replace the bag-of-words feature vectors with fine-tuned attributes for all nodes.
Besides DistilBERT, we also discuss other transformer models (i.e., BERT) as the text encoder and more details can be found in Section 5.5.
Image Pre-Trained Model.
Except for text information, image content is also very common for nodes in graph (e.g., Instagram).
We implement two different architectures (i.e., ResNets and
Transformer) as the image encoder.
As ResNets is widely used and has been proven with excellent performance, we implement ResNet50 [11] as the base architecture.
Following [12], we make similar modifications on ResNet and replace the global average pooling layer with an attention pooling mechanism.
Besides, we consider the recently introduced Swin Transformer [23] as the image encoder and closely follow their implementations.
We utilize f I to denote the image encoder to obtain the image representation h I ∈
R
d I .
After fine-tuning f I via contrastive loss optimization, we attach the fine-tuned image embedding to the corresponding nodes as the attribute features for downstream tasks
Problem 1.
Co-Modality Graph Contrastive Learning for Imbalanced Node Classification.
Given a graph G =
( V , E , X )
built on multi-modality data along with imbalanced class label Y , we aim to build a GCL model over G that can be further fine-tuned or applied to the downstream tasks with imbalanced labels (e.g., imbalanced node classification).
Methodology
In this section, we present the details of CM-GCL (Figure 1) which includes three steps: (i) graph construction (Figure 1.(a)); (ii) co-modality graph contrastive learning with network pruning (Figure 1.(b)); (iii) model fine-tuning on downstream imbalanced node classification tasks (Figure 1.(c)).
4.1 Graph Construction
Graph has been proven to be effective in modeling real-world networks (e.g., academic networks and social networks).
Merely taking the relational structure information among nodes into consideration is deemed insufficient to learn the node representations.
Thus, to model real-world graph data for certain tasks (e.g., domain prediction for AMiner academic network), we depict both the informative content and rich structure information within graph data.
Following existing works [18, 9, 37], we first convert the text content into sparse bag-of-words feature vectors attached to the corresponding nodes, which is denoted as x G .
For some graphs that have rich image content (e.g., Instagram), we implement the pre-trained image encoder (e.g., ResNet) to obtain the image feature vectors applied to the corresponding nodes.
Besides that, different from existing works, we further fine-tune the content encoder (denoted as f T for text encoder or f I for image encoder) via co-modality contrastive learning.
Inspired by CLIP [32] that co-trains the text encoder and the image encoder for the text image matching task, we view the rich content as the text (image) modality data and the corresponding graph structure with bag-of-words feature vectors as the graph modality data.
We argue the corresponding data in different modalities can achieve agreements in the learned representation space.
4.2 Co-Modality Graph Contrastive Learning
After graph construction, we first propose inter-modality contrastive learning to automatically generate contrastive pairs.
Then we introduce network pruning to discover minority nodes during contrastive learning.
Last, we design intra-modality contrastive learning to ensure that node embeddings with similar semantics (attribute features) stay closed.
4.2.1 Inter-Modality Graph Contrastive Learning
The main idea of contrastive learning is to make representations maximize the agreement on sample pairs under proper graph augmentations.
Yet, finding proper graph augmentation for every task is always resource-consuming [44].
Inspired by existing works [32, 53] in terms of cross-modality learning, we propose to fully utilize nodes’ content to automatically generate contrastive pairs, and further co-train encoders in different modalities via contrastive learning.
That is to say, we aim to learn two encoders such that embeddings in two modalities are close to each other in the learned space if they refer to the same entity, otherwise far away.
In this paper, we mainly consider two combinations of modalities as illustrated in Figure 1.(a)
(i.e., graph-text or graph-image).
For simplicity, we discuss graph-text modalities as an example in this section, and will conduct experiments on graph-image modalities.
Specifically, given a sample pair ( x iG , x iT ), where x iG is the original attribute feature (i.e., bag-of-words feature vector) of node v i ∈ V and x iT is the corresponding text content input, we feed x iG to the graph encoder f G and x iT to the text encoder f T respectively, and further obtain the corresponding representations ( h iG , h iT ).
Note that, as different types of nodes in G may have different feature dimensions, we first transform the input attribute features of all nodes in G to a common space.
After obtaining the representation pair ( h iG , h iT ), we apply nonlinear projection heads to convert embeddings from different modalities to the same space (denoted as ( z iG , z iT ) ) for comparison, which can be formally defined as z i
=
MLP ( h i ) .
We apply separate projection heads for representations generated by the encoder in different modalities.
Here we denote ( z iG , z iT ) as the projected embedding pair.
We consider ( z iG , z iT ) as a similar pair (positive pair) if the node and the text content refer to the same entity.
Otherwise, ( z iG , z iT ) is viewed as a dissimilar pair (negative pair).
4.2.2 Network Pruning
We are aware that imbalanced data fails many supervised approaches built on balanced benchmarks.
Whether contrastive learning is immune to data imbalance or not has remained controversial.
Recent studies [14, 15, 41] argue and demonstrate that self-supervised contrastive learning can alleviate the data imbalance issue in representation learning but is not fully immune to data imbalance.
Based on the above findings, we propose to alleviate the data imbalance issue during our proposed co-modality
GCL.
Some recent works [13, 14] discover that certain examples are particularly sensitive to sparsity via network pruning and these samples mostly impacted after pruning are termed as Pruning Identified
Exemplars (PIEs) [13].
They further demonstrate that PIEs often show up in minority classes.
Inspired by this, we extend network pruning to our proposed framework to uncover minority samples.
Unlike pruning a single encoder for image model in the previous works [13, 14, 8], we co-prune multiple encoders in different modalities.
With PIEs dynamically generated during contrastive training, more minority samples will be exposed along with training epochs, as we will show in Figure 2.
In this paper, we employ the magnitude pruning method [8] that uses the absolute value of weights to rank their importance and removes weights that are below a certain threshold.
Specifically, the detailed process is described as follows:
Randomly initialize encoders f G and f T with parameters θ 0 G and θ 0 T respectively.
At each iteration l , prune e % of smallest-magnitude parameters in θ l G and θ l T by creating the corresponding masks m lG and m lT .
Apply the masks m lG and m lT to the feed-forward encoders f G and f T respectively, and calculate the corresponding contrastive loss.
Release the masks m lG and m lT on the masked parameters and update encoders’ parameters via optimizing the contrastive loss.
In the above steps, the contrastive loss in inter-modality GCL is formulated as:
[Equation]
where ( (cid:101)
z i G , (cid:101)
z i T ) is the projected embedding pair generated by the pruned encoders (cid:101) f G and (cid:101)
f T , sim ( (cid:101) z iG , (cid:101) z iI ) is the cosine similarity between (cid:101) z iG and (cid:101) z iI , τ inter is the temperature value, and n is the size of mini-batch.
Following the iterative pruning process that repeatedly trains, prunes and releases weights in each round, it effectively boosts minority samples’ weights in L inter and leads to implicit loss re-balancing.
In this way, minority samples are more likely to be uncovered and further force the encoders to learn more about these minority samples.
4.2.3 Intra-Modality Graph Contrastive Learning
Inter-modality graph contrastive learning ensures that embeddings from different modalities w.r.t.
the same entity stay closed in the projected space but fails to ensure that embeddings with similar original features from the same modality stay closed.
We argue that nodes with similar attribute features naturally have similar semantics and their embeddings should be closer than those with dissimilar attribute features.
However, most existing GNNs are mainly dominated by the relationships among nodes in graph [29].
In light of this, we propose intra-modality graph contrastive learning to balance the importance of local neighbor relations and attribute features, and further ensure that node embeddings with similar attribute features will stay closer in the projected space.
Note that, we mainly focus on graph modality in intra-modality graph contrastive learning.
Specifically, we first define S as the set of node pairs having the top-R similarity, which is formally defined as:
[Equation]
in top-R of [
[Equation] [Equation]
Np =
[Equation] [Equation]
where sim ( x iG , x pG ) measures the cosine similarity of node attribute features between node v i and v p ,
R is the number of top pairs chosen for each node, and N is the number of nodes in G .
The main difference between intra-modality GCL and inter-modality GCL lies in that, intra-modality
GCL employs the graph encoder and the pruned graph encoder for contrastive training, while intermodality GCL implements the graph encoder and the text (image) encoder for co-modality contrastive training.
Hence, we have a dense graph encoder f G and a sparse graph encoder (cid:101)
f G pruned by iterative magnitude pruning.
Given a sample pair ( x iG , x pG ) , where i can equal p , it will be encoded by f G and (cid:101)
f G respectively.
The projected embeddings ( z iG , (cid:101) z pG ) is regarded as a positive pair if ( v i , v p ) in S , otherwise, it will be viewed as a negative pair.
The intra-modality GCL loss is defined as follows:
[Equation]
By this way, for atypical and minority nodes, our model can amplify the representation differences between the pruned and non-pruned graph encoders in intra level, and these nodes’ weights will be implicitly increased in the intra-modality contrastive loss L intra .
After performing inter-modality graph contrastive learning and intra-modality graph contrastive learning, the overall objective of co-modality graph contrastive learning can be formulated as:
[Equation]
where λ is the trade-off hyper-parameter for balancing two loss terms.
Pseudo-code of CM-GCL is provided in Section A of the Appendix.
4.3 Fine-tuning
After sufficient training on the co-modality GCL, different from most existing methods that directly fine-tune the pre-trained encoder f ∗ G on downstream tasks [31, 44], we adopt a multi-step protocol to evaluate the classification performance.
(i) Representation learning via CM-GCL: we pre-train encoders by optimizing the pre-training loss L in Eq. 4; (ii) Update the attribute feature: we leverage the learned content encoder f ∗ T to get the updated attribute features; (iii) Fine-tuning the GNN encoder: we adopt the pre-trained weights from f ∗ G as the initialized parameters for the fine-tuning encoder.
Then we train the GNN encoder together with the classifier (i.e., MLP) over imbalanced nodes.
To address the class imbalance problem, we introduce Focal Loss [21] that applies a modulating term to the cross-entropy loss to focus on hard misclassified nodes.
This can be considered as a dynamically scaled cross entropy loss, where the scaling factor can down-weight the contribution of easily classified nodes automatically during model fine-tuning and rapidly focuses the model on hard nodes.
In particular, the supervised multi-class focal loss L focal can be formally defined as:
[Equation]
where V l is the node sets of labeled nodes, ˆ y ic is the prediction score of node v i being classified as class c , γ is the focusing parameter to control the rate at which easy nodes will be down-weighted, and α c ∈ [0 , 1] is a weighting hyper-parameter for different classes.
Noted that L focal =
L ce when γ = 0 and α = 1 .
Experiments
In this section, we first introduce datasets we use to evaluate CM-GCL and baselines models.
Then we provide the detailed analysis to show the effectiveness of CM-GCL and its strong applicability to real-world graph datasets.
In addition, the baseline settings are provided in Section B of the
Appendix.
5.1 Experimental Setup 5.1.1 Dataset
[Table]
In this paper, we adopt four multi-modality graph datasets from existing works, i.e.,
AMiner [36], YelpChi [33], GitHub [29], and
Instagram [30], which contain the raw content (e.g., text or image) and the graph structure information.
Specifically, AMiner graph is a papercitation academic network with raw text content (i.e, title).
We select a sub-graph (having 18,089 papers and 22,864 authors) from AMiner dataset, and aim to predict the domain label (i.e., data mining, medical informatics, theory, visualization, and database) of each paper.
In order to solve a harder classification task, we cast aside venue nodes from AMiner academic network and consider two relation types (i.e., author-write-paper and author-collaborate-author) for this academic graph.
Different relation types are regarded as the same type.
Each paper is characterized by the bag-of-words of keywords.
We also count the number of published papers for each author as the attribute feature.
In this dataset, the class distribution is relatively balanced (Tabel 1), so we use an imitative imbalanced setting by randomly selecting two classes (two out of five) as the minority classes.
YelpChi provided by [33] is a benchmark graph dataset with the raw reviews for a set of restaurants and hotels in Chicago.
We utilize YelpChi dataset to identify genuine reviews (8,919) and fake reviews (58,476), which is a binary classification task.
Similar to AMiner, each review is characterized by the bag-of-words of keywords.
Followed by [33], we also extract three types of relations among reviews.
GitHub graph, including 6, malicious and 13,930 benign repositories, aims to detect malicious repositories on GitHub platform, which contains the raw content of repositories.
Please refer to the detailed graph information in [29].
To mimic the imbalanced situation, we employ an imitative imbalanced setting with different imbalance ratios.
Instagram graph contains the image and text information about users and posts and the corresponding rich relationships.
Different from other three datasets, we mainly utilize the image information within this dataset to pre-train our proposed model and further detect illicit drug traffickers (including 3,242 drug traffickers and 5,409 normal users) on Instagram.
5.1.2 Baseline Method
To evaluate the performance of CM-GCL, we compare CM-GCL with nine baseline methods which are divided into four groups (Table 2): feature-based method (G1), graph learning models (G2), methods against imbalanced data (G3), and GCL models (G4).
For G1 (Feature), we feed the attribute feature vector to a two-layer MLP [25].
For G2, we implement three graph learning methods, i.e.,
GCN [18], GAT [37], and GraphSAGE [9], to learn the node representations.
For G3, we implement three popular approaches, i.e., over-sampling [17], re-weighting [4], and GraphSMOTE [51], to handle the imbalanced node representations generated by GCN.
GraphSMOTE is the extension of
SMOTE [1] on graph data, which leverages node similarities to generate synthesized nodes and edges.
For G4, we reproduce two GCL methods, i.e., GCC [31] and GraphCL [44].
To compare them with
CM-GCL fairly, we fine-tune the pre-trained models with Focal loss L focal as well.
5.1.3 Experimental Settings
All experiments are conducted under the environment of the Ubuntu 16.04 OS, plus Intel i9-9900k
CPU, two GeForce GTX 2080 Ti Graphics Cards, and 64 GB of RAM.
We train all methods for each graph with a fixed epoch (e.g., 100 for AMiner graph).
Besides, all methods are trained ten times, and the average performance multiplied by 100 on testing data is reported.
We use 70% samples for training, 10% for validation, and the remaining 20% for testing.
Following existing works in evaluating imbalanced class classification [1, 51], we adopt two metrics, i.e., Macro F1-score (F1)
[Table]
and AUC-ROC score (AUC) to evaluate all models.
Besides, except for the initial imbalance ratio of datasets, to evaluate the model performance in more imbalanced scenarios, we set β (imbalance ratio) as 0.1 and 0.01 and further utilize D init , D 0.1 , and D 0.01 to denote dataset with initial imbalance ratio, 0.1, and 0.01, respectively.
Note that, we leverage D init to train CM-GCL and fine-tune the pre-trained encoder over D init , D 0.1 , and D 0.01 to conduct the downstream classification tasks.
With the grid search, pruning ratio e is set as 20%, the number of contrastive pairs R for each node in intra-modality GCL is different for different graphs (e.g., 5 for AMiner graph) , and the number of mini-batch n is different for different tasks, (e.g., 100 for AMiner graph).
Besides, the temperature parameter τ inter and τ intra are set as 0.1 and the trade-off hyper-parameter λ among co-modality GCL is set as 0.5.
For fine-tuning, α and γ in L focal for different graphs are different (e.g., (0.75, 1.0) for
AMiner graph).
5.2 Performance Comparison
Table 2 shows the performances of all models on four graphs with different imbalance ratios.
The best performances are highlighted in bold and the second-best performances are emphasized by underline.
In Table 2, we can conclude that: (i) Merely considering content data (Feature) is not supportive enough to learn node representations.
Integrating rich content and relations in graph can learn better representations.
(ii) Leveraging unlabeled data to train graph encoder via contrastive learning can enhance node representations (i.e., GCC and GraphCL).
Note that, to compare with baseline models fairly, most models (i.e., over-sampling, re-weighting, GraphSMOTE, GCC, GraphCL, and CM-GCL) are based on the embeddings generated from a two-layer GCN.
(iii) Some popular methods (i.e., over-sampling, re-weighting, and GraphSMOTE) against imbalanced data in supervised learning can alleviate the influence of imbalance and further improve the performance.
In addition, GraphSMOTE has excellent performance when the class is too imbalanced (i.e., D 0 . 01 ).
(iv) CM-GCL has excellent performance in most scenarios when dealing with imbalanced node classification over four real-world graphs.
By comparing CM-GCL with GCL models (i.e., GCC and GraphCL), we also conclude that
CM-GCL can learn better representations for real-world graphs.
5.3 Ablation Study
[Table]
To show the effectiveness of different components in CM-GCL, we conduct a set of ablation experiments over subset D 0.1 of AMiner (nodetext) and Instagram (node-image) datasets and further analyze the contribution of each component (i.e., co-modality GCL, pruning, and intra-modality GCL) by removing it separately.
First, we remove co-modality GCL from CMGCL, which means we directly employ a twolayer GCN to learn the node embedding for both datasets.
As shown in Table 3, we can conclude that our model is effective as the performances on both datasets drop significantly.
In addition, we remove network pruning and intra-modality on CM-GCL.
We find the performance on both datasets decreases obviously, validating the effectiveness of network pruning and intra-modality in CM-GCL.
5.4 CM-GCL Uncovers More Minority Samples
We further investigate whether network pruning is effective to discover minority samples during contrastive learning.
Based on the pruning process introduced in Section 4.2.2, we measure the distribution of PIEs mined by CM-GCL over the different subsets (i.e., D init , D 0.1 , D 0.01 ) of
Instagram (node-image) graph.
Specifically, we sample the top 1% training data that are most easily affected by pruning, and further calculate the percentages of PIEs that belong to the minority class.
As shown in Figure 2, minority samples in D 0.1 and D 0.01 are more easily to be detected by CM-GCL along with the training epochs.
Notwithstanding the minority samples in D 0.01 are less likely to be detected at the beginning, it shows a general increasing trend along with training epochs.
D init keeps a relatively low percentage during the pre-training process.
Hence, we again demonstrate that network pruning in CM-GCL is able to mine minority samples and further learn more balanced representations.
5.5 CM-GCL is Applicable to Mainstream Models in Different Modalities
Our proposed CM-GCL aims to study various real-world graphs: text-based graphs (i e.
AMiner, YelpChi, and GitHub), image-based graphs (i.e., Instagram).
To this end, CM-GCL is designed as a plug-and-play tool that is applicable to most mainstream models in different modalities.
Hence, we adopt at least two models in each modality (i.e., graph: GCN, GAT, and GraphSAGE (SAGE); text: BERT and DistilBERT (DBERT); image: ResNet50 (ResNet) and Swin Transformer (ST)) in Figure 3 to show the strong applicability and the effectiveness of CM-GCL.
This figure shows the model size and the performance (F1) of encoder combinations over the subset D init of GitHub (node-text) and Instagram (node-image) graphs.
We find that all combinations have excellent performances.
Due to efficiency and resource concerns, We choose GCN as the graph encoder, Swin Transformer as the image encoder, and DistilBERT as the text encoder for our experiments.
In addition, besides the above models in Figure 3, our framework is applicable to other mainstream models.
5.6 Hyper-parameter Sensitivity
To explore the hyper-parameter sensitivity of
CM-GCL, we conduct four sets of experiments w.r.t.
the number R in Eq.
2 of positive node pairs for each node in intra-modality GCL, the trade-off hyper-parameter λ in Eq.
4 between inter-modality GCL and intra-modality
GCL, the percent e of pruning parameters for co-modality GCL, and the hyper-parameters ( α , γ ) in L focal (Eq. 5) for the downstream classimbalanced node classification.
Note that, the hyper-parameter sensitivity analysis about R (Figure 4.(a)) and λ (Figure 4.(b)) are based on AMiner and Instagram graphs with the initial imbalance ratio ( D init ).
The analysis in terms of the class imbalance problem is conducted on the dataset D 0 .
1 of AMiner and Instagram graphs (Figure 4.(c) and Figure 4.(d)) .
Specifically, in
Figure 4.(a), we vary the value of R in the range of { 1 , 3 , 5 , 7 , 10 , 15 , 30 } to generate positive samples for each node in intra-modality GCL.
We can find that the optimal value for AMiner graph and Instagram graph is 5 and 7, respectively, while the performance drops obviously with too many positive pairs.
That is to say, the intra-modality
GCL has inverse performance if the positive pairs are not qualified enough.
Besides, in Figure 4.(b), we vary the trade-off value λ during pre-training.
By comparing the performance when λ = 0 and λ = 0 .
5 (optimal value), we can demonstrate the effectiveness of inter-modality GCL in enhancing the performance of CM-GCL.
Meanwhile, by comparing the performance when λ = 0 .
5 and λ = 1 . 0 , we also validate the effectiveness of intra-modality GCL in CM-GCL.
In addition, from Figure 4.(c), we vary the pruning percent e in the range of { 0 . 1 , 0 . 2 , 0 . 4 , 0 . 6 , 0 . 8 , 0 . 9 } to prune encoders’ network.
We conclude that network pruning is effective for CM-GCL, while the performance will drop if the percent is large (i.e., 0.9).
Lastly, we can conclude from Figure 4.(d) that the optimal value of ( α , γ ) in L focal is (0.75,1.0) for AMiner graph and (0.25,1.0) for Instagram graph.
Conclusion
In this work, we develop a graph contrastive learning model (CM-GCL) to handle real-world multi-modality graphs with the class imbalance problem.
Specifically, we propose co-modality
GCL including inter-GCL and intra-GCL to automatically generate contrastive pairs based on the rich content.
In addition, we propose network pruning to uncover minority samples during comodality GCL pre-training.
Our proposed CM-GCL is designed as a plug-and-play tool that is applicable to most mainstream models in different modalities.
Extensive experiments across realworld graph datasets demonstrate that CM-GCL outperforms most state-of-the-art baselines for multiple downstream tasks and can alleviate the data imbalance problem.
Acknowledgments
This work is partially supported by the NSF under grants IIS-2209814, IIS-2203262, IIS-2214376,
IIS-2217239, OAC-2218762, CNS-2203261, CNS-2122631, CMMI-2146076, and the NIJ 2018-
Checklist
For all authors... (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?
[Yes] (b) Did you describe the limitations of your work?
[Yes] Detailed discussion of limitation of this work is introduced in Section 6.
(c) Did you discuss any potential negative societal impacts of your work?
[No] There is no potential negative societal impact of our work.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes] 2.
If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
[N/A] (b) Did you include complete proofs of all theoretical results?
[N/A] 3.
If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[Yes]
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
[Yes] Please refer to experimental settings in Section 5.1.3.
(c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[Yes] All reported results are the average of ten times.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
[Yes] Please refer to experimental settings in Section 5.1.3.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
[Yes] (b) Did you mention the license of the assets?
[N/A] We use datasets/code provided in original papers.
(c) Did you include any new assets either in the supplemental material or as a URL?
[Yes] Code is provided in the attachment.
(d) Did you discuss whether and how consent was obtained from people whose data you’re using/curating?
[N/A] We use datasets/code provided in existing papers.
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content?
[N/A] We use datasets/code provided in existing papers.
If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?
[N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
[N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
[N/A]