Co-Modality Imbalanced Graph Contrastive Learning
Anonymous Author(s)
AffiliationAddress email
Abstract
Graph contrastive learning (GCL), leveraging graph augmentations to convertgraphs into different views and further train graph neural networks (GNNs), hasachieved considerable success on graph benchmark datasets.
Yet, there are stillsome gaps in directly applying existing GCL methods to real-world data.
First,handcrafted graph augmentations require expert knowledge as well as trials anderrors, but still can not yield consistent performance on multiple tasks.
Second,most real-world graph data present imbalanced distribution but existing GCLmethods are not immune to data imbalance.
Therefore, this work proposes toexplicitly tackle these challenges, via a principled framework called Co-Modality
Imbalanced Graph Contratsive Learning with Network Pruning ( CMI-GCL ) to automatically generate contrastive pairs without expert knowledge and further learn balanced representation over unlabeled data.
Specifically, we design inter- modality GCL to automatically generate contrastive pairs (e.g., node-image) basedon node content.
Inspired by the fact that minority samples can be “forgotten” by pruning deep neural networks, we naturally extend the ad-hoc compressiontechnique, network pruning, to our GCL framework for detecting minority nodes.
Based on this, we co-train two pruned encoders (e.g., GNN and image encoder) indifferent modalities by pushing the corresponding node-image pairs together andirrelevant node-image pairs away.
Meanwhile, we also propose intra-modality GCLby co-training non-pruned GNN and pruned GNN, to ensure node embeddings withsimilar attributed features stay closed.
By applying pre-trained CMI-GCL to twofine-tuning modes, we demonstrate that our model significantly outperforms stateof-the-art baseline models and learns more balanced representations on real-worldgraph datasets.
Introduction
Contrastive learning, aiming to contrast instance pairs generated from unlabeled data to train better representation models, has attracted considerable attention.
Inspired by the consistent success of contrastive learning in computer vision and natural language processing [4, 3, 31], an increasing numberof works have started to investigate the great potential of contrastive learning on graphs [43, 46, 41, 40].
Most existing graph contrastive learning (GCL) models leverage different graph transformationsto augment graph data into different views and further train graph neural networks (GNNs) bymaximizing the similarity between positive pairs while minimizing the similarity between negativepairs.
In particular, PyGCL [46] summarizes nine categories of graph augmentations from differentlevels (e.g., node dropping, edge perturbation, and graph diffusion), and then implements differentaugmentation combinations to transform graph into different views for contrastive learning.
Yet, directly applying the aforementioned GCL models from the controlled benchmark graph datasetsto uncontrolled real-word graph datasets still has some gaps.
First, notwithstanding suitable handcrafted augmentation methods need professional knowledge as well as trials and errors, the consistent excellent performance on multiple tasks can not be guaranteed [46].
Second, node classes in mostreal-world graphs present imbalanced distribution (e.g., Zipf long-tail distribution).
For instance, theamount of illicit activities (e.g., drug trafficking, malicious repository, and fake review) on socialnetworks is much less than that of normal activities [36].
To tackle these challenges, we aim to designa novel GCL model that automatically generates contrastive pairs without too much efforts, but alsocan generalize well in real-world imbalanced graph.
Accordingly, inspired by existing cross-modality contrastive learning methods in computer vision [35],we naturally hypothesize that node embedding in graph and the corresponding content embedding(e.g.
, image embedding) in different modality will be closer if we co-train GNN encoder andcontent encoder (e.g., image encoder) during contrastive learning.
In this way, we can automaticallygenerate contrastive pairs, e.g., (node-image), based on the natural content on nodes, which iscalled inter-modality GCL.
In addition, existing works signal that contrastive loss can guarantee abalanced representation space [42, 19].
However, a recent study [18] demonstrates that contrastivelearning can alleviate the data imbalance problem but fails to be fully immune to the data imbalanceissue.
Motivated by the fact that minority data may be easily “forgotten” by pruning deep neuralnetworks [17], we extend the ad-hoc compression tool, network pruning, to GCL framework andfurther detect minority samples not yet well represented by encoders during co-modality GCL.
To summarize, we propose a novel framework called co-modality imbalanced graph contrastivelearning with network pruning ( CMI-GCL ) to automatically generate contrastive pairs from unlabeled data and further learn more balanced node representations.
As illustrated in Figure 1, we firstprune the graph encoder and content encoder in inter-modality GCL to detect minority samples, andfurther co-train two encoders via optimizing contrastive loss.
Besides, we also aim to ensure thatnode embeddings with similar attributed features should also stay closed.
Therefore, we proposeintra-modality GCL that generates contrastive pairs based on the similarities among node attributefeatures and further co-trains non-pruned graph encoder and pruned graph encoder.
By the aforementioned steps, CMI-GCL can boost minority nodes weights in contrastive loss and leads to implicitloss re-balancing.
To conclude, our work makes the following contributions:
• As handcrafted graph augmentations are not efficient on real-world data, we devise a co-modalityframework to automatically generate contrastive pairs.
To our best knowledge, this is the first workon GCL that co-trains encoders in different modalities to facilitate node representation learning.• To address the data imbalance issue on graphs, our model extends the compression technique,network pruning, to prune encoders during co-modality GCL for detecting minority samples andfurther learns more balanced node representations.• Extensive experiments on real-world graph datasets show the effectiveness of our model by comparing with state-of-the-art methods.
Related Work
Graph Contrastive Learning.
Most GCL models [34, 43, 41, 13, 14] leverage different types of datatransformations to augment graph into different views and further train an encoder by discriminatingpositive pairs and negative pairs generated from unlabeled data.
Specifically, GraphCL [43] designsfour types of graph augmentations to generate contrastive pairs in different views for contrastivelearning.
GCC [34] implements random walk for each node to sample subgraph as augmentation.
Inheterogeneous domain, HeCo [40] employs cross-view (schema view and meta-path view) contrastivelearning to train an encoder.
These handcrafted data augmentations require trials and errors, andsufficient domain knowledge, but still cannot yield consistent excellent performance on multipletasks [43].
Hence, we naturally propose to automatically generate contrastive pairs for graph data.
Inspired by CLIP [35], a co-modality contrastive learning model on image and text data for imagecaption generation, we devise cross-modality GCL which leverages the rich content in graph toautomatically generate useful contrastive pairs without resorting to handcrafting or expert knowledge.
Data Imbalance in Self-supervised Learning.
Data imbalance is very common in real-worldapplications (e.g., anomaly detection [6, 8]).
Most existing methods against data imbalance focuson supervised learning (e.g., re-sampling [2, 28], re-weighting [20, 38], and GraphSMOTE [45]).
Some recent works [42, 19, 18] start to explore the benefits of the balanced feature space fromself-supervised learning.
Yang and Xu [42] first studied the label bias in self-supervised learning andconcluded that classifiers that are pre-trained in a self-supervised manner consistently outperform their corresponding supervised baselines.
Kang et al.
[19] found that contrastive learning can presenta more balanced feature space over imbalanced datasets by comparison with supervised learning.
Based on this, SDCLR [18] thinks contrastive learning is not fully immune to data imbalance.
Allaforementioned works study the imbalance problem on image.
Motivated by these works, we devisean algorithm that can well mitigate the data imbalance in our proposed co-modality GCL framework.
Network Pruning.
Network pruning [23, 26, 24] has been considered as a popular compression toolfor DNN models.
In recent, some works [11, 17, 18] begin to explore the deeper connection with DNNmemorization and generalization.
Specifically, Frankle and Carbin [11] found that pruning a dense
DNN can identify “significant subnetwork” that testing accuracy is comparable to the one in originalnetwork.
Hooker et al.
[17] thought a small data subset, i.e., data belonging to ambiguous classes andminority classes, are more likely to be impacted by the introduction of sparsity via network pruning.
Based on the conclusion, Jiang et al.
[18] constructed a self-competitor via network pruning to studyimbalanced image in contrastive learning.
In this work, we introduce network pruning in co-modalitycontrastive learning to detect minority nodes and further gain more imbalanced representation.
Preliminary
Graph Neural Networks.
Given a graph G =
( V , E , X ) , where V is the set of different typesof nodes, E
⊆
V × V is the set of edges, and X is the attributed feature set.
We aim to learnthe node representation by considering both structure and node attributed features.
Current GNNs(e.g., GCN [22], GAT [39], and GraphSage [12]) follow a messaging-passing framework and hasgained consistent performance on various graph data.
In this paper, we choose GCN as the graphencoder to learn the node embedding
h iG ∈ R d G corresponding to node v i .
In particular, GCN isformulated as H l +1 =
σ ( (cid:101)
A H l W l ) , where H l +1 denotes the node representations at l + 1 layer, (cid:101)
Ais a symmetric normalization of A with self-loop, W l is the weight matrix at l -th layer, and σ is theactivation function.
For simplicity, we use f G to denote the graph encoder and h G to denote the nodeembedding.
Besides GCN, we also discuss GAT and GraphSage as graph encoder in experiments.
Text Pre-Trained Model.
Existing graph representation learning methods [10, 44, 32] generallyimplement the pre-trained language models (e.g., BERT [7]) to extract features from text contentas the attributed feature for each node.
Instead of this, we first adopt DistilBERT [37] (a distilledversion of BERT) as the pre-trained text encoder to obtain the attributed features for each node, andthen fine-tune DistilBERT with contrastive loss.
The structure of DistilBERT we implemented is abase size with a 12-layer 512-wide model along with 8 attention heads.
Text sequence is bracketedwith [SOS] and [EOS] tokens and we take the activation of the highest layer of transformer at [EOS]token as text representation h T ∈ R d T .
We utilize f T to denote text encoder.
After fine-tuning f
Tvia optimizing co-modality contrastive loss, we update fine-tuned attributes for all nodes.
Besides
DistilBERT, we also discuss BERT as the text encoder and more details can be found in Section 5.5.
Image Pre-Trained Model.
We implement two different architectures (i.e., ResNets and Transformer)as image encoder.
As ResNets is widely used and has been proved with excellent performance, weimplement ResNet50 [15] as the base architecture.
Following [16], we make the similar modificationson ResNet and replace the global average pooling layer with an attention pooling mechanism.
Besides,we consider the recently introduced Swin Transformer [27] as the image encoder and closely followtheir implementation.
In this paper, with the consideration of performance and efficiency (as discussedin Section 5.5), we take Swin Transformer as the image encoder denoted as f I to obtain the imagerepresentation h I ∈
R
d I .
After training f I via contrastive loss optimization, we attach the obtainedimage embedding to the corresponding nodes as attributed features for downstream tasks.
Problem 1.
Co-Modality Imbalanced Graph Contrastive Learning.
Given a graph G =
( V , E , X )
built on multi-modality data along with imbalanced label Y , we aim to build a GCL model over
G that can be further fine-tuned or applied to the downstream tasks with imbalanced labels (e.g.,imbalanced node classification).
The Proposed Model
In this section, we present the details of CMI-GCL (Figure 1) which includes three steps: (i) graphconstruction (Figure 1.(a)); (ii) co-modality imbalanced graph contrastive learning with networkpruning (Figure 1.(b)); (iii) two fine-tuning modules for downstream
tasks(Figure 1.(c)).
4.1 Graph Construction
Graph has been proved to be effective in modeling real-world networked data (e.g., social network).
Thus, to model real-world data for certain tasks (e.g., malware detection on GitHub data), we introducegraph to depict both content and structure information in the data.
In this work, by following previousstudies [33, 32] that provides GitHub and Instagram graph datasets, we introduce four types of entities(i.e., user, repository, file, and keyword) and seven types of relations in GitHub graph; three types ofentities (i.e., user, post, and keyword) and eight types of relations in Instagram graph.
In addition, weimplement the pre-trained language model DistilBERT [37] and Swin Transformer [27] to encodethe text and image respectively, and apply them to the corresponding nodes as feature attributes.
Different from previous works, we use original content (e.g., image and text) to facilitate contrastivepairs generation and further co-train graph encoder and content encoder via contrastive learning.
4.2 Co-Modality Imbalanced Graph Contrastive Learning
After graph construction, we first propose inter-modality contrastive learning to automatically generatecontrastive pairs.
Then we introduce network pruning to detect minority nodes during contrastivelearning.
Last, we design intra-modality contrastive learning to ensure that node embeddings withsimilar semantics (attributed features) stay closed.
4.2.1 Inter-Modality Graph Contrastive Learning
The main idea of contrastive learning is to make representations maximize the agreement on samplepairs under proper graph augmentations.
Yet, finding proper graph augmentation for every task isalways resource-consuming [43].
Inspired by existing work [35] of multiple-modality learning, wepropose to fully utilize nodes’ content to automatically generate contrastive pairs, and further co-trainencoders in different modalities via contrastive learning.
That is to say, we aim to learn two encoderssuch that embeddings in two modalities are close to each other in the learned space if they referto the same entity, otherwise far away.
In this paper, we consider two combinations of modalitiesfor graphs constructed in Section 4.1 (i.e., graph-image for Instagram graph, graph-text for GitHubgraph).
For simplicity, we discuss graph-image modalities as an example in this section, and willalso conduct experiments on graph-text modalities.
Specifically, given a sample pair ( x iG , x iI ), wherex iG is the attributed feature of node v i ∈ V and x iI is the image input corresponding to node v i ,we feed x iG to graph encoder f G and x iI to image encoder f
I respectively and further obtain thecorresponding embedding ( h i G , h i I ).
Note that, as different types of nodes in G may have differentfeature dimensions, we first transform the input attributed features of all nodes to a common space.
After getting the embedding pair ( h iG , h iI ), we apply nonlinear projection heads to convert embeddingfrom different modalities to the same space (denoted as ( z iG , z iI ) ) for comparison.
We consider ( z iG ,z iI ) as a similar pair (positive pair) if the node and the image refer to the same entity.
Otherwise, ( z iG ,z iI ) is viewed as a dissimilar pair (negative pair).
4.2.2 Network Pruning for Imbalance Data
Whether contrastive learning is immune to data imbalance or not has remained controversial.
Recentstudy [18] claim that contrastive learning can alleviate the data imbalance issue in representationlearning but is not fully immune to data imbalance.
To validate that, we define a metric to describethe balanced property of a feature space.
Typically, the linear separability is evaluated by the accuracyof a linear classifier over representations.
A feature space U can be considered as balanced if therepresentation u i from different classes has similar degrees of linear separability.
Typically, the linearseparability is evaluated by the accuracy of a linear classifier over representations.
Specifically, weutilize a 1 , a 2 , ..., a c to denote the classification accuracy of a linear classifier over representationu i ∈ U from c classes.
Then the balancedness metric B of feature space U is defined as:
[Equation]
, where a k = # { u i | ˆ y i = y i
=
k, u i ∈ U } # { u i
[Equation]
y i
[Equation]
k
[Equation]
u
[Equation]
U
[Equation]
where ξ is a scaling hyper-parameter, and ˆ y is the classification result.
B ( U ) achieves its maximumwhen all class-wise accuracy are equal, which means there is no separability bias of learned representations to any class.
We conduct experiments with B ( U ) in Section 5.2.2 and further conclude thatexisting GCL will not be able to produce balanced representations if data is heavily imbalanced.
Based on the above finding, we propose to introduce a method to address the data imbalance issueduring co-modality GCL.
The recent work [17] finds that certain examples are particularly sensitive tosparsity via network pruning and these samples mostly impacted after pruning are termed as Pruning
Identified Exemplars (PIEs) [17].
They further demonstrate that PIEs often show up at the minorityclasses.
Inspired by this, we extend network pruning to our proposed framework to detect minoritysamples.
With PIEs dynamically generated during contrastive training, more minority samples willbe exposed along with training epochs, as we will show in Figure 2 of experiments.
We implementmagnitude pruning method that uses absolute value of weights to rank their importance and removesweights that are below a certain threshold.
Specifically, the detailed process is described as follows:
Randomly initialize encoders f G and f I with parameters θ 0 G and θ 0 I respectively.2.
At each iteration l , prune e % of smallest-magnitude parameters in θ lG and θ lI , creating thecorresponding masks m lG and m lI .
Apply masks m lG and m lI to the feed-forward encoders and calculate the contrastive loss.4.
Release masked weights and update encoders’ parameters via optimizing the contrastive loss.
In the above steps, the contrastive loss in inter-modality GCL is formulated as:
[Equation]
where ( (cid:101) z iG , (cid:101) z iI ) is the projected embedding generated by pruned encoders (cid:101) f G and (cid:101)
f I , sim ( (cid:101) z iG , (cid:101) z iI ) isthe cosine similarity between (cid:101) z iG and (cid:101)
z iI , τ inter is the temperature value, and n is the size of mini-batch.
Following the iterative pruning process that repeatedly trains, prunes and releases weights in eachround, it effectively boosts minority samples’ weights in L inter and leads to implicit loss re-balancing.
4.2.3 Intra-Modality Graph Contrastive Learning
Inter-modality graph contrastive learning ensures that embedding from different modalities w.r.t.
thesame entity stay closed in the projected space but fails to ensure that embedding with similar originalfeatures from the same modality stay closed.
We believe nodes with similar attributed featuresnaturally have similar semantics and their embedding should be closer, while most existing GNNsare mainly dominated by relation among nodes [32].
In light of this, we propose intra-modality graphcontrastive learning to balance the importance of local neighbor relations and attributed features, andfurther ensure that nodes with similar attributed features will stay closed in the projected space.
Notethat, in intra-modality graph contrastive learning, we mainly focus on graph modality as we believefeatures among nodes have rich semantics.
Specifically, we first define S t and S o as sets of node pairshaving the top-R similarity and the bottom-R similarity respectively, which is formally defined as:
[Equation]
where i ̸ = p , sim ( x iG , x pG ) measures the cosine similarity of node attributes between node v i and v p ,
R is the number of top/bottom pairs chosen for each node, and N is the number of nodes in G .
The main difference between intra-modality GCL and inter-modality GCL lies in that, intra-modality
GCL implements one graph encoder and another pruned graph encoder for contrastive training, whileinter-modality GCL implements one graph encoder and one image (or text) encoder for co-modalitycontrastive training.
Hence, we have a dense graph encoder f G and a sparse graph encoder (cid:101)
f G prunedby iterative magnitude pruning.
Given a sample pair ( x iG , x pG ) , it will be encoded by f G and (cid:101)
f
Grespectively.
The projected embeddings ( z iG , (cid:101) z pG ) is regarded as a positive pair if ( v i , v p ) in S t , whileit is a negative pair if ( v i , v p ) in S o .
The intra-modality GCL loss is defined as follows:
[Equation]
By this way, for atypical and minority nodes, our model can amplify the prediction differencesbetween the pruned and non-pruned graph encoders in intra-level, and further these nodes’ weightswill be implicitly increased in the intra-modality contrastive loss L intra .
After performing inter-modality pruning contrastive learning and intra-modality pruning contrastivelearning, the overall objective of co-modality pruning contrastive learning can be formulated as:
[Equation]
5) where λ is the trade-off hyper-parameter for balancing two loss terms.
Pseudo-code of CMI-GCL isprovided in Section A of Appendix.
4.3 Fine-tuning
Inspired by GCC [34], we introduce two fine-tuning strategies, i.e., full fine-tuning mode for downstream tasks and freezing mode for balancedness (linear separability) evaluation.
Full fine-tuning.
In full fine-tuning mode, we adopt a multi-step protocol to evaluate the classificationperformance.
(i) Representation learning via CMI-GCL: we pre-train encoders by optimizing thepre-training loss L .
(ii) Fine-tuning : we adopt the pre-trained weights from f ∗ G (pre-trained graphencoder) as the initialization of the fine-tuning encoder and update the attributed features generated byf ∗ I (pre-trained image encoder).
Then we train the fine-tuning encoder together with the classifier (i.e.,
MLP) using focal loss, a specific loss for imbalanced classification [25].
(iii) Classification evaluation:we evaluate the classification performance over imbalanced nodes with different imbalance ratios.
Freezing.
In freezing mode, we also adopt a three-stage protocol to evaluate the balancedness oflearned representations.
(i) Representation learning: similar to full fine-tuning mode, we obtain f ∗ Gvia CMI-GCL; (ii) Classifier learning with pre-trained embedding: we freeze parameters of f ∗ G , andfurther train a linear classifier ˆ y =
( w ⊤ h ∗ G + ϵ ) with the supervised cross-entropy loss.
Here wdenotes the classifier weight matrix, ϵ denotes the bias term, h ∗ G is the pre-trained node embedding.(iii) Representation evaluation: we evaluate the classification accuracy on h ∗ G and further calculatethe proposed balancedness metric B ( U ) .
Experiments
In this section, we first introduce datasets we use to evaluate CMI-GCL and baselines models.
Thenwe provide the detailed analysis to show the effectiveness of CMI-GCL and its strong applicabilityto real-world graph datasets.
More about experiments, i.e., implement details, hyper-parametersensitivity, and ablation study with different imbalanced ratios, are provided in Appendix.
Last, tomake CMI-GCL more convincing, we also apply CMI-GCL to AMiner data [1], a public papercitation benchmark data with raw content.
More details are provided in the revised appendix file.
5.1 Experimental Setup5.1.1 Dataset
In this paper, we adopt two multi-modality graph datasets from existing works, i.e., GitHub graph [32]and Instagram graph [33], which include raw content and graph information on these two platforms.
Specifically, GitHub graph provided by [32] aims to detect malicious repositories, which containsall content of repositories, users, and files on GitHub.
The GitHub graph consists of 174,661 nodes(i.e., 20,895 repository nodes, 52,530 keyword nodes, 82,451 file nodes, and 18,785 user nodes)and 696,464 edges among nodes.
Instagram graph provided by [33], having all image and textinformation about users and posts, aims to detect illicit drug traffickers on Instagram.
The Instagramgraph contains 129,894 nodes (i.e., 8,651 user nodes, 79,705 post nodes, 41,538 keyword nodes) and218,039 edges among nodes.
Both datasets have two types of node labels (i.e., 6,965 malicious and13,930 benign repositories on GitHub, 3,242 drug traffickers and 5,409 normal users on Instagram).
5.1.2 Baseline Method
To evaluate the performance of CMI-GCL, we implement two tasks, i.e., binary classification overimbalanced data (Task 1) and balancedness evaluation over balanced data (Task 2).
Specifically,for Task 1, we compare CMI-GCL with fourteen baseline methods which are divided into fourgroups (Table 1): feature-based method (G1), graph learning models (G2), GCL models (G3), andmethods against imbalanced data (G4).
For G1, we feed feature vector to a two-layer MLP [29].
For G2, we implement seven graph learning methods, i.e., Deepwalk [30], mp2vec [9], GCN [22],
GAT [39], and GraphSage [12], to learn the node representations.
Meta-AHIN [32] is the work thatprovides the GitHub data and develops a graph model to detect malicious repositories.
MetaHG [33]is the work that provides the Instagram data and develops a meta-learning model to detect illicitdrug traffickers.
For G3, we implement three popular approaches, i.e., over-sampling [21], reweighting [5], and GraphSMOTE [45], to handle the imbalanced node representations generated by
GCN.
GraphSMOTE is the extension of SMOTE [2] on graph data, which leverages node similaritiesto generate synthesized nodes and edges.
For G4, we reproduce three GCL methods, i.e., GCC [34],
GraphCL [43], and HeCo [40].
To compare them with CMI-GCL fairly, we fine-tune the pre-trainedmodels with Focal loss as well.
Besides, for Task 2, we compare CMI-GCL with SimCLR [3] (CMIGCL without pruning), GCC, GraphCL, and HeCo to evaluate the balancedness of representations(Table 2).
Details of baseline settings are described in Section B of Appendix.
5.1.3 Experimental Settings
All experiments are conducted under the environment of the Ubuntu 16.04 OS, plus Intel i9-9900k
CPU, two GeForce GTX 2080 Ti Graphics Cards, and 64 GB of RAM.
We train all methods witha fixed epoch (1000) and use the early stopping strategy with a patient of 50.
Besides, all methodsare trained ten times, and the average performance multiplied by 100 on testing data is reported.
Weuse 70% samples for training, 10% for validation, and the remaining 20% for testing.
Followingexisting works in evaluating imbalanced classification [2, 45], we adopt two metric, i.e., F-score (F1)and AUC-ROC score (AUC) to evaluate all models.
Besides, we define B ( U ) in Eq.
1 to evaluatethe balancedness of representations learned from GCL.
In this paper, the initial imbalance ratioβ of repository data and user data are 0.5 (1:2) and 0.6 (1:1.7), respectively.
To evaluate modelperformance in more imbalanced scenarios, we set β as 0.1 and 0.01 and further utilize D init , D b ,
D 0.1 , and D 0.01 to denote dataset with initial imbalance ratio, 1.0, 0.1, and 0.01, respectively.
Notethat for Task 1, we leverage D init to train CMI-GCL and fine-tune the pre-trained encoder over D init ,
D 0.1 , and D 0.01 to get the classification results.
For Task 2, we leverage D init , D b , D 0.1 , and D 0.01to train all GCL models respectively, and further apply the pre-trained embeddings as features to alinear classifier with D b .
When pre-train CMI-GCL, with the grid search, the scaling parameter ξ in
B ( U ) is set as 0.01, pruning ratio e is set as 20%, the number of contrastive pairs for each node R inintra-modailty GCL is set as 10, and n for each mini-batch is set as 200.
Besides, the temperatureparameter τ inter and τ intra are set as 1.0 and the trade-off hyper-parameter λ among co-modality GCLis set as 0.6.
For fine-tuning, α and γ in Focal loss [25] are set as 1.0 and 2.0, respectively.
5.2 Performance Comparison
5.2.1 CMI-GCL Helps Downstream Tasks with Imbalanced Classes
Table 1 shows the performances of all models on GitHub and Instagram graphs with differentimbalance ratios.
The best performance are highlighted in bold and the best baseline results areemphasized by underline.
In Table 1, we can conclude that: (i) Merely considering content data (G1)or relations among nodes (Deepwalk and mp2vec in G2) is not supportive to learn node representationsin imbalanced graphs.
Integrating rich content and relations in graph can learn better representations.
[Table]
(ii) Leveraging unlabeled data to train graph encoder via contrastive learning can enhance noderepresentations (G3).
Note that all models in G3 are based on the embeddings generated from a [Table] two-layer GCN.
(iii) Some popular methods against imbalanced data in supervised learning canalleviate the influence of imbalance and further improve the performance to some extent.
However,they fail to handle heavily imbalanced data ( D 0.01 ).
(iv) CMR-CLR has the best performance whendealing with heavily imbalanced data.
By comparing CMR-CLR with GCL models in G4, we canconclude that CMR-CLR can learn better representations for real-world imbalanced graphs.
5.2.2 CMI-GCL Improves Balancedness on Imbalanced Data
[Table]
[Table]
We now investigate if contrastive learning isvulnerable to data imbalance.
Based on Eq.
1,the maximum value of B ( U ) is 0.5 in this paperwhen the accuracy of the two classes is the same.
That is to say, the classification doesn’t haveany imbalance.
As shown in Table 2, we canconclude that: (i) For both datasets, models pretrained on D b show larger balancedness thanimbalanced subsets.
(ii)
The balancedness isworse when the imbalance ratio β is smaller.
Even so, contrastive learning can still alleviatethe influence of data imbalance if the data is notheavily imbalanced (i.e., D init and D 0.1 ).
(iii)
GCL models become fragile when trained over a heavily imbalanced dataset ( D 0.01 ), which validatesthe necessity of addressing the imbalance problem in GCL.
(iv) CMI-GCL can learn more balancedrepresentations over heavily imbalanced data, which demonstrates the effectiveness of CMI-GCL inlearning balanced representations.
5.3 Ablation Study
[Table]
F
AUC
[Table]
[Table] [Table] To show the effectiveness of different components in CMI-GCL, we conduct a set of ablationexperiments over subset D 0.1 and further analyze the contribution of each component (i.e.,co-modality GCL, pruning, and intra-modality
GCL) by removing it separately.
First, we remove co-modality GCL from CMI-GCL, whichmeans we directly employ a two-layer GCNto learn the node embedding for both datasets.
As shown in Table 3, we can conclude that ourmodel is effective as the performances on both datasets drop significantly.
In addition, we remove network pruning and intra-modality on CMI-GCL.
We find the performance on both datasets decreases obviously, validating the effectiveness of networkpruning and intra-modality in CMI-GCL.
5.4 CMI-GCL Mines More Samples from Minority Class
We further investigate whether network pruningis effective to detect minority samples duringcontrastive learning.
Based on the pruning stepsintroduced in Section 4.2.2, we measure the distribution of PIEs mined by CMI-GCL.
Specifically, when pre-train on different subsets (i.e.,
D init , D 0.1 , D 0.01 ), we sample top 1% trainingdata that are most easily affected by pruning,and further calculate the percentages of PIEsthat belong to the minority class.
As shown in
Figure 2, minority samples in D 0.1 and D 0.01 aremore easily to be detected by CMI-GCL alongwith the training epochs.
Notwithstanding theminority samples in D 0.01 are less likely to bedetected at the beginning, it shows a generalincreasing trend along with training epochs.
D init keeps a relatively low percentage during thepre-training process.
Hence, we again demonstrate that network pruning in CMI-GCL can detectminority samples and further learns more balanced representations.5.5 CMI-GCL is Applicable to Mainstream Models in Different Modalities
Our proposed CMI-GCL aims to study various real-world graphs: text-based graph (e.g.,
GitHub), image-based graph (e.g., Instagram).
To this end, CMI-GCL is designed to be a plugand-play tool which is applicable to most mainstream models in different modalities.
Hence,we adopt at least two models in each modality (i.e., graph: GCN, GAT, and GraphSAGE(SAGE); text: BERT and DistilBERT (DBERT);image: ResNet50 (ResNet) and Swin Transformer (ST)) to show the strong applicabilityand the effectiveness of CMI-GCL.
Figure 3shows the model size and the performance ofencoders (i.e., F1) on Task 1 with dataset D init .
We find that all combinations have excellent performances on imbalanced dataset.
Due to the efficiency and resource concerns, We choose GCN asthe graph encoder, Swin Transformer as the image encoder, and DistilBERT as the text encoder.
Conclusion
In this work, we aim to develop a graph contrastive learning model (CMI-GCL) on real-worldmulti-modality imbalanced graphs.
Specifically, we propose co-modality GCL including inter GCLand intra GCL to automatically generate contrastive pairs based on the rich content.
In addition,we propose network pruning to detect minority samples during co-modality GCL.
Our proposed
CMI-GCL is designed as a plug-and-play tool which is applicable to most mainstream models indifferent modalities.
Extensive experiments across real-world graph datasets demonstrate that CMIGCL outperforms state-of-the-art baselines for multiple applications and can significantly alleviatethe data imbalance problem in graph.
The limitation of this work lies in that, most benchmark graphdatasets merely provide graph data but the raw content (e.g., image and text) in graph is not available.
In the future, we will extend CMI-GCL to more real-world graph datasets.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’s
contributions and scope?
[Yes]
(b) Did you describe the limitations of your work?
[Yes] Detailed discussion of limitation of this work is introduced in Section 6.
(c) Did you discuss any potential negative societal impacts of your work?
[No] There is
no potential negative societal impact of our work.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to
them?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[N/A]
(b) Did you include complete proofs of all theoretical results?
[N/A]
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi-
mental results (either in the supplemental material or as a URL)?
[Yes] Please refer to
the Appendix.
We provide a URL about our source code.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
[Yes] Please refer to experimental settings in Section 5.1.3.
(c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[Yes] All reported results are the average of ten times.
(d) Did you include the total amount of compute and the type of resources used (e.g.,type of GPUs, internal cluster, or cloud provider)?
[Yes] Please refer to experimental settings in Section 5.1.3.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes]
(b) Did you mention the license of the assets?
[N/A] We use datasets/code provided in original papers.
(c) Did you include any new assets either in the supplemental material or as a URL?
[Yes]
Code is provided in the attachment.
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[N/A] We use datasets/code provided in existing papers.
(e) Did you discuss whether the data you are using/curating contains personally identifiable
information or offensive content?
[N/A] We use datasets/code provided in existing
papers.
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if
applicable?
[N/A]
[Table]
[Table]
[Table]
[Table]