G RAPH C ONTRASTIVE L EARNING WITH R EINFORCED A UGMENTATION
Anonymous authors
Paper under double-blind review
A BSTRACT
Graph contrastive learning (GCL), designing contrastive objectives to learn embeddings from augmented graphs, has become a prevailing method for learning embeddings from graphs in an unsupervised manner.
As an important procedure in GCL, graph data augmentation (GDA) directly affects the model performance on the downstream task.
Currently, there are three types of GDA styles: trial-and-error, precomputed method, and adversarial method.
However, these strategies ignore the connection between the two consecutive augmentation results because GDA is regarded as an independent process.
In this paper, we regard the GDA in GCL as a
Markov decision process.
Based on this point, we propose a reinforced method, i.e., the fourth type of GDA strategy, using a novel Graph Advantage Actor-Critic (GA2C) model for GCL.
On 23 graph datasets, the experimental results verify that GA2C outperforms the SOTA GCL models on a series of downstream tasks such as graph classification, node classification, and link prediction.
I NTRODUCTION
Graph representation learning aims to extract low-dimensional representation vectors from the graph data.
These representation vectors encode the abundant structural and semantic information of the graph.
Graph representation learning has become an effective technique of graph mining and can be applied to a series of fields including bioinformatics Ang et al.
; Wang et al.
(2021), social networks Matakos et al.
; Shen et al.
(2022), and
recommender systems Xu et al.
; Carroll et al.
Due to the sparsity of labeling information in the real world, learning representations from graphs in a self-supervised manner has become a new research highlight Wu et al.
Graph contrastive learning (GCL), as an important member of self-supervised representation learning on graphs, has achieved state-of-the-art (SOTA) performance on many downstream tasks Yu et al.
(2022b);
Suresh et al.
Graph data augmentation (GDA) is an important procedure in GCL.
It defines the augmented views (i.e., the specific contents of input data for contrastive learning) and affects the model performance on the downstream task directly.
Typically, GDA strategies include edge adding, edge removing, attribute masking, and so on.
Prior work designs three types of GDA styles in total, including trial-and-error, precomputed method, and adversarial method.
In the trial-and-error (e.g., GRACE Zhu et al.
and BGRL Thakoor et al.
and precomputed method (e.g., GCA Zhu et al.
(2021)), GDA is frozen in the training.
While in the adversarial method (e.g., AD-GCL Suresh et al.
(2021)), GDA is learnable and can be optimized by the view learner in the training.
Motivation: The Training Bottleneck in the Learnable GDA.
Currently, GCL with learnable
GDA unleashes the potential of contrastive learning and has achieved SOTA performance in the downstream task.
Then a new potential question arises: how does a good augmented view evolve to promote graph contrastive learning?
As an example, we plot the results of AD-GCL in Figure 1(a) where the x-axis, left y-axis, and right y-axis represents the epoch index, logarithmic number of removed edges, and area under the ROC curve (AUC) on the test data, respectively.
We can see that the evolution of GDA does not promote graph contrastive learning well: on the one hand, after the sixth epoch, GDA does not work anymore; on the other hand, the AUC performance on the test data is unstable and has high variance.
It indicates that the current learnable GDA in the GCL model has a potential bottleneck in the training, i.e., the problematic design mechanism of GDA makes the GCL model be updated effectively .
We expect to achieve an advanced GDA that can evolve progressively.
[Figure]
In this work, we propose reinforced GDA for GCL, making the augmentation procedure explore an optimal learning path actively.
The view learner in GDA is regarded as an agent that interacts with the encoder network, such as graph isomorphism networks (GIN)
Xu et al. (2019a), in the training environment.
Based on this idea, we design a Graph Advantage Actor-Critic (GA2C) model where the joint use of the Actor submodel and Critic submodel contributes to the progressive evolution of
GDA.
To illustrate it, we plot the result of GA2C in Figure 1(b).
Surprisingly, like a student doing exercises from easy problems to hard problems, we find that the view learner actively explores a learning path corresponding to the order from simple topology structures to complex ones.
At the same time, the AUC performance on the test data improves from 65% to 75% and keeps relatively stable after the 32nd epoch.
The main contributions of this work are summarized as follows:
• We propose a reinforced GDA strategy that formulates a Markov Decision Process for GDA.
• Under this strategy, we design a Graph Advantage Actor-Critic (GA2C) model to achieve continuous and learnable GDA.
The view evolution in this model significantly promotes GCL.
• On 17 graph datasets covering 5 different types, we verify that GA2C is robust and superior to other graph contrastive learning models on the graph classification task.
P RELIMINARIES
Graph contrastive learning.
Graph contrastive learning (GCL) is an important type of unsupervised learning on the graph Wu et al.
As shown in Figure 2, GCL typically includes three main procedures: graph data augmentation (GDA), network encoding, and contrastive loss based optimization.
Assume an input graph G i is sampled from the distribution P ( G ) defined over a universal graph set G , two augmented views are obtained by two specific GDA operations
A ´ ω and A ` ω where ´ ω and ` ω are two augmentation parameters.
A ´ ω and A ` ω are from the distribution P ( A 1 ,A 2 ) defined over all possible parameters of two augmentation strategies A 1 and A 2 .
The two views are then passed into the encoder network f ϕ ( · ) and the projection network g φ ( · ) .
Finally, the graph embedding is calculated by an optimization process based on a predefined contrastive loss.
The definition of graph contrastive learning is as follows:
Definition 1 .
(Graph contrastive learning) .
Given an attributed graph G i =( A G i , X G i ) and its two augmented views ´ G i =(
A ´ G i , X ´ G i ) , ` G i =(
A ` G i , X ` G i ) (adjacency matrices A G i , A ´ G i , A ` G i ∈ R N i × N i , attribute matrices
X G i ,
X ´
G i , X ` G i ∈
R N i × M i ), the goal of graph contrastive learning is to learn a function F GCL : G → Z or N → H , where G is graph space ( G i , ´ G i , ` G i ∈ G ), N node space ( u, v, · · · ∈ N ), Z graph embedding space, and H node embedding space.
For the graph embedding Z i ∈ R D ( D ≪ M i ) in Z , we call it the contrastive embedding of graph G i .
For the node embedding h u ∈ R D , we call it the contrastive embedding of node u .
[Figure]
Downstream tasks.
GCL is generally used as a pretext task which is followed by downstream tasks such as graph classification, node classification, and link prediction.
When GCL is complete, the trained parameters ϕ and φ in the encoder network and projection network are used
𝐴 ́𝝎 !
(#) (#) 𝐴 ̀𝝎 !
(#) (#) 𝑓 𝝓 (#) 𝑓 𝝓 (#) 𝑔 𝝋 (#) 𝑔 𝝋 (#) 𝐺 ’(() ́𝝎 ’(() , ̀𝝎 ’(() 𝐴 ́𝝎 !
()) (#) 𝐴 ̀𝝎 !
()) (#) 𝑓 𝝓 (#) 𝑓 𝝓 (#) 𝑔 𝝋 (#) 𝑔 𝝋 (#) + 𝐺~ℙ (*) ́𝝎 ’(+) , ̀𝝎 ’(+) Learnable part + Contrastive loss based optimization
Cumulative reward based optimization 𝐴 ́𝝎 !
(#) (#) 𝐴 ̀𝝎 !
(#) (#) 𝑓 𝝓 (#) 𝑓 𝝓 (#) 𝑔 𝝋 (#) 𝑔 𝝋 (#) 𝐺 ’(+) …
Time
Time T
Frozen part
First stage
Second stage ́𝝎 ’(+) , ̀𝝎 ’(+) as the initial solution of the model in a downstream task.
Taking the graph classification task for an example, its definition is as follows:
Definition 2 .
(Graph classification) .
Given an input set of n attributed graphs D = {G 1 , G 2 , · · · , G n } and the label l i of each graph G i , the goal of graph classification is to learn a function F GC : G → L ( D ⊂ G , l i ∈ L ), where G is graph space and L label set.
M ETHODS
In this section, we present the proposed graph reinforced augmentation framework and GA2C model.
For frequently used notations in the paper, we list and describe them in Table 1.
3. G RAPH REINFORCED AUGMENTATION FRAMEWORK
this paper
[Table]
Reinforcement learning involves no supervisor and only a reward signal is used for an agent to determine if it is doing well or not.
From this point, reinforcement learning is suitable to characterize an evolving view learner in GDA.
We illustrate the graph reinforced augmentation framework in Figure 3.
Specifically, there are two novel designs in this framework as follow, 1.
Alternate optimization of contrastive loss and cumulative reward .
We design two-stage model learning: the first stage is to maximize the expected cumulative reward between two augmented graph embeddings ´ Z ( t ) i and ` Z ( t ) i at time t by only updating GDA parameters; the second stage is to minimize the InfoNCE loss L ( · ) between two views by only updating the encoder and projection parameters ϕ and φ .
The whole optimization objective is as follows:
[Equation]
where J A ( J C ) is the cumulative reward function of the Actor (Critic) model.
Continuous and learnable GDA .
Different from the discrete and frozen GDA in the prior work, the GDA procedure in our framework is continuous and learnable.
The continuity ensures that the view learner can adjust the current augmentation parameters according to the state at the previous time.
This contributes to that the augmented views evolve to a better result steadily in the training.
3. G RAPH ADVANTAGE A CTOR -C RITIC MODEL
In this subsection, we present the design of GA2C in detail, including the Markov decision process for GDA, model architecture, updating Actor, and updating Critic.
Markov decision process for GDA.
The determination of GDA parameters is regarded as a Markov decision process: given the present augmented view ´
G ( t ) i of graph
G i at time t , the augmented view ´ G ( t +1)
i of next time is independent of any past view ´
G ( t ′ )
i where t ′ <
t (note that we omit the discussion about ` G ( t ) i and ` ω ( t +1)
i to save more space because they have the same property as ´ G ( t ) i and ´ ω ( t +1) i ).
It can be formalized as follows:
[Equation]
Since edge removing is an effective and popular GDA choice for graph classification, we adopt edge removing as the GDA strategy of GA2C and ´ ω ( t ) i represents the removing probability of edges in ´ G i at time t .
A Markov decision process for GDA can be formulated as a 4-tuple ( S, A, P, R ) :
State ( S S is the set of all possible
augmented views A state s in S represents
a certain augmented view ´ G i .
Also, the state at time t is denoted as ´ G ( t ) i
• Action ( A ):
A is the set of all possible augmentation operations.
An action a in A represents a certain augmentation operation
A
´ ω
i ( · ) .
Also, the action at time t is denoted as A ´ ω ( t ) i .
For example, given a graph G = {V , E} where V = { v 1 , v 2 , v 3 } and E = { ( v 1 , v 2 ) , ( v 1 , v 3 ) } , we define an edge index dictionary E = { 1:( v 1 , v 2 ) , 2:( v 1 , v 3 ) } .
Then the action set for the edge removing strategy is A = { [0 , 0] , [0 , 1] , [1 , 0] , [1 , 1] } where “0” and “1” represents removing and retaining, respectively (e.g., [0 , 1] represents removing the 1st edge ( v 1 , v 2 ) and retaining the 2nd edge ( v 1 , v 3 ) ).
• Transition probability ( P ): P is a state transition probability matrix where the entry p ss ′ corresponds to the probability transiting from state s to state s ′ .
• Reward ( R ): R is the total reward of the trajectory measuring the reward value generated by a series of states. Also, the reward at time t is denoted as R ( t ) .
To decrease the redundant information between two views as much as possible, we set R t as the negative mutual information between two augmented graph embeddings ´ Z ( t ) i and ` Z ( t ) i , i e., R ( t ) = − I ( ´ Z ( t ) i , ` Z ( t ) i ) where ´ Z ( t ) i =
g φ ( f ϕ ( A ´ ω
i ( G i ))) and ` Z ( t ) i = g φ ( f ϕ ( A ` ω i ( G i )))
[Figure]
Model architecture. The Graph Advantage
Actor-Critic (GA2C) model contains two submodels of Actor and Critic.
The advantage function in GA2C can be approximated by the temporal difference (TD) error.
We illustrate GA2C in Figure 4.
The Actor model with parameter θ A receives the information in ´ G ( t ) i and estimates the GDA parameter ´ ω ( t +1)
i corresponding to an augmentation operation
A ´ ω ( t +1) i ( · ) by threelayer graph neural networks:
[Equation]
where BN( · ) is the batch normalization layer, READOUT( · ) the readout layer calculating the graph embedding from node embeddings, h ( k ) v the embedding of node v ( v ∈V i ) at the k -th layer neural network, N ´ G ( t )
i ( v ) the node v ’s neighbor set in ´ G ( t ) i , and h (0) u corresponds to the attribute vector of node u .
The Critic model with parameter θ C receives ´ G ( t ) i , ´ ω ( t +1)
i and estimates the Q-value ´ Q ( t +1) i by three-layer graph neural networks.
Like simulating an action on a state, we conduct the edge removing operation with parameter ´ ω ( t +1) i on ´ G ( t ) i and then obtain ´ G ( t +1) i
[Equation]
The node v ’s neighbor set in ´ G ( t +1)
i is denoted as N ´ G ( t +1)
i .
The Critic model is formalized as follows:
[Equation]
We train the Actor model and Critic model using the gradient ascent algorithm.
Specifically, it contains the following two aspects.
Updating Critic.
The optimization objective of the Critic model is J C
= − (cid:80) Tt {+=
+}
[ ´ A ( t ) i ( ´ G ( t ) i , ´ ω i )]and the gradient of the Critic parameter θ C is computed as follows:
[Equation]
where the advantage function ´ A ( t ) i ( ´ G ( t ) i , ´ ω i ) denotes how much better it is to take the action of edge removing with parameter ´ ω
i compared to the average action at the given state ´ G ( t ) i .
For ´ A ( t ) i ( ´ G ( t ) i , ´ ω i ) , we use the difference of the Q-value ´ Q ( t ) i ( ´ G ( t ) i , ´ ω i ) and V-value ´ V ( t ) i ( ´ G ( t ) i
| ` G ( t ) i ) :
[Equation]
where ´ Q ( t ) i ( ´ G ( t ) i , ´ ω i ) is defined in Eq.
5 and ´ V ( t ) i ( ´ G ( t ) i | ` G ( t ) i ) is defined as follows:
[Equation]
Updating Actor.
The optimization objective of Actor is J A
= (cid:80) Tt =1 log π i [Equation]
| [Equation] i
[Equation] [Equation] )+ λ J reg ( λ is the coefficient of regularization term) and the gradient of the Actor parameter θ
A is computed as follows:
[Equation]
where the policy function π i (´ ω i | ´ G ( t ) i ) is designed by using the sigmoid function to turn each entry [´ ω ( t +1)
i ] j,k in ´ ω ( t +1) i (defined in Eq. into a probability value:
[Equation]
and the regularization term J reg = ´ ω (
[Equation]
+1) i prevents GA2C not to remove massive edges from the graph.
The pseudocode for the complete training process of GA2C is shown in Algorithm 1.
For the computational complexity analysis of GA2C, please see Appendix A.
[Table]
[Table] [Table]
[Table] [Table]
Update
[Equation] [Equation]
E XPERIMENTS
In the experiments, we first describe the used datasets and baselines (for the detailed experimental setups, please see Appendix B).
Next, we present the experimental results on three downstream tasks.
Then we conduct an ablation study and sensitivity analysis on GA2C. Finally, we analyze the cumulative reward in GA2C and visualize the embedding results learned by GA2C.
4. D ATASETS AND B ASELINES
Datasets.
We adopt the same datasets and data splitting in AD-GCL Suresh et al.
These datasets are for graph classification and from TU Benchmark Morris et al.
and Open Graph
Benchmark (OGB) Hu et al.
They cover five types of graphs including biochemical molecules (NCI1, PROTEINS, MUTAG, and DD), social networks (COLLAB, RDT-B, RDT-M, IMDB-B, and IMDB-M), physical chemistry (ESOL, Lipophilicity, and FreeSolv), biophysics (BACE), and physiology (BBBP, ClinTox, Tox21, and SIDER).
The statistics of these datasets are shown in Table 2.
For the datasets containing more than one task (e.g., ClinTox has two binary classification tasks for 1,477 drug compounds with known chemical structures), we report the mean metric values over all tasks.
For the datasets of node classification and link prediction, please see Appendix C.
Baselines.
For graph classification, we compare GA2C with eight baselines totally, including one supervised learning model (F-GIN Xu et al.
(2019b)), three GCL models with frozen GDA (RUGIN Xu et al.
(2019b),
InfoGraph Sun et al.
(2020), and GraphCL You et al.
(2020)), three GCL models with learnable GDA (AD-GCL Suresh et al.
LP-InfoMin You et al.
(2022), and
LP-InfoBN You et al.
and one GCL model with no GDA (SimGRACE Xia et al.
For node classification, we compare GA2C with five baselines including four GZL models with frozen
GDA (DGI Velickovic et al.
(2019), MVGRL Hassani & Khasahmadi (2020),
GraphCL You et al.
(2020), and BGRL Thakoor et al.
and one GZL model with precomputed GDA (GCA Zhu et al.
For link prediction, we compare GA2C with four GCL models (DGI Velickovic et al.
(2019), MVGRL Hassani & Khasahmadi (2020), GMI Peng et al.
(2020), and SAIL Yu et al.
(2022b))
and one unsupervised model (CAN Meng et al.
(2019)) which are from Yu et al.
(2022b).
4. P ERFORMANCE ON DOWNSTREAM TASKS
Graph classification.
We run experiments on five types of graphs and the corresponding experimental results are shown in Table 3.
From Table 3, we can see that the proposed model GA2C has better overall performance than other baselines on biochemistry and social networks.
On some datasets (e.g., PROTEINS, DD, and RDT-M), GA2C has even outperformed the supervised F-GIN model.
Particularly, on IMDB-M, GA2C relatively outperforms F-GIN by about 4 . 1% .
Based on this observation, we guess that the reinforced GDA plays an important role in GCL.
From the RMSE (root mean square error) or AUC results on physical chemistry, biophysics, and physiology, we can see that GA2C still has good performance.
For example, on the datasets of BACE and BBBP, GA2C relatively outperforms the runner-ups AD-GCL and LP-InfoBN by 4.03% and 6.57%, respectively.
Moreover, on ESOL, FreeSolv, and Tox21, seven baselines fall behind the supervised F-GIN but
GA2C outperforms it.
This indicates the advantage of reinforced GDA in GCL again.
The left part is the biochemical and social networks from TU
Benchmark Morris et al.
The right part is the physical chemistry, biophysics, and physiology from Open Graph Benchmark (OGB)
Hu et al.
Here “#Nodes” and “#Edges” are the average node number and average edge number in each graph, respectively.
[Table] [Table]
48.79 1.245 0.974 7.698 LP-InfoBN 76.23 70.45 76.38 72.10 60.95 69.
48.43 1.220 0.955 7.304 SimGRACE 76.05 70.24 76.70 71.39 60.61 68.
49.56 1.387 0.920 5.485 GA2C(ours) 79.45 75.08 81.62 75.41 63.84 72.
[Figure]
Node classification.
The results of the node classification task are shown in Figure 5(a).
If the out-ofmemory error happens in some model (e.g., GCA on ogbn-arxiv), we do not plot its corresponding bar.
From Figure 5(a), we can see that GA2C and GCA (precomputed method) perform better than other baselines on Cora and CS.
We infer that precomputed GDA and learnable GDA can generate more suitable views for the input graph than the trial-and-error method.
In addition, GA2C outperforms other baselines in the three datasets, which indicates the graph reinforced augmentation framework works well on the node classification task.
Link prediction.
The results of the node classification task are shown in Figure 5(b).
From it, we can see that GA2C is competitive with the SOTA GCL models.
Also, SAIL adopts self-augmented views to advance the expressivity of GNNs and achieves the runner-up in the comparison.
Therefore, we conclude that the procedure of GDA is crucial for either GCL or GNNs.
4. A BLATION S TUDY
To evaluate which reinforcement learning model is more suitable for the graph reinforced augmentation framework, we conduct an ablation study on GA2C by replacing the advantage Actor-Critic model with other two reinforce models: vanilla policy gradient (namely GPG) and standard ActorCritic model (namely GAC).
The experimental results are shown in Table 4.
From it, we can see the obvious advantage of GA2C over GPG and GAC.
We explain that GA2C using advantage Actor-Critic has more stable performance and lower variance, which improves GPG and GAC.
In addition, we evaluate the effects of different GDA strategies on the model performance on downstream tasks.
The used GDA strategies include edge removing (“ER”), edge perturbation (“EP”), and attribute masking (“AM”).
The corresponding results are shown in the last three lines in Table 4.
We find that AM fails behind the EP and ER.
AM is not suitable for GA2C because the evolution of topological structure, rather than semantic information, reflects the significant changes in graph characteristics.
4. S ENSITIVITY A NALYSIS
In this part, we conduct the sensitivity analysis to GA2C. The hyperparameters to be analyzed include the number of encoder network layers, learning rate α E (note that we set α E = α A = α C ), dropout ratio, and regularization coefficient λ .
Based on the testing results on three representative datasets (i.e., MUTAG, IMDB-B, and IMDB-M) under different hyperparameter values, we plot the Accuracy curve in Figure 6.
From it, we can see that GA2C is robust to these hyperparameters apart from the number of encoder network layers.
Based on the results in Figure 6(a), we find that the optimal number of encoder network layers is 3 from the candidate set of { 1 , 3 , 5 , 7 , 9 } .
When the number
[Table]
[Figure]
set too small, GA2C has because shallow layers are not enough to encode the abundant structural and semantic information in the graph.
4. A NALYSIS OF C UMULATIVE R EWARD
[Figure]
In this part, we evaluate the cumulative reward of GPG, GAC and
The learning curves in terms of cumulative reward on two representative datasets (i.e., ESOL and FreeSolv) are shown in Figure 7.
From Figure 7, we can see that the performance of GPG (in blue) is unstable and worse than the other two models due to the existence of exploding gradients.
Specifically, the exploding gradients happen at the 156th epoch and 213-rd epoch on
ESOL and FreeSolv, respectively.
By replacing the sampled reward values with the expected reward values learned by the Critic, both GAC (in red) and GA2C (in green) keep stable performance in the training.
4. V ISUALIZATION
To evaluate the quality of node embeddings learned by baselines and GA2C qualitatively, we plot their t-SNE 2D projection results on Cora in Figure 8.
We select DGI, GCA, and MVGRL as representatives because they perform well on Cora (see Figure 5(a)).
From Figure 8(d), we can see that the boundary of clusters is clear, which indicates the embedding learned by GA2C is reasonable.
On the contrary, we notice that the boundary of clusters in Figure 8(a) is blurred, which reveals that the embeddings learned by DGI are not well matched to their own clusters.
[Figure]
R ELATED WORK 5. G RAPH C ONTRASTIVE L EARNING
GCL has three procedures: graph data augmentation (GDA), network encoding, and contrastive loss based optimization.
According to the difference in GDA, we categorize GCL methods as follows:
GCL Methods with frozen graph data augmentation.
Traditional GCL methods You et al.
Sun et al.
(2020) adopt frozen augmentation parameters to generate different graph views.
Generally, the augmentation parameters are set empirically (i.e., trial-and-error).
Different from the trial-anderror method, GCA Zhu et al.
(2021) adopts precomputed method: it calculates GDA parameters based on some indicators of centrality in the graph (e.g., degree centrality and eigenvector centrality) and uses these precomputed parameters to augment the graph.
GCL Methods with learnable graph data augmentation.
For the GCL with frozen augmentation, there is a risk that the redundant information is captured by the InfoMax principle Tschannen et al.
The redundant information hinders the better performance of GZL on the downstream task.
To avoid this, learnable GDA is introduced into GCL, i.e., the parameters of GDA can be learned automatically in the training.
For example, AD-GCL Suresh et al.
(2021) adopts adversarial training, i.e., the contrastive optimization aims to i) maximize the correspondence between the embeddings of different views when fixing GDA and updating network encoding; ii) minimize the correspondence between the embeddings of different views when fixing network encoding and updating GDA.
Also, inspired by image manifolds, You et al.
(2022) extend the frozen and discrete GDA parameters to the learnable ones and leverage both principles of information minimization (InfoMin) and information bottleneck (InfoBN) to regularize the learned GDA parameters.
GCL Methods with no graph data augmentation.
Recent work removes GDA from GCL and adopts the learning framework without GDA to extract representations from graphs.
For example,
SimGRACE Xia et al.
(2022) takes the raw graph as the input and uses the Graph Neural Networks (GNNs) encoder as well as its perturbed version to generate two correlated views for contrast.
Instead of GDA, AF-GCL Wang et al.
(2022) constructs positive and negative pairs based on the representations obtained by GNNs.
SimGCL Yu et al.
(2022a) creates contrastive views by adding random noises to the representations obtained by GNNs.
Compared to prior work, our work differs in that we introduce reinforced GDA into GCL, making the view learner explore an optimal learning path actively.
5. G RAPH R EINFORCEMENT L EARNING
To address the problem that existing GNNs ignore the semantics of subgraphs, SUGAR Sun et al.
(2021) learns the representations of the significant subgraphs which are adaptively selected by Qlearning.
Policy-GNN Lai et al.
(2020) adopts the DQN algorithm to determine the aggregation range of nodes in a large-scale graph.
To adaptively learn the optimal curvature in the graph, ACE-HGNN Fu et al.
leverages multi-agent reinforcement learning where the agents are updated by the Nash
Q-leaning algorithm.
Removing noise data from graphs is helpful to improve the performance of graph representation learning.
GDPNet Xue et al.
and GAM Lee et al.
(2018) adopt Markov
Decision Process (MDP) and Partially Observable Markov Decision Process (POMDP) to model the noisy neighborhoods removing process and the information collection process, respectively.
For heterogeneous graphs, RIOGNN Peng et al.
(2022) leverages the reinforcement learning algorithm to select the most similar neighbors within a relation; RL-HGNN Zhong et al.
(2020) models the process of designing meta-paths as MDP to replace the manual designing process.
Different from these graph reinforcement learning methods, our work uses the Advantage Actor-Critic model to adjust the current optimal GDA parameters adaptively.
C ONCLUSION
In this paper, we design a novel graph reinforced augmentation framework which can ensure that the augmented views evolve well to promote graph contrastive learning.
Under this framework, the graph advantage Actor-Critic (GA2C) model is proposed to learn graph embeddings in an unsupervised manner.
Through extensive experiments, we verify that GA2C outperforms the SOTA graph contrastive learning models on downstream tasks such as graph classification, node classification, and link prediction.
In the future, we plan to study graph contrastive learning with learnable graph data augmentation on heterogeneous graphs and text-rich graphs.
A C OMPUTATIONAL C OMPLEXITY ANALYSIS
Let L E be the encoder network layer number, L A neural network layer number in Actor (or Critic),
N avg.
average node number in each augmented graph, and M avg.
average attribute number in each augmented graph, for each training epoch, the total computational complexity of GA2C is
O( n ( TL A + L E ) N avg.
M 2 avg. ) ) where n and T are the graph number and duration time of Actor or
Critic.
B E XPERIMENTAL SETUPS AND METRICS
Experimental setups.
We use Adam optimizer to train the encoder network, projection network,
Actor model, and Critic model in GA2C.
The learning rates α E , α A , and α C are set as 0.01 for
COLLAB, RDT-M and 0.001 for other datasets.
The layer number of the encoder network is set as 3.
For biochemistry and social networks, the embedding dimension is set as 32 and the batch size is set as 128; for physical chemistry, biophysics, and physiology, the embedding dimension is set asand the batch size is set as 64.
The drop ratio and regularization coefficient are set as 0.0 and 5.0, respectively.
For the downstream tasks, we adopt Linear Support Vector Classification (LinearSVC) as the downstream classifier.
We follow the data splitting method in AD-GCL Suresh et al.
(2021): for OGB
Benchmark, we adopt the proceed data including the training set, validation set, and test set; for TU
Benchmark, we adopt 10-fold cross-validation where the total data are split as 70% training set, 20% validation set, and 10% test set.
Metrics.
For the regression task (i.e., ESOL, Lipophilicity, and FreeSolv), we adopt the metric of
RMSE (root mean square error).
For the classification task, we evaluate the dataset of biochemical and social networks under the metric of Accuracy and the other datasets under the metric of AUC (area under the receiver operating characteristic curve).
C D ATASETS FOR NODE CLASSIFICATION AND LINK PREDICTION
Datasets for node classification.
We use three datasets for the node classification task, including
Cora, CS, and ogbn-arxiv.
Cora is a citation network where the node and edge represent the paper and citation relationship between papers, respectively.
The node attributes represent the paper topics.
For Cora, the task is to classify each paper into a domain class.
CS is a collaboration network where the node and edge represent the author and co-author relationship between authors, respectively.
The node attributes represent the keyword information in the papers published by the author.
For CS, the task is to classify each author into a research field.
ogbn-arxiv is also a citation network where the papers are all from the open-access repository of arXiv.
The detailed statistics of the above three datasets are shown in Table 5.
[Table]
Datasets for link prediction.
We use three datasets for the link prediction task, including Citeseer,
Computers, and Photo.
The dataset of CiteSeer is a citation network, which consists ofscientific publications classified into one of 6 classes.
CiteSeer consists of 4732 links.
Each publication in CiteSeer is described by a 0/1-valued word vector indicating the absence/presence of the corresponding word from the dictionary.
The dictionary consists of 3703 unique words.
The datasets of Computers and Photo are both co-purchase networks extracted from Amazon, where nodes represent products, edges represent the co-purchased relations of products, and features are bag-of-words vectors extracted from product reviews.
D E XPERIMENTAL RESULTS ON RUNNING TIME
Here we compare the running time of GA2C and baselines when the space complexity of models is the same.
On some representative datasets (MUTAG, PROTEINS, and DD), the results are shown in
Table 6.
We find that GA2C is more efficient than AD-GCL and performs better than all the baselines.
[Table]
In adiition
[Table]
we plot the wall-clock time of the training dynamics of different methods in Figureto observe the relationship between running time and model performance.
In this figure, the x-axis represents training time and y-axis represents validation performance.
We find that GA2C performs better than AD-GCL in most cases.
Also, the performance of AD-GCL is unstable and it has high variance, while GA2C is more stable than AD-GCL.