A
C LASS -A WARE
R EPRESENTATION R EFINEMENT
F RAMEWORK FOR G
RAPH C LASSIFICATION
Anonymous authors
Paper under double-blind review
A BSTRACT
Graph Neural Networks (GNNs) are widely used for graph representation learning.
Despite its prevalence, GNN suffers from two drawbacks in the graph classification task, the neglect of graph-level relationships, and the generalization issue.
Each graph is treated separately in GNN message passing/graph pooling, and existing methods to address overfitting operate on each individual graph.
This makes the graph representations learnt less effective in the downstream classification.
In this paper, we propose a Class-Aware Representation rEfinement (CARE) framework for the task of graph classification.
CARE computes simple yet powerful class representations and injects them to steer the learning of graph representations towards better class separability.
CARE is a plug-and-play framework that is highly flexible and able to incorporate arbitrary GNN backbones without significantly increasing the computational cost.
We also theoretically prove that
CARE has a better generalization upper bound than its GNN backbone through
Vapnik-Chervonenkis (VC) dimension analysis.
Our extensive experiments with 10 well-known GNN backbones on 9 benchmark datasets validate the superiority and effectiveness of CARE over its GNN counterparts.
I NTRODUCTION
In recent years, Graph Neural Networks (GNNs) have attracted a surge of attention and been extensively used to learn graph representations for downstream tasks such as graph classification (Xu et al., 2018;
Ying et al., 2018), link prediction (Zhang & Chen, 2018), graph clustering (Kipf &
Welling, 2016b; Zhang et al., 2019a), etc.
They have been applied to graph structured data from a wide range of application domains, including social networks (Kipf & Welling, 2016a; Hamilton et al., 2017; Fan et al., 2019), molecular data (Dai et al., 2016; Gilmer et al., 2017), recommendation system (Wu et al., 2020a; Xia et al., 2021), brain networks (Kim & Ye, 2020), and many more.
The GNN-based graph representation learning can be divided into two categories: (1) some of the methods (Duvenaud et al., 2015; Lee et al., 2019) use a down-sampling strategy (Wu et al., 2020b) to aggregate and transform node representations to graph representations; (2) while some other methods (Ying et al., 2018; Zhang et al., 2019b; Nouranizadeh et al., 2021) obtain the graph embedding by exploring the hierarchical structure of each input graph.
These approaches suffer from two major drawbacks when applied to the downstream classification task: (1) Neglect of graph-level relationships; and (2) Generalization issue.
Neglect of graph-level relationships.
Existing GNN architectures consider each input graph independently in their training processes.
Input graphs are passed individually to GNN to yield node representations.
In addition, the model also treats each graph separately in its design of loss.
The relationships (similarity and/or discrepancy) among different input graphs are fully neglected.
Though the model parameters are trained by the set of input graphs collectively, it is done through a long pathway from node representations to graph representations and finally to the loss.
As a result, the effectiveness of the model will be significantly compromised when applied to the downstream classification.
With molecular data, for instance, one would want the molecules from the same class to share similar representations.
This is natural as molecules belonging to the same class often carry certain common substructures (e.g., the same set of functional groups).
These substructures could be class-specific and serve as a perfect signature of a class.
Without considering such graph-level information, the graph representations learnt would be less effective in separating different graph classes.
Generalization issue.
This is an inherent issue in GNN that the model tends to overfit when the network gets deeper or the hidden dimensionality gets larger (Song et al., 2021).
Some methods have been proposed to alleviate this issue.
Several graph-argumentation based methods improve the generalization ability by modifying input graphs (Papp et al., 2021), or generating new graphs for adversarial learning (Ding et al., 2018) and contrastive learning (You et al., 2020).
Some other works (Byrd & Lipton, 2019; Lin et al., 2017) resample or reweight data instances to remit the overfitting problem.
However, these methods operate on each individual graph and fail to explore the effectiveness of graph-level information in improving generalization.
The discussion of related works is provided in Appendix A.
In this paper, we develop a Class-Aware Representation rEfinement (CARE) framework to address the two above-mentioned limitations of GNNs on the task of graph classification.
Built upon the two existing steps of GNN models for node representation learning and graph pooling, we introduce a new block for extracting class-specific information, namely the Class-Aware Refiner.
The idea of this refiner is simple but effective.
Under the supervision of ground-truth labels, the refiner learns the class representation from a bag of subgraph representations (generated by graph pooling) from each class, and uses it to refine the representation of each graph within the same class.
In this way, we inject the class-specific information to graph representations, with the hope that it can steer the graph representation learning to reflect class signatures.
Inspired by the separability in clustering (Wen et al., 2016), we also propose a class loss that takes into account intra-class graph similarity and inter-class graph discrepancy.
This class loss is combined with the classification loss to directly influence the training of class representations to gain better class separability.
We further design two different architectures of CARE so that our framework can incorporate arbitrary
GNN backbones.
In terms of the model generalization, we analyze the Vapnik-Chervonenkis (VC) dimension (Vapnik & Chervonenkis, 2015) of CARE and provide theoretic guarantee that the upper bound of the VC dimension of CARE is lower than that of the GNN backbone.
The better ability of
CARE in alleviating overfitting is also evidenced in our experiments.
We empirically validate the graph classification performance of CARE on 9 benchmark datasets with 10 commonly used GNN backbones.
The results demonstrate that CARE significantly outperforms its GNN backbones: it achieves up to 11% improvement in classification accuracy.
We also perform a series of ablation studies to assess the effect of each component.
We use a case study to showcase that the graph representations refined by CARE is able to achieve better class separability.
Though CARE introduces an additional refiner to the backbone model, our results show that the consideration of graph-level information drives the model to converge with fewer training epochs.
Consequently, the training time of CARE is comparable to or even shorter than its GNN counterparts.
The main contributions of this paper are summarized as follows:
• We propose a novel graph representation refinement framework CARE, which considers class-aware graph-level relationships.
CARE is a flexible plug-and-play framework that can incorporate arbitrary GNNs without significantly increasing the computational cost.
• We provide theoretic support through VC dimension analysis that CARE has better generalization upper bound in comparison with its GNN backbone.
• We perform extensive experiments using 10 GNN backbones on 9 benchmark datasets to justify the superiority of CARE on graph classification task in terms of both effectiveness and efficiency.
O UR P ROPOSED M ETHOD
In this section, we first give the problem formulation of graph classification, followed by the introduction of our proposed CARE framework.
We also discuss two architectures for applying CARE to existing GNNs.
Finally, we provide theoretic support for the generalization of CARE.
2. P ROBLEM F ORMULATION
We represent a graph as G =
( A , X ) , where A ∈ { 0 , 1 } n × n is its adjacency matrix, and X ∈ R n × c denotes the feature matrix with each node characterized by a feature vector of c dimensions.
The node set of G is denoted by V G and |V G | = n .
We use X v to denote the feature vector of a node v ∈ V G .
Table 6 in Appendix summarizes the notations used throughout the paper.
Given a data set of labeled graphs D =
( G , Y )
= { ( G, y G ) } , where y G ∈ Y is the corresponding label of graph G ∈ G , the problem of graph classification aims to learn a predictive function f :
G → Y that maps graphs to their labels.
2. C LASS -A WARE R EPRESENTATION R EFINEMENT F RAMEWORK
[Figure]
We now describe our proposed Class-Aware Representation rEfinement framework (CARE), which aims to refine graph representations by considering the graph-level similarity.
CARE contains four main components, including a graph encoder, a subgraph selector, a class-aware refiner and a class loss.
The former two allow the flexible incorporation of existing GNN methods, while the latter two are newly proposed in our framework.
Figure 1 depicts the CARE framework.
The remainder of this section describes the four components in detail.
We first introduce the graph encoder to get the initial node/graph representation, and then describe the subgraph selector to extract an appropriate substructure for the subsequent class representation learning.
The class-aware refiner learns class representations from different graph classes, which are used to refine graph representations.
A class loss is proposed to further improve class separability.
The two new components in CARE only contain a small number of parameters and are easy to plugin arbitrary GNN backbone.
Graph Encoder A graph encoder extracts the node representations H and the graph-level representation hg G for graph G .
CARE does not impose any constraint on the architecture of the graph encoder.
Any message-passing GNN model could be applied here, which is formalized as H ( l +1) v = UP [Equation] l ) [Equation]
( l
[Equation]
, AGG
[Equation] l [Equation] , where H ( l +1) ∈ R n × m denotes the ( l + 1) -th layer node representation with m dimensions, UP and AGG are arbitrary differentiable update and aggregate functions, N ( v ) represents the neighbor node set of node v ∈ V G , and H (0) v is initialized as the input feature vector X v .
After a few message-passing layers, we can obtain a set of node representations.
A READOUT function can be applied to produce the graph representation hg G
[Equation]
READOUT ( { H v | v ∈ V G } ) .
Subgraph Selector The class-aware refiner in CARE aims to maintain generic features for graph samples from different classes.
However, the READOUT function treats all nodes equally without considering the class information.
In fact, graphs in different classes are likely to have various substructures.
To address this limitation, CARE introduces a subgraph selector sub ( · ) to filter nodes in the original graph, which is defined as A ( l +1) , H ( l +1)
=
sub ( A ( l ) , H ( l ) ) .
Any graph pooling methods could be applied here to select subgraphs.
Typical ones include node drop pooling methods (Lee et al., 2019) and node clustering pooling methods (Baek et al., 2021).
Class-Aware Refiner
As existing GNN models ignore the relationships of graphs from different classes, a new component is designed in CARE to fill this gap.
In the training process, the classaware refiner utilizes the ground truth label of each training instance.
It maintains a bag of encoded (sub)graph representations B i and aggregates these representations to obtain a class representation hc i for each class i ∈ Y .
The aggregation function is a universal Set Encoder, e.g., DeepSets (Zaheer et al., 2017) or PointNet (Qi et al., 2016).
Herein, we apply DeepSets as in Eq.
(1), in which ρ ( · ) is a multilayer perceptron (MLP) with a non-linear function ReLU, and ϕ ( hg ) =
hg/ |B i | .
[Equation]
where hg subG is the subgraph representation of a graph G ∈ G , obtained by passing the output of the subgraph selector through the READOUT function.
The class representation hc i is then used to refine graph representations for all graphs in the same class:
[Equation]
where Trans ( · ) is a transformation function and hg ′ G is the refined graph representation.
Herein, we set Trans ( · ) =
ρ ( · ) .
In the validation and the test processes, the ground truth label y G is not available for a validation/test graph G .
However, we need a label for each validation/test graph to determine which class representation to be used to composite its refined representation.
In this case, the Class-Aware Refiner will predict a pseudo label (cid:101)
y G for graph G by classifying it to the most similar class and use the corresponding class representation for graph representation refinement.
We use the cosine similarity as a metric to quantify the similarity between a graph representation and a class representation.
The pseudo label is obtained as (cid:101) y G = arg max i ∈Y ( cos sim ( hg subG , hc i )) .
Note that class representations are kept unchanged in the validation/test process.
The training algorithm of the class-aware refiner is summarized in Algorithm 1.
Class Loss For graph classification task, it would be beneficial to exploit the graph similarity within the same class and the graph discrepancy between different classes.
This is essential for making different classes more separable.
Using the classification loss only fails to learn such graph-level relations.
Therefore, a class loss L class ( · ) is proposed in CARE to enforce the intra-class similarity and the inter-class discrepancy.
The former L intra is defined as the similarity between each graph representation and its class representation, while the latter L inter is defined as the similarity between different class representations:
[Equation]
We again use the cosine similarity as a metric.
The class loss L class = exp( L inter − L intra ) is then defined as a function that maximizes L intra and minimizes L inter , where AV G ( · ) is the average function.
The predicted class label is still supervised by a classification loss L cls ( · ) .
Herein, we apply the commonly used cross-entropy loss (Cox, 1958).
The overall loss L of CARE is defined as:
[Equation]
where λ is a trade-off hyperparameter for balancing the classification loss and the class loss.
2. M ODEL A RCHITECTURE V ARIANTS
As GNNs can be categorized as hierarchical ones and non-hierarchical ones, we design two corresponding architectures that apply CARE to plug-and-play in different GNN backbones.
Global Architecture Several GNN models (e.g., GCN (Kipf & Welling, 2016a), GAT (Veli ˇ ckovi ´ c et al., 2017) and GraphSAGE (Hamilton et al., 2017)) apply the readout function only at the end of graph convolution.
The global architecture of CARE is designed to apply the Class-Aware Refiner only after the readout function.
The outputs are then passed to a linear layer for graph classification.
Hierarchical Architecture
Some other GNN models, such as GIN (Xu et al., 2018) and UGformer (Nguyen et al., 2019), have a readout function in each graph convolutional layer.
The graph representations from each layer are taken into account when making the final prediction.
The hierarchical architecture of CARE is designed to apply the class-aware refiner on each layer in order to cope with the hierarchical GNN backbones.
2. G ENERALIZATION A NALYSIS
In this section, we present the theoretical support that the proposed CARE has a better model generalization than its GNN backbone in the case of binary classification.
We use the VC dimension to measure the capacity of a model.
Based on the VC theory (Vapnik, 2000), reducing the VC dimension of a model has the effect of eliminating potential generalization errors.
Our analysis is grounded on the VC theory for neural nets (Bartlett & Maass, 2003): the VC dimension of a neural network is upper bounded by a function with respect to the number of model parameters t and the number of operations p .
In the following, we first derive the computational complexity of the GNN backbone and CARE measured by the number of multiplications, based on which we obtain an upper bound of the VC dimension for each model.
We then present a theorem that states that CARE has a lower VC dimension upper bound than its GNN backbone when the number of parameters is identical.
In subsequent discussions, we use GCN as an example backbone.
The conclusion generally applies to other backbones by plugging in their corresponding computational complexity.
We present the theoretical results here and defer the detailed proofs to Appendix.
Computational Complexity of Models.
CARE and its backbone GCN are both composed of GCN layers, an embedding layer, and several fully-connected layers.
[ Complexity of GCN Backbone .]
We denote the GCN layer in the GCN model as gcn ( · ) and its input/output dimensions as h gcn in / h gcn out .
The layer mapping is given by gcn ( A, H )
=
σ gcn ( ˆ AHW gcn ) , where σ gcn is the activation function, W gcn ∈ R h gcnin × h gcnout is the weight matrix, and ˆ A is the normalized adjacency matrix.
The computational complexity of the GCN network measured by the multiplication number, denoted as q 1 ( d ) , for d number of layers, is given by: [Equation]
[Equation]
[ Complexity of CARE .
The GCN-based CARE network with a hierarchical architecture is composed of the following:
• a GCN layer same as the GCN backbone, whose computational complexity is q lgcn =
n 2 h lgcn in + nh lgcn in h lgcn out .
• a subgraph selector (SAGPool), which contains a score layer (GCN with 1 dimensional output) and a top-k pooling algorithm.
The complexity is q lsubgraph = n 2 h lgcn out + nh lgcn out .
• a class-aware refiner contains a set encoder Eq.
(1) and a transformation layer Eq.
The mapping of the fully-connection layer fc ( · ) from input H is given by fc ( H )
=
σ fc ( HW fc ) , where σ fc is the activation function, W fc ∈ R
h fcin × h fcout , and h fc in /h fc out are the input/output dimensions.
The complexities for the set encoder and the transformation layer are q lset = nh lset in h lset out and q ltrans =
nh ltrans in h ltrans out , respectively.
Therefore, the computational complexity of the GCN-based CARE is given by:
[Equation]
VC Dimension Upper Bound.
Inspired by the theoretical analysis in Kabkab et al.
(2016) that derives an upper bound of the VC dimension for a CNN model, we extend its result to a GCN model, as given by the following lemma.
Lemma 1 Let C d be the set of GCN models with d convolutional layers.
Let H d ≜ { h c : I → { 0 , 1 }| c ∈ C d } be the set of boolean functions implementable by all GCNs in C d .
The VC dimension of GCNs, as well as CARE, satisfies V C dim ( H d ) ⩽
α ( d · q ( d )) 2 for some constant α .
Here, q ( d ) is the computational complexity of the model under consideration.
VC Dimension Comparison We now compare the upper bounds of VC dimension on the GCN backbone and CARE, which is formalized by the following theorem.
Theorem 1.
Assume that the number of parameters in a GCN backbone and CARE is identical.
Let upperVC(GCN) and upperVC(CARE) be the upper bounds of VC dimension on the two models, respectively, which are given by Lemma 1.
We have upperVC ( GCN ) > upperVC ( CARE ) .
Based on Theorem 1 and the VC theory, we conclude that our CARE has a better generalization potential than its GCN backbone.
E XPERIMENTS
Due to the space limitation, we provide the experiment setup in Appendix C. In this section, we first assess the performance of CARE in comparison with state-of-the-art GNN backbones.
We then conduct ablation studies to analyze the effects of different components in CARE.
A case study is presented to further investigate how CARE affects the separability of graphs from different classes.
Finally, an evaluation of model efficiency is performed.
The sensitivity test of hyperparameters in
CARE is discussed in Appendix D.
3. P ERFORMANCE C OMPARISON WITH GNN B ACKBONES
Effectiveness Analysis.
We assess the graph classification performance on the first 8 datasets and the last dataset using two different metrics.
The former is assessed by the classification accuracy and the latter by the ROC-AUC.
This is because the OGBG-MOLHIV dataset has a severe class imbalance issue.
The results on the first 8 datasets are reported in Table 1.
Each row in the table shows the performance of an original GNN backbone and the performance after applying CARE.
Each column reports the results on a dataset.
In total there are 80 backbone/dataset pairs and the best result in each pair is highlighted in bold.
As shown in Table 1, CARE is a clear winner: it outperforms the GNN backbone in 74 out of 80 cases.
CARE gains over 1% improvement in the absolute accuracy in 47 out of 74 winning cases, while it drops over 1% in accuracy in only 1 out of 6 losing cases.
In particular, the improvement of CARE is up to 11.48%, which is achieved on the FRANKENSTEIN dataset with GraphSAGE as backbone.
The same observation is made when testing on the OGBG-MOLHIV dataset.
As shown in Table 2, CARE outperforms the GNN backbones in most cases with improvements up to 5.63%.
To sum up, the results demonstrate that
CARE is able to serve as a general framework to boost up the graph classification performance over state-of-the-art GNN models on various datasets.
To match with the setting of Theorem 1, we also conduct experiments under the same parameter numbers in
Appendix E.
The experimental comparison with GIB (Yu et al., 2020) is provided in Appendix F.
[Table]
[Figure]
[Table]
Generalization Performance.
We also observe that CARE is able to alleviate the overfitting in GNN backbones.
An example is shown in Figure 2, where we plot the accuracy curves on the PROTEINS dataset with GCN as the backbone.
It shows that the test accuracy of
GCN (in blue) exhibits a steep and continuous downward trend starting from epoch [Table] while its corresponding training accuracy continues to climb up.
This indicates an obvious overfit of GCN to the training data.
After applying
CARE on GCN (in red), the steep drop in the test accuracy vanishes, which demonstrates the ability of CARE in remitting overfitting [Table]
[Table]
3. A BLATION S TUDIES
We perform two ablation studies to show how different designs of the Subgraph Selector and the loss function L influence the performance of CARE.
Subgraph Selector We compare the performance of three CARE variants with different subgraph selectors on 5 datasets with 4 GNN backbones.
The results are presented in Table 3.
The first CARE variant, denoted as ”None”, uses the whole graph for class representation learning without selecting any subgraph.
The other two models apply SAGPool and HGSPLPool respectively as the subgraph selector.
The pooling ratio for both SAGPool and HGSPLPool is set to 0.5.
We can see that using the subgraph selector achieves the best result in 15 out of 20 cases.
When comparing the performance of using SAGPool and HGSPLPool, the former beats the latter in 14 out of 20 cases.
Therefore, we choose SAGPool as the default subgraph selector.
[Table]
in terms of Classification Accuracy.
Winner in each backbone/dataset pair is highlighted in bold.
Class Loss We investigate different designs of the loss function to study the impact of the class loss we proposed.
We consider four designs of the overall loss function L : a) cls : uses the classification loss L cls only; b) intra : uses a combination of the classification loss and the intra-class loss as given by L
= L cls − λ ∗ exp( L intra ) ; c) inter : uses a combination of the classification loss and the inter -class loss
as
given by L = L cls + λ ∗ exp ( L
inter
and d) combine , the overall loss function in Eq.
The results in Table 4 show that the proposed loss function performs the best among all designs.
This demonstrates the effectiveness of the proposed class loss that takes into account the intra-class similarity and inter-class discrepancy.
3. C ASE S TUDY FOR C LASS S EPARABILITY
We design a case study to further investigate the effect of CARE, in particular its class-aware components, in refining graph representations for the classification task.
The idea is to study how CARE affects the separability of graph classes in the training process.
Is it able to direct the graph representation learning to move towards better class separability?
In order to answer this question, we use four class separability metrics as follows.
For all metrics, the larger their values, the better the class separability is.
We refer the reader to Appendix H for their formal definitions.
Silhouette Coefficient (Rousseeuw, 1987)
It measures how similar a sample is to its own class (cohesion) compared to those from other classes (separation).
Its value ranges from -1 to 1.
Separability Index (Thornton, 1998)
It computes the fraction of samples that have a nearest neighbour with the same class label.
Its value ranges from 0 to 1.
Hypothesis Margin (Gilad-Bachrach et al., 2004)
It measures the distance between a sample’s nearest neighbor from the same class (near-hit) and the nearest neighbor from the opposing class (near-miss) and averages over all samples.
Centroid
1 21 41 61 81 101 121 141 161 181 201 221 241 261 281 301 epoch
SilhouetteCoefficient 0.620.640.660.
1 21 41 61 81 101 121 141 161 181 201 221 241 261 281 301 epoch
SeparabilityIndex 0 26 81 21 41 61 81 101 121 141 161 181 201 221 241 261 281 301 epoch
HypothesisMargin 00.511.522. 1 21 41 61 81 101 121 141 161 181 201 221 241 261 281 301 epoch CentroidDistance 0 13 4 1 21 41 61 81 101 121 141 161 181 201 221 241 261 281 301 epoch CentroidDistance 22.533.
1 21 41 61 81 101 121 141 161 181 201 221 241 261 281 301 epoch
HypothesisMargin 0.56 0.
1 21 41 61 81 101 121 141 161 181 201 221 241 261 281 301 epoch
SeparabilityIndex 0 0.05 0.1 0.
1 21 41 61 81 101 121 141 161 181 201 221 241 261 281 301 epoch
SilhouetteCoefficient
[Figure]
Distance It sums up the distances between the centroids for all pairs of classes, where the centroid of a class is the mean of all samples in the class.
Figure 3 reports the results on the PROTEINS dataset with
GCN as the backbone.
We compute the four metrics on the graph representations produced by CARE and GCN on the training data.
CARE uses the refined graph representations, while GCN uses the original ones.
It can be seen that the training curves of CARE exhibit an upward trend under all the four separability metrics.
At the time when models converge, CARE outperforms GCN in all metrics.
In particular,
CARE achieves 49.26% improvement on Silhouette Coefficient, 1.96% on Separability Index, 45.21% on Hypothesis
Margin, and 30.51% on centroid distance.
A visualization of the graph representations in each model is shown in Figure 4.
Graph representations are passed into T-SNE (Van der Maaten & Hinton, 2008) for dimensionality reduction and colored by their class labels.
This demonstrates that CARE is indeed able to steer the graph representation learning towards better class separability, which is also reflected by its superior classification performance over GNN backbones.
Similar conclusions can be drawn from the results on the test data, which indicates that the class separability of CARE generalizes well to the test data.
3. T IME E FFICIENCY
CARE, when applied to a GNN backbone, introduces an additional refiner and the class loss.
A natural question arises: will CARE significantly sacrifice the efficiency of its GNN backbone for better classification performance?
This subsection aims to answer this question.
Table 5 in Appendix reports the number of epochs and the total time needed (including training, validation and test) for CARE and the backbones GraphSAGE and GCN.
It can be seen that CARE takes less number of epochs to converge than its GNN counterpart in all cases.
Consequently, the running time of CARE is shorter than (4 out of 6 cases) or comparable to its backbones.
The results demonstrate that CARE is able to work on top of existing GNN models with superior effectiveness and improved/comparable efficiency, making it a practical choice in real applications.
C ONCLUSIONS
In this paper, we proposed CARE, a novel graph representation refinement framework for GNNbased graph classification.
It fills the gap that existing GNNs fail to consider graph-level relationships in model design, and meanwhile improves the model generalization as proven theoretically and evidenced empirically.
Its plug-in-play nature makes it a powerful framework to build upon arbitrary GNN models and boost up their classification performance without sacrificing efficiency.
A R ELATED W ORKS A. G RAPH N EURAL N ETWORKS
Kipf and Welling (Kipf & Welling, 2016a) were the first to introduce the convolution operation to graphs.
Later on, Hamilton et al.
Hamilton et al.
(2017) proposed to use sampling and aggregation to learn node representations.
The attention mechanism was introduced to graph convolution in Veliˇckovi´c et al.
(2017) to yield node representations by unequally considering messages from different neighbors.
The Graph Isomorphism Network (GIN) was proposed in Xu et al.
[Table] which has been shown to be as powerful as the Weisfeiler-Lehman (WL) test (Weisfeiler & Leman, 1968).
Recently [Table] the transformer architecture (Vaswani et al., 2017) has been introduced to GNN
(Nguyen et al., 2019; Rong et al., 2020), which considers the relatedness between nodes in the node representation learning.
For GNN-based graph classification, graph pooling is commonly applied as the readout function to generate graph representations.
Existing pooling methods can be categorised under node drop pooling (Duvenaud et al., 2015;
Zhang et al., 2018; Lee et al., 2019) and node clustering pooling (Ying et al., 2018; Zhang et al., 2019b; Li et al., 2020; Nouranizadeh et al., 2021).
Node drop pooling uses learnable scoring functions to drop nodes with lower scores while node clustering pooling casts the graph pooling problem into the node clustering problem (Baek et al., 2021).
In a recent development, Yang et al.
(Yang et al., 2021) leverage the mask mechanism to select the subgraph by considering the consistency over samples.
Both categories of the pooling methods focus on node level manipulations and neglect the graph-level information.
A. M ETHODS TO T REAT GNN O VERFITTING
Expressiveness of GNN could be improved by increasing the number of model parameters through e.g., expanding the hidden dimension of the GNN layer or adding more layers.
However, this process could detriment the performance and induce overfitting (Song et al., 2021).
A typical method to treat GNN overfitting is via data augmentation.
Several works (Papp et al., 2021;
Rong et al., 2019) used node/edge dropping augmentations.
Ding et al.
(2018) proposed a generatorclassifier network under the adversarial learning setting to generate fake nodes.
Feng et al.
(2019) performed adversarial perturbations to node features.
Recently, the paradigm of contrastive learning has been introduced to GNN to perform graph contrastive learning (You et al., 2020; Thakoor et al., 2021).
SUGAR (Sun et al., 2021) generates subgraphs and uses these subgraphs for reconstruction, which still follows the graph contrastive learning paradigm.
All these methods perform augmentation on individual graphs and again neglect the graph-level information.
Several resampling (Byrd & Lipton, 2019;
Zhou et al., 2020) and reweighting (Lin et al., 2017; Shi et al., 2021) methods have also been proposed to prevent GNN from overfitting.
In general, these methods design algorithms to control the influence of each sample on the model, while the sample relations are not taken into account.
To the best of our knowledge, our work is the first to explore the use of class-aware graph-level relationships to alleviate the overfitting in GNNs.
B T HEORETICAL P ROOFS B. P ROOF S KETCH OF L EMMA
Our proof follows the same flow as Lemma 1 in Kabkab et al.
A parametrized class of functions with parameters in R t that is computable in no more than p operations has a VC dimension which is O ( t 2 p 2 ) (Bartlett & Maass, 2003).
t in GCN and CARE can be formulated as:
[Equation]
By plugging in the number of multiplications q 1 ( d ) and q 2 ( d ) given by Eqs.
(6) and (7), together with the above equations on the number of parameters t , into O ( t 2 p 2 ) , we complete the proof of
Lemma 1 for both GCN and CARE.
B. P ROOF OF T HEOREM
We compare the VC dimension upper bounds of a GCN layer and a GCN-based CARE layer under the identical number of parameters.
According to Section 2.2 of Abu-Mostafa et al.
(2012), the
VC dimension provides a loose generalization bound for models and can be used as a guideline for generalization comparison - models with a lower upper bound tend to have better generalization capability.
The number of parameters in a GCN layer t 1 and that in a GCN-based CARE layer tare formulated as:
[Equation]
In our setting, we choose a basic hidden dimension h 1 and h 2 for GCN and CARE respectively.
We set each layer to be an integer multiple of the basic hidden dimension.
Thus, t 1 = h 21 and t 2 = h
+ h 2 + h
+ 2 h
= 4 h
+ h 2 , respectively.
Note that h trans in = h set out + h gcn out as we concatenate the class representation with the subgraph representation.
Similarly, the computational complexities q 1 and q 2 can be rewritten as:
[Equation]
[Equation]
When d = 1 , the complexity can be written as:
[Equation]
[Equation] Under the identical number of parameters, we let t 1 = t 2 , and have h 1 = (cid:112) 4 h
+
h 2 .
Thus,
[Equation]
The difference between q 1 (1) and q 2 (1) satisfies:
[Equation]
[Equation] Because (cid:112) 4 h 22 + h 2 − 2 h 2 > 0 , we have: [Equation]
[Equation]
According to our setting, the input and output feature map sizes of all layers is identical, which means that the ‘ n ’ in each layer’s complexity equation are identical.
Thus, we extend Eq.
(18) to the full model and have:
[Equation]
With Eq.
(19) and Lemma 1, we complete the proof of Theorem 1.
C E XPERIMENTAL S ETUP
In this section, we present the experimental setup, including datasets, GNN backbones and detailed model implementation.
C. D ATASETS
[Table]
Nine commonly used benchmark datasets were tested in our experiments.
Eight of them were selected from TUDataset (Kersting et al., 2016) and include DD, PROTEINS, MUTAG, NCI1,
NCI109, FRANKENSTEIN (FRANK), Tox21 and ENZYMES.
The last dataset OGBG-MOLHIV was selected from Open Graph Benchmark (Hu et al., 2020) and consists of 41K+ graphs.
The statistics of the datasets are summarized in Table 7.
C. GNN B ACKBONES
We test the effectiveness of CARE on a wide range of GNN backbones, including GCN (Kipf &
Welling, 2016a), GraphSAGE (Hamilton et al., 2017), GIN (Xu et al., 2018), GAT (Veli ˇ ckovi ´ c et al., 2017), UGformer (Nguyen et al., 2019) , GXN (Li et al., 2020), SAGPool (Lee et al., 2019),
DiffPool (Ying et al., 2018), HGPSLPool (Zhang et al., 2019b) and MEWISPool (Nouranizadeh et al., 2021).
We apply CARE on each of them and compare the performance with the original backbone model.
Among the 10 models selected, GIN and UGformer are hierarchical ones.
We thus apply the hierarchical architecture CARE on them.
The global architecture CARE is applied to the rest models.
A brief introduction of each model is provided as follow:
GCN (Kipf & Welling, 2016a) is a mean pooling baseline with graph convolution network as a message-passing layer.
GAT (Veliˇckovi´c et al., 2017) is a mean pooling baseline, which adopts an attention mechanism to learn the relative weights between the node and its neighbors.
GraphSAGE (Hamilton et al., 2017) is a mean pooling baseline, which adopts sampling to obtain a fixed number of neighbors for each node. GXN (Li et al. 2020) is a sort
pooling
(Zhang
et al., 2018) baseline, which uses vertex infomax pooling to select nodes that can maximally express their
corresponding neighborhoods SAGPool (Lee et al., 2019) is a graph pooling method
that
uses graph convolution in graph pooling to consider both node features and graph structure.
DiffPool (Ying et al., 2018) is an end-to-end trainable graph pooling method that can produce hierarchical representations for graphs.
HGPSLPool (Zhang et al., 2019b) is a pooling method that introduces a structure learning mechanism to refine graph structure after pooling.
MEWISPool (Nouranizadeh et al., 2021) is a pooling method, which introduces Shannon capacity to maximize the mutual information between the input graph and the output graph.
GIN (Xu et al., 2018) is a sum pooling baseline that uses a learnable parameter to adjust the weight of the central node, thus improving the message-passing network’s ability to distinguish different graph structures.
UGformer (Nguyen et al., 2019) is a sum pooling baseline which identifies meta-paths to transform the graph structure for node representation learning and adopts Transformer for aggregation.
C. I MPLEMENTATION D ETAILS
The default number of graph convolutional layers in both CARE and GNN backbones is 4.
We use
SAGPool with a pooling ratio of 0.5 as the default subgraph selector in CARE.
Notice that we did not apply any subgraph selector on GNNs that are already equipped with their own pooling methods for substructure extraction.
This includes SAGPool, DiffPool, HGPSLPool and MEWISPool.
The trade-off hyperparameter λ in Eq.
(5) is set to 1 by default.
The whole network is trained in an end-to-end manner using the Adam optimizer (Kingma & Ba, 2014).
We use the early stopping criterion, i.e., we stop the training once there is no further improvement on the validation loss during 25 epochs.
The learning rate is initialized to 10 − 4 and the maximum number of epochs is set to 1000.
We set the hidden size to 146 and batch size to 20 for all models.
The only exception is
DiffPool when tested on the D&D dataset.
Since the D&D dataset has a large number of nodes (see Table 7), the hidden size and batch size are set to 32 and 6 to achieve an acceptable number of parameters in DiffPool.
For TUdataset, we split it into 8:1:1 for training, validation and test.
For all experiments of
CARE and GNN backbones, we evaluate each model with the same random seed for 10-fold crossvalidation.
We use the scaffold splits for the OGBG-MOLHIV dataset and report the average ROCAUC with 10 random seeds.
All the codes were implemented using PyTorch (Paszke et al., 2017) and Deep Graph Library (Wang et al., 2019) packages.
The experiments were conducted in a Linux server with Intel(R) Core(TM) i9-10940X CPU (3.30GHz), GeForce GTX 3090 GPU, and 125GB
RAM.
D H YPERPARAMETER A NALYSIS
In this section, we study the sensitivity of two important hyperparameters in CARE, the trade-off parameter λ in the loss function and the number of layers.
We test on three datasets using GIN as backbone for this set of experiments.
Trade-off Parameter λ .
This hyperparameter is used in the overall loss function L (Eq.
to trade-off between the classification loss and the class loss.
We tune the value of λ from 0.01 to 10 5 .
The results are presented in Table 8.
It shows that the choice of λ affects the performance marginally and there doesn’t exist a value that works best for all datasets.
In practice, we could use the validation set to find the best value of λ .
[Table]
Number of Layers .
The depth of the neural network can certainly affect the model performance.
We adjust the number of layers to investigate whether CARE can adapt to different depths of neural networks.
We vary the number of layers from 2 to 5, and report the results in Table 9.
For each dataset, we underline the best result among all the numbers of layers tested.
As shown in Table 9, CARE consistently outperforms GIN at different number of layers, except for 4 layers on PROTEINS where the performance difference is marginal at 0.09%. The best results
[Table]
are achieved with 2 layers on D&D and PROTEINS and with 5 layers on MUTAG.
Therefore, the number of layers should also be selected through the validation process for different datasets.
E E FFECTIVENESS A NALYSIS UNDER THE SAME P ARAMETER N UMBER
CARE is proposed as a plug-and-play framework. However, in addition to directly plugging it in a GNN backbone without changing the number of model parameters in the backbone (as what we have done in experiments in the submitted version), it could also be used in a way that the resultant CARE after plug-in has a comparable number of parameters to the original GNN backbone before plug-in. This can be achieved by adjusting the number of parameters in the GNN backbone at the time when CARE is plugged in.
To demonstrate this, we conduct a new experiments to match with the setting of Theorem 1.
For each GNN backbone, we first set its number of parameters to 100K.
For CARE, we adjust the hidden dimension of each GNN backbone to which CARE is applied such that the number of parameters of CARE is also 100K. As shown in the table below, CARE still outperforms its GNN backbone in 8 out of 9 cases.
The results demonstrate that CARE can boost up the graph classification performance without introducing additional parameters.
[Table]
F C OMPARISON WITH GIB
GIB proposes a novel objective to recognize the maximally informative subgraph.
It designs a bilevel optimization scheme and a connectivity loss to optimize the GIB objective.
As both GIB and our work consider graph-level similarity, we add it as a new baseline in our experiments.
The results of GIB are taken from its original paper.
As shown in the table below, as a plugin component, CARE outperforms GIB in 7 out of 12 cases. G A BLATION S TUDY ON S IMILARITY M ETRIC FOR C LASS L OSS We study the effect of the similarity metric in computing the class loss. Besides the cosine similarity,
L2 distance is also commonly used to measure the dissimilarity between two vectors.
We use L distance in place of the cosine similarity in Eqs.
(3) and (4) to define the intra-class and inter-class losses.
As L2 distance measures the dissimilarity, we take an inverse of the class loss when L
[Table]
CARE under these two different metrics.
Table 12 shows the results under GCN as the GNN backbone.
It shows that CARE with the cosine similarity is more powerful than that with the L2 distance.
[Equation]
[Table]
H C LASS S EPARABILITY M ETRICS IN C ASE S TUDY H. S ILHOUETTE C OEFFICIENT
The Silhouette of a sample x i is defined as:
[Equation]
[Equation] (24) where a i denotes the average distance between x i and all other samples in the same class, and b i denotes the smallest mean distance from x i to all samples in any other class.
H. S EPARABILITY I NDEX
The Separability Index SI is defined as:
[Equation]
where m is the total number of samples, x i denotes the i -th sample, y i denotes its corresponding class label, and Y denotes the set of classes.
The nearest neighbour distance function ∥·∥ is assumed to utilise a suitable metric, e.g., a Manhalobis metric for symbolic data or a Euclidean metric for spatial data.
H. H YPOTHESIS M ARGIN
The Hypothesis Margin ( HM ) is defined as:
[Equation]
where nearhit ( x i ) and nearmiss ( x i ) denote the nearest sample to x i with the same and different label, respectively.
∥·∥ denotes the L2 distance.
Note that a chosen set of features affects the margin through the distance measure.
H. C ENTROID D ISTANCE
The Centroid Distance ( CD ) is defined as:
[Equation]
[Equation]
(31) where y x denotes the class label of a sample x , Y denotes the set of classes and ∥ · ∥ denotes the L distance.