D IVERSIFIED M ULTISCALE G RAPH L EARNING WITH G RAPH S ELF -C ORRECTION
Tencent AI } @tencent.com
Yu Rong
Tencent AI Lab { yu.rong } @hotmail.com
A BSTRACT
Though the multiscale graph learning techniques have enabled advanced feature extraction frameworks, we find that the classic ensemble strategy shows inferior performance while encountering the high homogeneity of the learnt representation, which is caused by the nature of existing graph pooling methods.
To cope with this issue, we propose a diversified multiscale graph learning model equipped with two core ingredients: a graph self-correction mechanism to generate informative embedded graphs, and a diversity boosting regularizer to achieve a comprehensive characterization of the input graph.
The proposed mechanism compensates the pooled graph with the lost information during the graph pooling process by feeding back the estimated residual graph, which serves as a plug-in component for popular graph pooling methods.
Meanwhile, pooling methods enhanced with the self-correcting procedure encourage the discrepancy of node embeddings, and thus it contributes to the success of ensemble learning strategy.
The proposed regularizer instead enhances the ensemble diversity at the graph-level embeddings by leveraging the interaction among individual classifiers.
Extensive experiments on popular graph classification benchmarks show that the approaches lead to significant improvements over state-of-the-art graph pooling methods, and the ensemble multiscale graph learning models achieve superior enhancement.
I NTRODUCTION
While many GNNs learn graph representation at a fixed scale, the multiscale graph learning has also attracted a surge of interests, for its capability of capturing both fine and coarse graph structures and features.
Several advanced feature extraction frameworks have been proposed for multiscale graph learning.
Some attempts adopt the encoder-decoder pipeline to embed the input-graph and perform feature updating in the latent coarsest graph (Chen et al., 2017; Liang et al., 2018; Deng et al., 2019).
Other works utilize the pyramid architecture to extract multiscale graph features, and perform feature aggregation via skip-connection (Gao & Ji, 2019; Lee et al., 2019), or cross-layer summation fusion (Li et al., 2020).
Orthogonal to various multiscale feature learning techniques, one straightforward and promising strategy for adequately leveraging the extracted graph embeddings is to construct ensembles of multiple individual classifiers in a stacking style, where each classifier receives a graph embedding at a [Table] fixed granularity, and predicted logits of all classifiers are then averaged to produce the final prediction.
As a generic method, such ensemble models are expected to obtain more reliable predictions.
However, a crucial dependency of high diversity among classifiers, may not hold in the scenario.
As indicated in recent researches (Mesquita et al., 2020; Bianchi et al., 2020), the indispensable graph pooling operation for establishing multiscale graph learning may have led to the high homogeneity of node embeddings.
Mesquita et al.
(Mesquita et al., 2020) observe that graph pooling exacerbate the homogeneity of node embeddings.
Due to the homogeneity of the node embeddings, the generated graph embeddings at various granularity tend to become homogeneous as well.
Under this circumstance, the graph embeddings as inputs lead to limited diversity among learned classifiers.
As a result, the ensemble strategy will fail in boosting the performance of GNNs equipped with these pooling modules, which have been verified by the experimental results in Table 1.
To address the issues discussed above, we design a diversified multiscale graph learning model which improve the graph pooling quality from two aspects: 1) Less information loss: we propose a graph self-correction mechanism to generate informative pooled graphs, which also hampers the homogeneity of node embeddings and thus promotes the ensemble diversity at the node-level embeddings , and 2) More diversity: we introduce a diversity boosting regularizer to directly model and optimize the ensemble diversity at the graph-level embeddings .
The ensembled multiscale graph learning framework and the schemata of our approaches is depicted in Figure 1.
G RAPH S ELF -C ORRECTION M ECHANISM
A core idea of Graph Self-Correction (GSC) mechanism is to reduce such information loss through compensating information generated by the feedback procedures.
It contains three phases: graph pooling, compensated information calculation and information feedback.
2. P HASE 1: G RAPH POOLING
In the settings of GSC, the initial graph pooling layer concentrates on searching for the optimal structure of the coarsened graphs rather than directly learning both the topological and informative embeddings.
As a preliminary step of the approach, it can be implemented by any kind of existing pooling methods.
As a explanation case, SAGPool (Lee et al., 2019) adopts a 1 -layer GCN (Kipf & Welling, 2016) as the node scorer, and uses the Top- K selection strategy to select K nodes with index of Idx ( t ) at the t -th pooling layer, constructing the pool coarsened graph.
Assumed the input graph has the vertex set V with N nodes.
Its node features and adjacency matrix is X ( t ) and A ( t ) .
The ones in the pooled graph is denoted as X ( t +1) and A ( t +1) .
2. P HASE 2: C OMPENSATED INFORMATION CALCULATION
As noticed in the pooling process given by SAGPool, the subsequent pooled node features discard all information preserved in those unselected nodes, which might hurt the exploitation of rich semantic features in original graphs.
Inspired by the feedback networks (Carreira et al., 2016; Haris et al., 2018; Li et al., 2016), GSC introduces a residual estimation procedure to calculate compensated information that empowers the self-correction of the embedded coarsened graph in the feedback stage.
Specifically, we propose two parallel schemes to calculate such residual signals: 1) complement graph fusion, and 2) coarsened graph back-projection.
1) Complement graph fusion: The graph composited by the unselected nodes after the graph pooling layer and their adjacency relations is defined as the complement of the pooled graph here.
Then
the
Readout
Multiscale Graph Embedding Extraction
Input
GraphConv
Self-Correcting Pooling
GraphConv
Self-Correcting Pooling
GraphConv
Self-CorrectingPooling 𝓛
𝐃𝐁𝐑 𝓛 𝐃𝐁𝐑 : Eq.(18) Diversity
Boosting Regularizer
G
G 3 G
G
G G
Maximize the orthogonality share logitslogitslogits
Avg
EnsembledLogits
MLP
Stacking Predictors for Ensembling
MLP share
MLP 𝓛 𝐂𝐄
III: Feedback Fusion
Graph Self-Correction Mechanism
I: Graph Pooling (1)
II: Compensation: (1) Complement graph or (2) Back-projection residual ⊖ ⊕
G
G P G P
G(2) indices of nodes in the complement graph formulate: CompIdx = { i ( v ) | v ∈ V and i ( v ) ̸∈ Idx ( t ) } , where i ( · ) gives a unique index to a node. The complement denotes the information that has been lost during the first phase of GSC, and it can be adopted as the residual signal to be fused with the pooled graph.
The approach is to propagate node features from the complement graph to the pooled graph leveraging an UnPool layer: E ( t ) Comp = UnPool ( X ( t ) Comp , A ( t ) ) , where the UnPool denotes an unpooling process.
It receives node features of the complement graph as the input feature and the original graph structure as adjacency relation, and interpolates compensated information to those selected nodes of the pooled graph.
Similar to (Gao & Ji, 2019), it can be devised by initializing the input feature matrix X ( t ) Comp ∈ R N × d as:
[Equation]
here we denote the index selection operator in the brackets of the left-hand side for clarity, and then
UnPool propagate it by a message passing process, which is implemented by a 1-layer GCN.
2) Coarsened graph back-projection: Another source of the compensated information is inspired by the classic back-projection algorithm (Irani & Peleg, 1991) used for iteratively refining the restored high-resolution images.
The intuition is that a pooled graph is ideal if it contains adequate information for reconstructing the original graph.
Rather than introducing additional knowledge to model the pattern of lost information, coarsened graph back-projection manages to restore the original graph using the pooled node features themselves, and then calculates the reconstruction error to serve as the compensated signals.
The restoration process is given by: X ( t ) Recon =
UnPool ( X ( t ) Coarse , A ( t ) ) , where the input feature matrix of X ( t ) Coarse ∈ R N × d is initialized by the pooled node features
X ( t +1) ∈ R | Idx |× d with padding zero vectors:
[Equation]
After that, the residual graph is calculated by: E ( t ) Recon = X
( l t [Equation] P t ) − X ( t ) Recon
[Equation]
2. P HASE 3: I NFORMATION FEEDBACK
GSC finally refines the pooled graph X ( t +1) with the residual embedded graph of E ( t ) ∈
R N × d as: ˜ X ( t +1) =
X ( t +1) + E ( t )
Idx , : , where E ( t ) Idx , : ∈ R | Idx |× d is tailored from the residual signal, either
E ( t ) Comp ∈ R N × d
or E ( t ) Recon ∈ R N × d , using the indices Idx ( t ) .
[Table]
D IVERSITY B OOSTING R EGULARIZERS
The collection of graph embeddings with T scales formulates G =
( x (1) G , · · · , x ( T ) G ) T ∈ R T × d , where each element x ( t ) G ∈ R 1 × d is readout from X ( t ) .
Motivated by the theory of determinant point processes (DPPs) (Kulesza & Taskar, 2012), we define the diversity of multiscale graph embeddings as DoG = det( Σ ) , where Σ is the similarity matrix of the multiscale graph embeddings.
For example, it could be the gram matrix, where each entry represents the inner-product similarity score of a pair of graph embeddings Σ = ˜ G ˜ G T , where ˜ G is row-wise normalized from G , for guaranteeing the property of positive semi-definiteness on Σ .
According to the matrix theory (Zhang, 2011), the value of det( GG T ) equals to the squared volume of subspace spanned by graph embeddings { x ( t ) G | t ∈ { 1 , ..., T }} , and hence DoG reaches the maximum value if and only if the graph embeddings are mutually orthogonal.
Noticed that the normalization of graph embeddings would reduce the variance of them, and lead the optimization problem to become trivial for regularizing the networks.
We introduce the gaussian kernel to parameterize the similarity matrix, which formulates Σ i,j = exp( − γ · d 2 ( G i , G j )) , where d ( · , · ) calculates the Euclidean distance and γ is a hyper-parameter to control the flatness of the similarity matrix.
Under this definition, we propose the Diversity Boosting Regularizer (DBR) to further diversify the multiscale graph embeddings,
[Equation]
where the first term is the logarithm of embeddings diversity, the second term as normalization controls the magnitude of the similarity matrix.
The calculations of DBR is efficient since the pooling number T grows much slower with the scale of problem.
Lastly, we combine the regularizer α · L DBR ( α as loss weight) with the summed cross-entropy loss Σ T L CE over all predicted logits as the training objective function.
E XPERIMENTS
We compare the approach with previous graph learning methods, including hierarchical pooling based models of DiffPool (Ying et al., 2018), TopK (Gao & Ji, 2019), SAGPool (Lee et al., 2019),
ASAP (Ranjan et al., 2020), and global pooling based models of Set2Set (Vinyals et al., 2015),
GlobalAttention (Li et al., 2015) and SortPool (Zhang et al., 2018).
Among them, we select the two state-of-the-art pooling methods, SAGPool and ASAP as the evaluation baseline backbones to conduct in-depth comparisons.
We use the same hyper-parameters search strategy for all the baselines and our method.
We denote GSC that calculates the feedback signal from complement graph as ‘GSC-C’, and the one from graph back-projection as ‘GSC-B’.
The overall results are summarized in Table 2.
One can observe that the GSC mechanism achieves consistent and considerable performance improvement
[Table]
N o r m a li z e d d i s t a n c e 0.
N o r m a li z e d d i s t a n c e
[Figure] [Figure]
DD
PROTEINS
NCI NCI
MOLHIV Datasets
[Figure]
for all the cases.
It improves around 1 .
5% accuracy and 1 . 0% accuracy over six benchmarks on the baseline of SAGPool and ASAP, respectively.
Generally, the complement graph fusion (‘GSC-C’) performs better than the back projection (‘GSC-B’) on the SAGPool method, because the complement graph provides the pooled graph with richer information of the semantic representations of unselected nodes.
Instead, in the cases of NCI109 and MOLHIV on ASAP, GSC based on back projection (‘GSC-B’) achieves superior performance, which might indicate that the reconstruction residuals have well refined the updating pooled node features.
We particularly compare the ensemble performance between with and without graph self-correction mechanism on the pooling process.
The results are separately given in Table 2 and Table 1.
Table 1 illustrates the failure of ensemble strategy on boosting multiscale GNNs built on baseline pooling modules.
The homogeneity of node features of standard pooling methods is shown in Figure(a).
In contrast, GSC could relieve such homogeneity and further contribute to the success of the ensemble strategy.
For the multiscale GNNs equipped with GSC procedure (‘GSC-C’ or ‘GSC-B’), the ensemble learning (‘GSC-C+’ or ‘GSC-B+’) successfully achieves over 0 .
5% average accuracy improvement on the even stronger baselines.
We conduct a comparison study on the ensemble multiscale GNNs equipped with GSC, under the difference between training with and without the proposed DBR.
The results are displayed in Figure 2 (b), which verify that DBR can jointly improve the ensemble performance with the GSC mechanism.
One can see that DBR achieves smaller standard deviation under the 10-fold cross validation, which means the diversified graph embeddings indeed provide a more comprehensive characterization of the input graph, and thus improve the training stability.
C ONCLUSION
We provide an orthogonal perspective for multiscale graph learning by establishing a practical ensemble model.
The graph self-correction not only leads to considerable improvements over existing graph pooling methods by serving as a plug-in component, but also contributes to promoting the ensemble diversity on node-level embeddings.
Working together with the diversity boosting regularizer that enhances diversity on graph-level embeddings, they jointly lead the ensemble model to achieve superior performances.
A CKNOWLEDGEMENTS
This work was supported in part by the National Natural Science Foundation of China (61972219), the Research and Development Program of Shenzhen (JCYJ20190813174403598,
SGDX20190918101201696), the National Key Research and Development Program of China (2018YFB1800601), and the Overseas Research Cooperation Fund of Tsinghua Shenzhen International Graduate School (HW2021013).
A T HE HOMOGENIZATION OF NODE FEATURES DURING POOLING
Figure 3 shows the phenomenons that the nodes become severely homogenized after the second graph pooling process with SAGPool (Lee et al., 2019), while the GSC mechanism helps relieve this issue.
The visualization analysis has provided a deeper insight into the beneficial effect of graph self-correction on the ensemble multiscale learning, that is, promoting the preserving of nodes discrepancy in the coarse graph via restoring the fine graph, which further helps to increase the diversity of input information of the ensemble model.
[Figure]
instead
B E VALUATION PROTOCOL FOR POOLING - BASED METHODS
A fair evaluation protocol is crucial for evaluating the performance of ensemble-based models.
Since the ensemble strategy aims at making generalized predictions, the ensemble model with good ability would show superior performance than the individual base learners while the distribution of validation set is inconsistent with test set.
However, if one performs model selection on the testset (Xu et al., 2018; Huang et al., 2019; Bianchi et al., 2020; Li et al., 2020), it could result in a misleading conclusion about the performance of ensemble strategy.
This could be verified by the experimental example provided in Table 3.
[Table]
C H YPER - PARAMETER SEARCH STRATEGY
For all the experiments
[Table]
we use Adam optimizer to train the model for 200 epochs in total, with learning rate decay of 0.5 after every 50 epochs.
We follow previous works to perform hyperparameter search for each experiment, and the search range is given by Table 4 (a).
For the γ in the gaussian kernel and α the loss weight of DBR, the search range is provided in Table 4 (b).
Specially, we fix the γ for the experiments on MOLHIV as 0 .
003 due to the statistical difference of the data ego-features.
[Table]