M ULTI - TASK S ELF - SUPERVISED G RAPH N EURAL N ETWORKS E NABLE S TRONGER T ASK G ENERALIZATION
Anonymous authors
Paper under double-blind review
A BSTRACT
Self-supervised learning (SSL) for graph neural networks (GNNs) has attracted increasing attention from the graph machine learning community in recent years, owing to its capability to learn performant node embeddings without costly label information.
One weakness of conventional SSL frameworks for GNNs is that they learn through a single philosophy, such as mutual information maximization or generative reconstruction.
When applied to various downstream tasks, these frameworks rarely perform equally well for every task, because one philosophy may not span the extensive knowledge required for all tasks.
In light of this, we introduce
P ARETO GNN, a multi-task SSL framework for node representation learning over graphs.
Specifically, P ARETO GNN is self-supervised by manifold pretext tasks observing multiple philosophies.
To reconcile different philosophies, we explore a multiple-gradient descent algorithm, such that P ARETO GNN actively learns from every pretext task while minimizing potential conflicts.
We conduct comprehensive experiments over four downstream tasks (i.e., node classification, node clustering, link prediction, and partition prediction), and our proposal achieves the best overall performance across tasks on 11 widely adopted benchmark datasets.
Besides, we observe that learning from multiple philosophies enhances not only the task generalization but also the single task performance, demonstrating that P ARETO GNN achieves better task generalization via the disjoint yet complementary knowledge learned from different philosophies.
I NTRODUCTION
Graph-structured data is ubiquitous in the real world (McAuley et al., 2015; Hu et al., 2020a).
To model the rich underlying knowledge for graphs, graph neural networks (GNNs) have been proposed and achieved outstanding performance on various tasks, such as node classification (Kipf & Welling, 2016a; Hamilton et al., 2017), link prediction (Zhang & Chen, 2018;
Zhao et al., 2022), node clustering (Bianchi et al., 2020; You et al., 2020b), etc.
These tasks form the archetypes of many real-world practical applications, such as recommendation systems (Ying et al., 2018; Sankar et al., 2021) and predictive user behavior models (Pal et al., 2020; Wang et al., 2020).
Existing works for graphs serve well to make progress on narrow experts and guarantee their effectiveness on mostly one task or two.
However, given a graph learning framework, its promising performance on one task may not (and usually does not) translate to competitive results on other tasks.
Consistent task generalization across various tasks and datasets is a significant and well-studied research topic in other domains (Wang et al., 2018;
Yu et al., 2020).
Results from the Natural
Language Processing (Radford et al., 2019; Sanh et al., 2021) and Computer Vision (Doersch &
Zisserman, 2017; Ni et al., 2021) have shown that models enhanced by self-supervised learning (SSL) over multiple pretext tasks observing diverse philosophies can achieve strong task generalization and learn intrinsic patterns that are transferable to multiple downstream tasks.
Intuitively, SSL over multiple pretext tasks greatly reduces the risk of overfitting (Baxter, 1997; Ruder, 2017), because learning intrinsic patterns that well-address difficult pretext tasks is non-trivial for only one set of parameters.
Moreover, gradients from multiple objectives regularize the learning model against extracting task-irrelevant information (Ren & Lee, 2018; Ravanelli et al., 2020), so that the model can learn multiple views of one training sample.
Nonetheless, current state-of-the-art graph SSL frameworks are mostly introduced according to only one pretext task with a single philosophy, such as mutual information maximization (Velickovic et al., 2019; Zhu et al., 2020; Thakoor et al., 2022), whitening decorrelation (Zhang et al., 2021a), and generative reconstruction (Hou et al., 2022).
Though these methods achieve promising results in some circumstances, they usually do not retain competitive performance for all downstream tasks across different datasets.
For example, DGI (Velickovic et al., 2019), grounded on mutual information maximization, excels at the partition prediction task but underperforms on node classification and link prediction tasks.
Besides, G RAPH MAE (Hou et al., 2022), based on feature reconstruction, achieves strong performance for datasets with powerful node features (e.g., graph topology can be inferred simply by node features (Zhang et al., 2021b)), but suffers when node features are less informative, which is empirically demonstrated in this work.
To bridge this research gap, we ask:
How to combine multiple philosophies to enhance task generalization for SSL-based GNNs?
A very recent work, A UTO SSL (Jin et al., 2022), explores this research direction by reconciling different pretext tasks by learning different weights in a joint loss function so that the node-level pseudo-homophily is promoted.
This approach has two major drawbacks: (i) Not all downstream tasks benefit from the homophily assumption.
In experimental results shown by Jin et al.
, we observe key pretext tasks (e.g., DGI based on mutual information maximization) being assigned zero weight.
However, our empirical study shows that the philosophies behind these neglected pretext tasks are essential for the success of some downstream tasks, and this phenomenon prevents
GNNs from achieving better task generalization.
(ii)
In reality, many graphs do not follow the homophily assumption (Pei et al., 2019; Ma et al., 2021).
Arguably, applying such an inductive bias to heterophilous graphs is contradictory and might yield sub-optimal performance.
In this work, we adopt a different perspective: we remove the reliance on the graph or task alignment with homophily assumptions while self-supervising GNNs with multiple pretext tasks.
During the self-supervised training of our proposed method, given a single graph encoder, all pretext tasks are simultaneously optimized and dynamically coordinated.
We reconcile pretext tasks by dynamically assigning weights that promote the Pareto optimality (D ´ esid ´ eri, 2012), such that the graph encoder actively learns knowledge from every pretext task while minimizing conflicts.
We call our method
P ARETO GNN.
Overall, our contributions are summarized as follows:
• We investigate the problem of task generalization on graphs in a more rigorous setting, where a good SSL-based GNN should perform well not only over different datasets but also at multiple distinct downstream tasks simultaneously.
Hence, the model is required to learn several patterns transferable to multiple tasks on various graphs.
We evaluate state-of-the-art graph SSL frameworks in this setting and unveil their sub-optimal task generalization.
• To enhance task generalization, we first design five simple and scalable pretext tasks according to philosophies proven to be effective in the SSL literature and propose P ARETO GNN, a multi-task SSL framework for GNNs.
P ARETO GNN is simultaneously self-supervised by these pretext tasks, which are dynamically reconciled to promote the Pareto optimality, such that the graph encoder actively learns knowledge from every pretext task while minimizing potential conflicts.
• We evaluate P ARETO GNN along with 7 state-of-the-art SSL-based GNN methods on 11 community acknowledged benchmarks over 4 downstream tasks (i
e., node classification, node clustering, link prediction, and partition prediction).
Our experiments show that P ARETO GNN improves the overall performance by up to +5.3% over the state-of-the-art SSL-based GNNs, demonstrating the significantly stronger task generalization of our proposal.
Besides, we observe that P ARETO GNN achieves SOTA single-task performance, proving that P ARETO GNN achieves
better task generalization via the disjoint yet complementary knowledge learned from different philosophies.
M ULTI -T ASK S ELF - SUPERVISED L EARNING VIA P ARETO GNN
In this section, we illustrate our proposed multi-task self-supervised learning framework for GNNs, namely P ARETO GNN.
As Figure 1 illustrates, P ARETO GNN is trained with different SSL tasks simultaneously to enhance the task generalization.
Specifically, given a graph G with N nodes and their corresponding D -dimensional input features X ∈ R N × D , P ARETO GNN learns a GNN encoder f g ( · ; θ g ) :
G → R N × d parameterized by θ g , that maps every node in G to a d -dimensional vector
𝒯
[Figure]
Forward Pass
GNN Encoder 𝑓(•;𝜽 " )
Feat.
Recon.
𝑓 !
(•;𝜽 𝟏 )
Mutual Info.
𝑓 # (•;𝜽 𝟐 )
Decorrelation 𝑓 % (•;𝜽 𝟑 )
K-th SSL 𝑓 ’ (•;𝜽 𝑲 ) … ℒ !
𝒢 𝒯 !
𝜽 𝒈 𝜽ℒ % 𝒢 𝒯 % 𝜽 𝒈 𝜽ℒ ’ 𝒢 𝒯 ’ 𝜽 𝒈 𝜽ℒ )
𝒢 𝒯 )
𝜽 𝒈 𝜽 𝐊 …
Multi-Task Reconciliation ∇ ) !
ℒ ! ∇ ) !
ℒ # ∇ ) !
ℒ % ∇ ) !
ℒ *
C o n v e x
H u ll
Gradient
Calculation 𝛼 !
∗ ∇ ) !
ℒ !
𝛼 # ∗ ∇ ) !
ℒ # 𝛼 % ∗ ∇ ) ! ℒ % 𝛼 ’ ∗ ∇ ) !
ℒ *
Min-norm Promoting
Pareto Optimality
Full Graph
SSL Task-specific Augmentation ∇ + !
=
𝛼 !
∗ ∇ + !
ℒ !
+ 𝛼 % ∗ ∇ + !
ℒ % + 𝛼 ’ ∗ ∇ + !
ℒ ’ + 𝛼 - ∗ ∇ + ! ℒ ) …
Descent Direction for 𝑓(•;𝜽 " )
Node RepresentationGeneration
Task Specific Projection
ObjectiveFormulation
[Figure]
[Figure]
(∇ + ! ℒ ! ,∇ + !
ℒ % ,∇ + !
ℒ ’ ,…,∇ + ! ℒ ) )
[Figure] [Figure]
(s.t. d ≪ N ).
The resulting representations should retain competitive performance across various downstream tasks without any update on θ g .
With K self-supervised tasks, we consider the loss function for k -th SSL task as L k ( G ; T k , θ g , θ k ) :
G → R + , where T k refers to the graph augmentation function required for k -th task, and θ k refers to task-specific parameters for k -th task (e.g., MLP projection head and/or GNN decoder).
In P ARETO GNN, all SSL tasks are dynamically reconciled by promoting Pareto optimality, where the norm of gradients w.r.t.
the parameters of our
GNN encoder θ g is minimized in the convex hull.
Such gradients guarantee a descent direction to the
Pareto optimality, which enhances task generalization while minimizing potential conflicts.
2. M ULTI - TASK G RAPH S ELF - SUPERVISED L EARNING
P ARETO GNN is a general framework for multi-task self-supervised learning over graphs.
We regard the full graph G as the data source; and for each task, P ARETO GNN is self-supervised by sub-graphs sampled from G , followed by task-specific augmentations (i.e., T k ( · ) ).
The rationale behind the exploration of sub-graphs is two-fold.
Firstly, the process of graph sampling is naturally a type of augmentation (Zeng et al., 2019) by enlarging the diversity of the training data.
Secondly, modeling over sub-graphs is more memory efficient, which is significant especially under the multi-task scenario.
In this work, we design five simple pretext tasks spanning three high-level philosophies, including generative reconstruction, whitening decorrelation, and mutual information maximization.
However we note that P ARETO GNN is not limited to the current learning objectives and the incorporation of other philosophies is a straightforward extension.
Three high-level philosophies and their corresponding five pretext tasks are illustrated as follows:
• Generative reconstruction .
Recent studies (Zhang et al., 2021b;
Hou et al., 2022) have demonstrated that node features contain rich information, which highly correlates to the graph topology.
To encode node features into the representations derived by P ARETO GNN, we mask the features of a random batch of nodes, forward the masked graph through the GNN encoder, and reconstruct the masked node features given the node representations of their local sub-graphs (Hou et al., 2022).
Furthermore, we conduct the similar reconstruction process for links between the connected nodes to retain the pair-wise topological knowledge (Zhang & Chen, 2018).
Feature and topology reconstruction are denoted as FeatRec and TopoRec respectively.
• Whitening decorrelation .
SSL based on whitening decorrelation has gained tremendous attention, owing its capability of learning representative embeddings without prohibitively expensive negative
pairs or offline encoders (Ermolov et al., 2021; Zbontar et al., 2021).
We adapt the same philosophy to graph SSL by independently augmenting the same sub-graph into two views, and then minimize the distance between the same nodes in the two views while enforcing the feature-wise covariance of all nodes equal to the identity matrix.
We denote this pretext as RepDecor .
• Mutual Information Maximization .
Maximizing the mutual information between two corrupted views of the same target has been proved to learn the intrinsic patterns, as demonstrated by deep infomax-based methods (Bachman et al., 2019; Velickovic et al., 2019) and contrastive learning methods (Hassani & Khasahmadi, 2020; Zhu et al., 2020).
We maximize the local-global mutual information by minimizing the distance between the graph-level representation of the intact sub-graph and its node representations, while maximizing the distance between the former and the corrupted node representations.
Besides, we also maximize the local sub-graph mutual information by maximizing similarity of representations of two views of the sub-graph entailed by the same anchor nodes, while minimizing the similarities of the representations of the sub-graphs entailed by different anchor nodes.
The pretext tasks based on node-graph mutual information and node-subgraph mutual information are denoted as MI-NG and MI-NSG , respectively.
Technical details and objective formulation of these five tasks are provided in Appendix B.
As described above, pretext tasks under different philosophies capture distinct dimensions of the same graph.
Empirically, we observe that simply combining all pretext SSL tasks with weighted summation can sometimes already lead to better task generalization over various downstream tasks and datasets.
Though promising, according to our empirical studies, such a multi-task self-supervised GNN falls short on some downstream tasks, if compared with the best-performing experts on these tasks.
This phenomenon indicates that with the weighted summation there exist potential conflicts between different SSL tasks, which is also empirically shown by previous works from other domains such as
Computer Vision (Sener & Koltun, 2018; Chen et al., 2018).
2. M ULTI - TASK G RAPH SSL PROMOTING P ARETO O PTIMALITY
To mitigate the aforementioned problem and simultaneously optimize multiple SSL tasks, we can derive the following empirical loss minimization formulation as:
[Equation]
where α k is the task weight for k -th SSL task computed according to pre-defined heuristics.
For instance, A UTO SSL (Jin et al., 2022) derives task weights that promote pseudo-homophily.
Though such a formulation is intuitively reasonable for some graphs and tasks, heterophilous graphs, which are not negligible in the real world, directly contradict the homophily assumption.
Moreover, not all downstream tasks benefit from such homophily assumption, which we later validate in the experiments.
Hence, it is non-trivial to come up with a unified heuristic that suits all graphs and downstream tasks.
In addition, weighted summation of multiple SSL objectives might cause undesirable behaviors, such as performance instabilities entailed by conflicting objectives or different gradient scales (Chen et al., 2018; Kendall et al., 2018).
Therefore, we take an alternative approach and formulate this problem as multi-objective optimization with a vector-valued loss L , as the following:
[Equation]
The focus of multi-objective optimization is approaching Pareto optimality (D´esid´eri, 2012), which in the multi-task SSL setting can be defined as follows:
Definition 1 (Pareto Optimality) .
A set of solution ( θ ⋆g , θ ⋆ 1 , ..., θ ⋆K ) is Pareto optimal if and only if there does not exists a set of solution that dominates ( θ ⋆g , θ ⋆ 1 , ..., θ ⋆K ).
( θ ⋆g , θ ⋆ 1 , ..., θ ⋆K ) dominates ( ˆ θ g , ˆ θ 1 , ..., ˆ θ K ) if for every SSL task k , L k ( G ; T k , ˆ θ g , ˆ θ k ) ≥
L k ( G ; T k , θ ⋆g , θ ⋆k ) and
L ( G , ˆ θ g , ˆ θ 1 , . . .
, ˆ θ K ) ̸
= L ( G , θ ⋆g , θ ⋆ 1 , . . .
, θ ⋆K ) .
In other words, if a self-supervised GNN is Pareto optimal, it is impossible to further optimize any
SSL task without sacrificing the performance of at least one other SSL task.
Finding the Pareto optimal model is not sensible if there exist a set of parameters that can easily fit all SSL tasks (i.e., no matter how different SSL tasks are reconciled, such a model approaches Pareto optimality where every SSL task is perfectly fitted).
However, this is rarely the case, because solving all difficult pretext tasks is non-trivial for only one set of parameters.
By promoting the Pareto optimlaity, P ARETO GNN is enforced to learn intrinsic patterns applicable to a number of pretext tasks, which further enhances the task generalization across various downstream tasks.
2. P ARETO O PTIMALITY BY M ULTIPLE G RADIENT D ESCENT A LGORITHM
To obtain the Pareto optimal parameters, we explore the Multiple Gradient Descent Algorithm (MGDA) (D´esid´eri, 2012) and adapt it to the multi-task SSL setting.
Specifically, MGDA leverages the saddle-point test and theoretically proves that a solution (i.e., the combined gradient descent direction or task weight assignments in our case) that satisfies the saddle-point test gives a descent direction that improves all tasks and eventually approaches the Pareto optimality.
We further elaborate descriptions of the saddle-point test for the share parameters θ g and task-specific parameters θ k in
Appendix G.
In our multi-task SSL scenario, the optimization problem can be formulated as:
[Equation]
k
[Equation]
α k =and ∀
[Equation]
(3) where ∇ θ g L k ( G ; T k , θ g , θ k ) ∈
R 1 ×| θ g | refers to the gradients of parameters for the GNN encoder w.r.t.
the k -th SSL task.
P ARETO GNN can be trained using Equation (1) with the task weights derived by the above optimization.
As shown in Figure 1, optimizing the above objective is essentially finding descent direction with the minimum norm within the convex hull defined by the gradient direction of each SSL task.
Hence, the solution to Equation (3) is straight-forward when K = 2 (i.e., only two gradient descent directions involved).
If the norm of one gradient is smaller than their inner product, the solution is simply the gradient with the smaller norm (i.e., ( α 1 = 0 , α 2 = 1 ) or vice versa).
Otherwise, α 1 can be calculated by deriving the descent direction perpendicular to the convex line with only one step, formulated as:
[Equation] [Equation]
When K > 2 , we minimize the quadratic form α ( ∇ θ g L )( ∇ θ g L ) ⊺ α ⊺ , where ∇ θ g L ∈ R K ×| θ g | refers to the vertically concatenated gradients w.r.t.
θ g for all SSL tasks, and α ∈ R 1 × K is the vector for task weight assignments such that
||
α ||
= 1 .
Inspired by Frank-Wolfe algorithm
(Jaggi, 2013), we iteratively solve such a quadratic problem as a special case of Equation (4).
Specifically, we first initialize every element in α as 1 /K , and we increment the weight of the task (denoted as t ) whose descent direction correlates least with the current combined descent direction (i.e., (cid:80)
Kk =1 α k ·∇ θ g L k
( G ; T k , θ g , θ k ) ).
The step size η of this increment can be calculated by utilizing the idea of Equation (4), where we replace ∇ θ g L 2 ( G ; T 2 , θ g , θ 2 ) with (cid:80) Kk =1 α k · ∇ θ g L k
( G ; T k , θ g , θ k ) and replace ∇ θ g L 1 ( G ; T 1 , θ g , θ 1 ) with ∇ θ g L t ( G ; T t , θ g , θ t ) .
One iteration of solving this quadratic problem is formulated as:
[Equation]
where t = arg min r (cid:80) Ki =
α i · ∇ θ g L i ( G ; T i , θ g , θ i ) · ∇ θ g L r
( G ; T r , θ g , θ r ) ⊺ and ˆ ∇ θ g = (cid:80)
Kk
=
α k · ∇ θ g L k
( G ; T k , θ g , θ k ) .
e t in the above solution refers to an one-hot vector with t -th element equal to 1.
The optimization described in Equation (5) iterates until η is smaller than a small constant ξ or the number of iterations reaches the pre-defined number γ .
Furthermore, the above task reconciliation of
P ARETO GNN satisfies the following theorem.
Theorem 1.
Assuming that α ( ∇ θ g L )( ∇ θ g L ) ⊺ α ⊺ is β -smooth.
α converges to the optimal point at a rate of O (1 /γ ) , and α is at most 4 β/ ( γ + 1) away from the optimal solution.
The proof of Theorem 1 is provided in Appendix A.
According to this theorem and our empirical observation, the optimization process in Equation (5) would terminate and deliver well-approximated results with γ set to an affordable value (e.g., γ = 100 ).
E XPERIMENTS 3. E XPERIMENTAL S ETTING
Datasets .
We conduct comprehensive experiments on 11 real-world benchmark datasets extensively explored by the graph community.
They include 8 homophilous graphs, which are Wiki-CS ,
Pubmed , Amazon-Photo , Amazon-Computer , Coauthor-CS , Coauthor-Physics , ogbn-arxiv , and ogbn-products (McAuley et al., 2015; Yang et al., 2016;
Hu et al., 2020a), as well as 3 heterophilous graphs, which are Chameleon , Squirrel , and Actor (Tang et al., 2009; Rozemberczki et al., 2021).
Besides the graph homophily, this list of datasets covers graphs with other distinctive characteristics (i.e., from graphs with thousands of nodes to millions, and features with hundred dimensions to almost ten thousands), to fully evaluate the task generalization under different scenarios.
The detailed description of these datasets can be found in Appendix C.
Downstream Tasks and Evaluation Metrics .
We evaluate all models by four most commonlyused downstream tasks, including node classification, node clustering, link prediction, and partition prediction, whose performance is quantified by accuracy, normalized mutual information (NMI), area under the characteristic curve (AUC), and accuracy respectively, following the same evaluation protocols from previous works (Tian et al., 2014; Kipf & Welling, 2016a; Zhang & Chen, 2018).
Evaluation Protocol .
For all downstream tasks, we follow the standard linear-evaluation protocol on graphs (Velickovic et al., 2019; Jin et al., 2022; Thakoor et al., 2022), where the parameters of the GNN encoder are frozen during the inference time and only logistic regression models (for node classification, link prediction and partition prediction) or K-Means models (for node clustering) are trained to conduct different downstream tasks.
For datasets whose public splits are available (i.e., ogbn-arxiv , and ogbn-products ), we utilize their given public splits for the evaluations on node classification, node clustering and partition prediction.
Whereas for other datasets, we explore a random 10 % /
% split for the train/validation/test split, following the same setting as explored by other literature.
To evaluate the performance on link prediction, for large graphs where permuting all possible edges is prohibitively expensive, we randomly sample 210,000, 30,000, and 60,000 edges for training, evaluation, and testing.
And for medium-scale graphs, we explore the random split of 70 % /
% , following the same standard as explored by (Zhang & Chen,
; Zhao et al., 2022).
To prevent label leakage for link prediction, we evaluate link prediction by another identical model with testing and validation edges removed.
Label for the partition prediction is induced by metis partition (Karypis & Kumar, 1998), and we explore 10 partitions for each dataset.
All reported performance is averaged over 10 independent runs with different random seeds.
Baselines .
We compare the performance of P ARETO GNN with 7 state-of-the-art self-supervised
GNNs, including DGI (Velickovic et al., 2019), GRACE (Zhu et al., 2020), MVGRL (Hassani &
Khasahmadi, 2020), A UTO SSL (Jin et al., 2022), BGRL (Thakoor et al., 2022), CCA-SSG (Zhang et al., 2021a) and G RAPH MAE (Hou et al., 2022).
These baselines are experts in at least one of the philosophies of our pretext tasks, and comparing P ARETO GNN with them demonstrates the improvement brought by the multi-task self-supervised learning as well as promoting Pareto optimlaity.
Hyper-parameters .
To ensure a fair comparison, for all models, we explore the GNN encoder with the same architecture (i.e., GCN encoder with the same layers), fix the hidden dimension of the GNN encoders, and utilize the recommended settings provided by the authors.
Detailed configurations of other hyper-parameters for P ARETO GNN are illustrated in Appendix D.
3. P ERFORMANCE G AIN FROM THE M ULTI - TASK S ELF - SUPERVISED L EARNING
We conduct experiments on our pretext tasks by individually evaluating their performance as well as task generalization, as shown in Table 1.
We first observe that there does not exist a single-task model that can simultaneously achieve competitive performance on every downstream task for all datasets, demonstrating that knowledge learned through a single philosophy does not suffice the strong and consistent task generalization.
Models trained by a single pretext tasks alone are narrow experts (i.e., delivering satisfactory results on only few tasks or datasets) and their expertise does not translate to the strong and consistent task generalization across various downstream tasks and datasets.
For instance, TopoRec achieves promising performance on link prediction (i.e., average rank of 3.7), but falls short on all other tasks (i.e., ranked 5.9, 5.3, and 5.9).
Similarly, MI-NSG performs reasonably well on the partition prediction (i.e., average rank of 3.0), but underperforms on the link
[Table]
prediction task (i.e., average rank of 4.4).
However, comparing them with the model trained by combining all pretext tasks through the weighted summation (i.e., w/o Pareto ), we observe that the latter achieves both stronger task generalization and better single-task performance.
The model w/o Pareto achieves an average rank of 2.4 on the average performance, which is 1.2 ranks higher than the best single-task model.
This phenomenon indicates that multi-task self-supervised GNNs indeed enable stronger task generalization.
Multiple objectives regularize the learning model against extracting redundant information so that the model learns multiple complementary views of the given graphs.
Besides, multi-task training also improves the performance of the single downstream task by 1.5, 0.7, 0.3, and 0.6 ranks, respectively.
In some cases (e.g., node clustering on P UBMED and C HAMELEON , or link prediction on W IKI .CS and C O .CS), we observe large performance margins between the best-performing single-task models and the vanilla multi-task model w/o Pareto , indicating that there exist potential conflicts between different SSL tasks.
P ARETO GNN further mitigates these performance margins by promoting Pareto optimality, which enforces the learning model to capture intrinsic patterns applicable to a number of pretext tasks while minimizing potential conflicts.
As shown in Table 1, P ARETO GNN is the top-ranked at both average metric and other individual downstream tasks, demonstrating the strong task generalization as well as the promising single-task performance.
Specifically, P ARETO GNN achieves an outstanding average rank of 1.0 on the average performance of the four downstream tasks.
As for the performance on individual tasks, P ARETO GNN achieves an average rank of 1.7, 1.6, 1.0, and 1.2 on the four individual downstream tasks, outperforming the corresponding best baselines by 0.2, 1.2, 1.3, and 1.1 respectively.
This phenomenon demonstrates that promoting Pareto optimality not only helps improve the task generalization across various tasks and datasets but also enhances the single-task performance for the multi-task self-supervised learning.
[Table]
3. P ERFORMANCE C OMPARED WITH O THER U NSUPERVISED B ASELINES
We compare P ARETO GNN with 7 state-of-the-art SSL frameworks for graphs, as shown in Table 2.
Similar to our previous observations, there does not exist any SSL baseline that can simultaneously achieve competitive performance on every downstream task for all datasets.
Specifically, from the perspective of individual tasks, CCA-SSG performs well on the node classification but underperforms on the link prediction and the partition prediction.
Besides, DGI works well on the partition prediction but performs not as good on other tasks.
Nevertheless, from the perspective of datasets, we can observe that A UTO SSL has good performance on homohpilous datasets but this phenomenon does not hold for heterophilous ones, demonstrating that the assumption of reconciling tasks by promoting graph homophily is not applicable to all graphs.
P ARETO GNN achieves a competitive average rank of 1.0 on the average performance, significantly outperforming the runner-ups by 2.7 ranks.
Moreover, P ARETO GNN achieves an average rank of 1.8, 1.9, 1.0, and 1.2 on the four individual tasks, outrunning the best-performing baseline by 0.9, 2.3, 3.3, and 1.6 respectively, which further proves the strong task generalization and single-task performance of P ARETO GNN.
3. P ERFORMANCE WHEN S CALING TO L ARGER D IMENSIONS
To evaluate the scalability of P ARETO GNN, we conduct experiments from two perspectives: the graph and model dimensions.
We expect our proposal to retain its strong task generalization when applied to large graphs.
And we should expect even stronger performance when the model dimension scales up, compared with other single-task frameworks with larger dimensions as well, since multi-task SSL settings enable the model capable of learning more.
The results are shown below.
A v e r a g e P e r f .
[Figure]
[Figure] [Table] [Figure] [Table]
[Figure] [Table] [Figure] [Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure] From Figure 2, we observe that the task generalization of P ARETO GNN is proportional to the model dimension, indicating that P ARETO GNN is capable of learning more, compared with single-task models like BGRL, whose performance gets saturated with less-parameterized models (e.g.,for A M .C OMP .
and 256 for A RXIV ).
Moreover, from Table 3, we notice that the graph dimension is not a limiting factor for the strong task generalization of our proposal.
Specifically, P ARETO GNN outperforms the runner-ups by 2.5 on the rank of the average performance.
R ELATED W ORKS
Graph Neural Networks .
Graph neural networks (GNNs) are powerful learning frameworks to extract representative information from graphs (Kipf & Welling, 2016a; Veli ˇ ckovi ´ c et al., 2017;
Hamilton et al., 2017; Xu et al., 2018b; Klicpera et al., 2019; Xu et al., 2018a; Wu et al., 2020).
They aim at mapping the input nodes into low-dimensional vectors, which can be further utilized to conduct either graph-level or node-level tasks.
Most GNNs explore layer-wise message passing scheme (Gilmer et al., 2017), where a node iteratively extracts information from its first-order neighbors, and information from multi-hop neighbors can be captured by stacked layers.
Self-supervised Learning for GNNs .
For node-level tasks, current state-of-the-art graph SSL frameworks are mostly introduced according to a single pretext task with a single philosophy (You et al., 2020b), such as mutual information maximization (Velickovic et al., 2019; Zhu et al., 2020;
Hassani & Khasahmadi, 2020; Thakoor et al., 2022), whitening decorrelation (Zhang et al., 2021a), and generative reconstruction (Kipf & Welling, 2016b; Hou et al., 2022).
AutoSSL (Jin et al., 2022) explores a multi-task setting where tasks are reconciled in a way that promotes graph homophily.
Whereas for graph-level tasks, previous works explore mutual information maximization to encourage different views of the same graphs sharing similar representations (You et al., 2020a;
Xu et al., 2021;
You et al., 2021; Li et al., 2022).
Multi-Task Self-supervised Learning .
Multi-task SSL is broadly explored in computer vision (Lu et al., 2020; Doersch & Zisserman, 2017; Ren & Lee, 2018; Yu et al., 2020; Ni et al., 2021) and natural language processing (Wang et al., 2018;
Radford et al., 2019; Sanh et al., 2021; Ravanelli et al., 2020) fields.
For the graph community, AutoSSL (Jin et al., 2022) explores a multi-task setting where tasks are reconciled in a way that promotes graph homophily (Zhao et al., 2021). Besides, Hu et al.
(2020b) focuses on the graph-level tasks and pre-trains GNNs in separate stages.
In these frameworks, tasks are reconciled according to weighted summation or pre-defined heuristics.
C ONCLUSION
We study the problem of task generalization for SSL-based GNNs in a more rigorous setting and demonstrate that their promising performance on one task or two usually does not translate into good task generalization across various downstream tasks and datasets.
In light of this, we propose P ARETO GNN to enhance the task generalization by multi-task self-supervised learning.
Specifically, P ARETO GNN is self-supervised by manifold pretext tasks observing multiple philosophies, which are reconciled by a multiple-gradient descent algorithm promoting Pareto optimality.
Through our extensive experiments, we show that multi-task SSL indeed enhances the task generalization.
Aided by our proposed task reconciliation, P ARETO GNN further enlarges the margin by actively learning from multiple tasks while minimizing potential conflicts.
Compared with 7 state-of-the-art SSL-based GNNs , P ARETO GNN is top-ranked on the average performance.
Besides stronger task generalization, P ARETO GNN achieves better single-task performance, demonstrating that disjoint yet complementary knowledge from different philosophies is learned through the multi-task SSL.
E THICS S TATEMENT
We observe no ethical concern entailed by our proposal but we note that both ethical or unethical applications based on graphs may benefit from the stronger task generalization of our work.
Generally speaking, care should be taken to ensure socially positive and beneficial results of machine learning algorithms.
R EPRODUCIBILITY S TATEMENT
To ensure the reproducibility of our experiments and benefit the research community we provide the source code for P ARETO GNN along with the submission in the supplementary materials.
We will open-source our code upon the acceptance of this paper.
The hyper-parameters and other variables required to reproduce our experiments are described in Appendix D.
A P ROOF TO T HEOREM
Here we re-state Theorem 1 before diving into its proof:
Theorem 1.
Assuming that α ( ∇ θ g L )( ∇ θ g L ) ⊺ α ⊺ is β -smooth.
α converges to the optimal point at a rate of O (1 /γ ) , and α is at most 4 β/ ( γ + 1) away from the optimal solution.
Proof.
Let ϕ ( α ) denotes α ( ∇ θ g L )( ∇ θ g L ) ⊺ α ⊺ .
If ϕ ( · ) is β -smooth, we have:
[Equation]
from which we can also derive a quadratic upper-bound:
[Equation]
where α ′ refers to the weight combination after one iteration from α .
Combining Equations (5) and (7), we have:
[Equation]
where R = S UP α 1 , α 2 ∈X ( || α 1 − α 2 || ) refers to the diameter of the domain of α ( ∇ θ g L )( ∇ θ g L ) ⊺ α ⊺ , and S UP ( · ) is the supremum operation.
In our case, R = √ 2 since || α ||
= 1 .
The derivation above is valid because α ′ =
(1 − η ) · α + η · e t , as shown in Equation (5).
Since t = arg min r (cid:80) Ki
=
α i · ∇ θ g L i ( G ; T i , θ g , θ i ) · ∇ θ g L r
( G ; T r , θ g , θ r ) ⊺ , we have:
[Equation]
where α ∗ is the unknown optimal solution.
By rearranging terms in the above inequality, we have:
[Equation]
The left hand side of the above equation is the distance between the updated solution and the optimal solution (we denote this distance as δ ′ =
ϕ ( α ′ )
− ϕ ( α ∗ ) ).
And on the right hand side, inside the parenthesis we have the distance between the previous solution and the optimal solution (denoting this distance as δ =
ϕ
( α ) − ϕ
( α ∗ ) )
as:
[Equation]
where in our case, η = ˆ ∇ θ g · (cid:0) ˆ ∇ θ g −∇ θ g L t ( G ; T t , θ g , θ t ) (cid:1) ⊺ (cid:12)(cid:12)(cid:12)(cid:12)
[Equation]
g
[Equation] [Equation]
F .
In this proof, we assume η = 2 γ t +1 , where γ t refers to the index of the current iteration.
Such an assumption is a relaxed version of our formulation upon η , hence any proof that holds for this assumption also holds for our case.
Next, we show δ γ ≤ 4 β γ +1 in Theorem 1 through proof-by-induction:
It’s straight-forward and easy to validate that our derivation stands for the base case γ = 2 , where δ 2 ≤ 4 β 3 .
Here we show that it also holds for γ + 1 :
[Equation]
B D ESCRIPTION OF THE P RETEXT T ASKS
In this section, we demonstrate the design of our proposed five pretext tasks, including two based on generative reconstruction (i.e., FeatRec and TopoRec ), one based on whitening decorrelation (i.e., RepDecor ) and two based on mutual information maximization (i.e., MI-NG and MI-NSG ).
We first explain the operation of graph convolution as proposed by Kipf & Welling (2016a).
The key mechanism of graph convolution is layer-wise message passing where a node iteratively extracts information from its first-order neighbors and information from milti-hop neighbors can be captured through stacked convolution layers.
Specifically, at l -th layer, this process is formulated as follows:
[Equation]
where H 0 = X , A ∈ { 0 , 1 } N × N is the adjacency matrix of the input graph, σ ( · ) refers to the non-linear activation function, W l ∈ R d l × d l +1 refers to the learnable parameters of l -th layer, and d l and d l +1 are the hidden dimensions at these two consecutive layers respectively.
The graph encoder f g ( · ; θ g ) : G → R N × d of P ARETO GNN is constructed by stacked graph convolution layers as:
[Equation]
where L stands for the number of layers in the encoder of P ARETO GNN and θ g = { W l } L − 1 l =0 .
As briefly described in Section 2.1, we regard the full graph G as the data source; and for each task, P ARETO GNN is self-supervised by sub-graphs sampled from G , followed by task-specific augmentations (i.e., T t ( · ) ).
The graph sampling strategy is fairly straightforward.
For the pretext task t , we select the sub-graph constituted by nodes within k t hops of N t randomly selected seed nodes, where k t and N t are two task-specific hyper-parameters.
The graph augmentation operations we explore include feature masking, edge dropping, and node dropping.
For the simplicity of denotation, we unify the operation of graph sampling and graph augmentation together as T t ( · ) .
Task-specific hyper-parameters for graph augmentation and sub-graph sampling are covered in Appendix D.
B. G ENERATIVE R ECONSTRUCTION
Feature reconstruction , denoted as FeatRec , utilizes the high-level idea from Zhang et al.
(2021b)
, proving that topological information can be referred purely from the node features.
Hence to utilize such inductive bias, following the implementation of GraphMAE (Hou et al., 2022), we mask the node features and forward the masked graphs through f g ( · ; θ g ) .
Then we re-mask the previous masked nodes and feed the resulted graph to a convolution-based decoder, formulated as:
[Equation]
) where ⊙ refers to Hadamard product, A ′ is the adjacency matrix of the sampled sub-graph
G ′ ∼ T FeatRec ( G ) , M ∈ { 0 , 1 } N ′ ×
d is the feature mask matrix whose rows equal to 1 if their corresponding nodes are targeted for reconstruction, and W Dec ∈ R d × D is the parameter matrix for the feature decoder.
The objective for FeatRec is formulated as:
[Equation]
(16) where ˆ M ∈ { 0 , 1 } N ′
×
D is the mask matrix defined similarly to M with different dimension, and
X ′ ∈ R N ′
× D is the feature matrix of the sampled sub-graph G ′ ∼ T FeatRec ( G ) .
Topological reconstruction , denoted as TopoRec , aims at capturing the pair-wise relationships between the connected nodes.
Given a sampled sub-graph G ′ ∼ T TopoRec ( G ) , we randomly select B pairs of nodes V + = { ( i, j ) |
A ′ i,j = 1 } and another B pairs of nodes V − =
{ ( i, j ) |
A ′ i,j = 0 } .
The connection between two node i and j is measured by a logit calculated as:
[Equation]
where [ · ] refers to the indexing operation, and
W
Topo ∈ R d × 1 is the parameter vector.
The objective of
TopoRec is maximizing the P TopoRec for nodes in V + and minimizing for nodes in V − , formulated as a binary cross entropy loss as:
[Equation]
B. W HITENING D ECORRELATION
Representation decorrelation , denoted as RepDecor , encourages the similarities between the representations of the same nodes in two independently augmented sub-graphs.
During this process, to the prevent the node representations from collapsing into a trivial solution, the covariance between representations matrices of two sub-graphs are enforced to be an identity matrix, such that the knowledge learned by each dimension in the hidden space is orthogonal to each other (Ermolov et al., 2021; Zbontar et al., 2021; Zhang et al., 2021a).
Given two sub-graphs G ′ 1 , G ′ 2 ∼ T TopoRec ( G ) that are constituted by the same seed nodes but augmented differently, the objective of RepDecor is formulated as:
[Equation]
where the first term encourages the node similarity and the second term regularize the solution from collapsing, I d × d is the square identity matrix with dimension d × d , and α refers to a pre-defined balancing term (i.e., we use an α of 1e-3 across all datasets).
B. M UTUAL I NFORMATION M AXIMIZATION
Mutual information between nodes and the whole graph , denoted as MI-NG , enables the graph encoder to learn coarse graph-level knowledge.
Specifically, given a sampled sub-graph G ′ ∼
T MI-NG ( G ) , we first corrupt G ′ into G ′′ by feature shuffling (Velickovic et al., 2019).
Then we extract the hidden graph-level representation of G ′ by the graph mean pooling (Xu et al., 2018a), and enforce the representations of nodes in G ′ similar to the pooled representation while nodes in G ′′ far away from the pooled representation.
This pretext task allows the graph encoder to capture the perturbation brought by the topological change (i.e., feature shuffling).
The objective of MI-NG is formulated as a binary cross entropy as:
[Equation]
where P OOL ( · ) refers to the graph mean pooling function (Xu et al., 2018a), || is the horizontal concatenation operation, and W MI-NG ∈ R 2 d × 1 is the parameter vector.
Mutual information between nodes and their sub-graphs , denoted as MI-NSG , enables the graph encoder to learn fine-grained graph-level knowledge.
Unlike MI-NG that enforces mutual information between node representations and the graph-level representation, MI-NSG maximizes the mutual information between the independently augmented sub-graphs entailed by the same anchor nodes, which learns fine-grained knowledge compared with MI-NG .
Specifically, given two sub-graphs
G ′ 1 , G ′ 2 ∼ T TopoRec ( G ) that are constituted by the same seed nodes but augmented differently
, the objective of MI-NSG is formulated as a variant of InfoNCE (Chen et al., 2020):
[Equation]
[Table]
(cid:16)
[Table]
(cid:17)(cid:17) , (21) where S IM ( h 1 , h 2 ) =
h 1 · h ⊺ 2 ||
h 1 || F ·||
h
|| F is the similarity metric, E XP ( · ) stands for the exponential function, and τ is the temperature hyper-parameter used to control the sharpness of the similarity distribution (i.e., we explore a τ of 0.1 across all datasets).
C D ATASET D ESCRIPTION
We evaluate our proposed P ARETO GNN as well as unsupervised SSL-based GNNs on 11 realworld datasets spanning various fields such as citation network and merchandise network.
Their statistics are shown in Table 4.
For Wiki-CS , Pubmed , Amazon-Photo , Amazon-Computer ,
Coauthor-CS , and Coauthor-Physics , we use the API from Deep Graph Library (DGL) to load the datasets.
For ogbn-arxiv and ogbn-products , we use the API from Open Graph
Benchmark (OGB) 2 .
For Chameleon , Actor and Squirrel , the datasets are downloaded from the official repository of Geom-GCN (Pei et al., 2019) 3 .
[Table]
D A DDITIONAL E XPERIMENTAL S ETTINGS
Hyper-parameters .
The hyper-parameters for P ARETO GNN across all datasets are listed in Table 5.
, and we use its node version.
[Table]
Wiki
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
FeatureMaskRatio
[Table]
[Figure] [Table] [Figure] [Table]
[Figure] [Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
SamplingStrategy
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
N
[Table]
Hardware and software configurations .
We conduct experiments on a server having one RTX
GPU with 24 GB VRAM.
The CPU we have on the server is an AMD
Baseline Implementation .
As for the baseline models that we compare P ARETO GNN with, we explore the implementations provided by code repositories listed as follows:
• DGI (Velickovic et al., 2019): https://github.com/dmlc/dgl/tree/master/ examples/pytorch/dgi .
• GRACE (Zhu et al., 2020): https://github.com/dmlc/dgl/tree/master/ examples/pytorch/grace .
• MVGRL (Hassani & Khasahmadi, 2020): https://github.com/dmlc/dgl/ tree/master/examples/pytorch/mvgrl .
• A UTO SSL
(Jin et al., 2022): https://github.com/ChandlerBang/AutoSSL .
• BGRL (Thakoor et al., 2022): https://github.com/dmlc/dgl/tree/master/ examples/pytorch/bgrl .
• CCA-SSG (Zhang et al., 2021a): https://github.com/hengruizhang98/ CCA-SSG .
• G RAPH MAE (Hou et al., 2022): https://github.com/THUDM/GraphMAE .
We sincerely appreciate the authors of these works for open-sourcing their valuable code and researchers at DGL for providing reliable implementations of these models.
E T RAINING T IME AND M EMORY C ONSUMPTION
The scalability w.r.t.
the graph dimensions is well leveraged by our utilization of sampled sub-graphs and experimentally verified by P ARETO GNN’s strong performance over large graphs, as shown in
Table 3.
On top of this, we also measure the training time and GPU memory consumption to give a direct empirical understanding of P ARETO GNN’s overhead, as shown in Table 6.
For A UTO SSL, we notice that the calculation of the pseudo-homophily is extremely slow because such a process cannot enjoy the GPU acceleration.
GPU remains mostly idle during the training of A UTO SSL.
Though P ARETO GNN is not as efficient as BGRL when the graphs are small-scaled; for large graphs such as OGBN - PRODUCTS , all methods require sampling strategies and in this case the efficiency of
P ARETO GNN is on par with that of BGRL.
BGRL learns from one large graph (though sampled), and P ARETO GNN learns from multiple relatively small graphs, which entails similar computational overheads.
[Table]
F P ERFORMANCE OF I NDIVIDUAL T ASKS ON L ARGE G RAPHS
In Table
[Table]
we demonstrate the task generalization of all models over large graphs (i
[Table]
e., ogbn-arxiv and ogbn-products ) quantified by the average performance over the four downstream tasks.
Here
[Table]
we provide additional experimental results on models’ performance of every individual task, as shown in Table 7.
We notice that the graph dimension is not a limiting factor for the strong task generalization of our proposal.
Besides the conclusion we have drawn in Section 3.4, where P ARETO GNN outperforms the runner-ups by 2.5 on rank of the average performance calculated over the four tasks, we also observe strong single-task performance on some tasks, demonstrating that P ARETO GNN achieves better task generalization via the disjoint yet complementary knowledge learned from different philosophies.
G S ADDLE - POINT T EST FOR P ARAMETERS IN P ARETO GNN
In P ARETO GNN, the saddle-point test conditions (D ´ esid ´ eri, 2012) for the shared GNN encoder (i.e., f g ( · ; θ g ) ) as well as the task-specific heads (i.e., f k ( · ; θ k ) ) are defined as the following:
• For θ g , there exist α such that every element in α is greater than or equal to 0,
||
α ||
= 1 , and (cid:80)
Kk
=
α k · ∇ θ g L k
( G ; T k , θ g , θ k )
= 0 .
• For { θ k } Kk =1 , we have ∇ θ k L k ( G ; T k , θ g , θ k ) = 0 .
According to MGDA, the solution above gives a descent direction that improves all tasks, which improves the task generalization while minimizing potential conflicts. H C ONNECTIONS
TO O THER P ARETO L EARNING F RAMEWORKS
This section is greatly inspired by valuable comments from the reviewers of this paper 4 .
We sincerely appreciate endeavors from the reviewers to help us further refining this paper.
Task reconciliation by Pareto learning is mostly explored by works in the Computer Vision community (Lin et al., 2019; Mahapatra & Rajan, 2020;
Chen et al., 2021).
In their settings, usually there exist one main task and multiple auxiliary tasks.
They explore Pareto learning with preference vectors to enforce the learning models to remain good performance on the main task while extracting information from auxiliary tasks as much as possible.
Pareto learning with user-defined preference vectors is sensible in their cases, as they do not want the auxiliary tasks to hinder the learning of the main task (i.e., a preference of the main task over the auxiliary tasks).
For example, PSST (Chen et al., 2021) enhances the model’s few-shot learning capability (i.e., the main task) by optimizing the model with the main task as well as additional pretext tasks (i.e., the auxiliary tasks).
In this case, while remaining competitive performance on the main task, PSST also regularizes the learning model against extracting task-irrelevant information (Ren & Lee, 2018; Ravanelli et al., 2020) by training with auxiliary tasks, which empirically further improves the performance on the main task.
However, our scenario is completely different from theirs because we do not want a task governing others (i.e., all tasks are main tasks).
The task reconciliation proposed in our paper is not biased toward any task.
If preference vectors are explored, some pretext tasks will definitely be jeopardized due to the definition of Pareto optimality (i.e., Definition 1).
Such a bias over the pretext tasks would cause the self-supervised GNNs not performing equally well on every downstream task and dataset.
It hurts the overall average performance and task generalization, which are the main focuses of this paper.
To empirically validate that an utilization of a preference vector contradicts our goal, we conduct experiments on P ARETO MTL (Lin et al., 2019) with five preference vectors (i.e., five vectors for different preferences over five tasks), as shown in Table 8.
[Table]
We observe that multi-task self-supervised learning via Pareto learning with a preference vector under most of the time does not even outperform the vanilla weighted summation (i.e., w/o Pareto ) due to the bias introduced by the preference vectors.
Our proposed P ARETO GNN consistently outperforms variants based on P ARETO MTL with different preferences.
Though sometimes variants with preference vectors approach the performance of P ARETO GNN (i.e., P ARETO MTL with MI-NSG in W IKI .CS), coming to this results needs a grid-search on all possible preference vectors, which is inefficient when heuristics such as the prior knowledge are not available.
Including preferences in multi-task learning could be very helpful when one or a set of training tasks need to be highlighted, whose success is demonstrated by the experiments in P ARETO MTL.
But this is not our case, because to achieve strong task generalization for GNNs, philosophies contained in all pretext tasks are equally important.
Our implementation of P ARETO MTL comes from its official Github repository 5 .