A Closer Look at Distribution Shifts and Out-of-Distribution Generalization on Graphs
Abstract
Distribution shifts, in which the training distribution differs from the testing distribution, can signiﬁcantly degrade the performance of Graph Neural Networks (GNNs).
We curate GDS, a benchmark of eight datasets reﬂecting a diverse range of distribution shifts across graphs.
We observe that: (1) most domain generalization algorithms fail to work when applied to domain shifts on graphs; and (2) combinations of powerful GNN models and augmentation techniques usually achieve the best out-of-distribution performance.
These emphasize the need for domain generalization algorithms tailored for graphs and further graph augmentation techniques that enhance the robustness of predictors.
Introduction
Distribution shifts, in which training and testing distributions differ, often make machine learning systems fail in spectacular ways [1].
The over-reliance on the training distribution makes it challenging to apply systems in practical scenarios where distribution shifts are common, such as graph classiﬁcation problems.
Graphs are standard data structures that abstract complex systems of interacting objects, such as molecular graphs [2], biological networks [3] and social networks [4].
Distribution shifts arise naturally in many graph classiﬁcation problems since it is infeasible to prepare a training set that covers all domains of interest.
In problems such as drug discovery and social media fact-checking, molecular graph structures often differ at inference [5, 2] and news propagation graphs may grow with time [6]
In this paper we consider domain generalization on graphs (see Appendix A for detailed deﬁnition) where training and test graphs are collected from related but different domains.
Such domain shifts challenge the out-of-distribution (OOD) generalization abilities of graph neural networks (GNNs), as we often see that OOD test performance is usually signiﬁcantly lower than in-distribution (ID) test performance (see Fig. 1).
Despite the real-world prevalence of distribution shifts on graphs, they have not been thoroughly evaluated and discussed to the best of our knowledge.
Graph learning benchmarks such as [7] note the broad existence of distribution shifts on graphs and have considered using domain information to split training and test sets.
We further aim to (1) provide domain labels separately from features which allows the applications of domain generalization algorithms; (2) characterize the types of domain shifts exhibited in the datasets using statistical means; (3) and evaluate what challenges such domain shifts impose on GNN models.
We curate GDS (Graph Distribution Shift), a benchmark of eight datasets reﬂecting a diverse range of distribution shifts across graphs; see Fig. 1.
To facilitate further research, we provide an open-source package that administers the GDS benchmarks with modular combinations of popular ∗ Equal
Workshop on Distribution Shifts, 35th Conference on Neural Information Processing Systems (NeurIPS 2021).
MolPCBA
GossipCop
Isolation
[Figure]
ColoredMNIST
Train Set
[Figure]
[Figure] [Table] MetaLearning [Table]
AugmentationTechniques
[Table]
Types
[Table] Global [Table] Context Techniques
Virtual Node [15]
Spectral Convolution
ChebNet [16]
WL-Isomorphism Hierarchy 3WL-GNN [17]
Structure-Aware Networks
[Table]
domain generalization algorithms, graph augmentation techniques, and GNN backbone models.
This framework can streamline rigorous and reproducible experiments in the future. Another major contribution of this work is a rigorous comparison of methods for improving OOD generalization on graphs with respect to GDS. The three high-level categories considered are (1) existing OOD generalization algorithms; (2) augmentations for graph data; and (3) GNN models and techniques that improve OOD generalization; see Table 1. The results of our comprehensive evaluation support two primary conclusions (demonstrated in Tables 2 to 6): (1) most OOD generalization algorithms fail to work when applied to domain shifts on graphs; and (2) combinations of best performing GNN models and augmentation techniques usually achieve the best OOD performance.
These observations underscore the need for new OOD generalization algorithms tailored for domains shifts on graphs or further graph augmentation techniques that make the predictor robust to more types of spurious correlations.
Graph Distribution Shift (GDS) Datasets
We curate eight datasets reﬂecting a diverse range of distribution shifts across graphs; see Fig.
1 for illustrations, Table 7 for statistics, and Appendix B for details.
• Maximum common subgraph: M OL HIV and M OL PCBA are two real-world molecular graph datasets adopted from [2, 7].
The task is to predict molecular properties.
The domains are deﬁned by scaffolds, the core structures of small molecules [19].
In Fig.
1, we see that molecular graphs within a domain share a maximum common subgraph. • Multi-hop neighborhoods: PPA is a real-world dataset of protein association networks adopted from [3, 7], and the task is to predict which taxonomic group (out of 37 groups) a graph originates from.
Each graph in PPA is a subgraph sampled from the 2-hop neighborhood of a protein, where the species of the center protein (1,581 species in total) deﬁnes the domain.
• Size growth: G OSSIP C OP is a real-world dataset consisting of news propagation graphs adopted from [4, 20], which are abstractions of social engagement information (retweets between users). [Table] [Table]
[Table]
The task is to predict the credibility of news.
We split the entire dataset into ten domains according to graph sizes, where each domain corresponds to a decile (every 10% percentile group).
• Subgraph compositions: I SOLATION and E NVIRONMENT are two artiﬁcial graph datasets generated with Stochastic Block Models (SBMs) [21], where each graph consists of communities, each with a speciﬁc intra-community connection probability.
In both datasets, the domain is characterized by the composition of communities selected, but their prediction tasks differ.
We predict the extra-community connection probabilities in I SOLATION , and the composition with respect to another set of communities in E NVIRONMENT .
• Structural distortion: R OTATED MNIST is a collection of semi-artiﬁcial super-pixel graphs converted from the “image Rotated MNIST [22] dataset” using the SLIC [23] pipeline.
The task is to classify digits, and the domain is deﬁned by the rotation angle of the underlying image.
The rotation of the underlying image leads to a distortion in the super-pixel graph; see
Fig.
1. • Feature shift only: C OLORED MNIST is another semi-artiﬁcial graph dataset converted from the “image Colored MNIST dataset [9]” using SLIC.
The domain affects the true node-feature-to-label correlations while having nothing to do with the graph structures, fooling the algorithms that only learn from the node features.
Generalization Algorithms, Augmentation Techniques, and GNN Models
We consider a variety of (1) domain generalization algorithms; (2) graph augmentation techniques; and (3) GNN models/techniques; see Table 1 for the list of algorithms and models considered, Sectionfor discussions and Appendix C for details
• Domain generalization algorithms aim to bias a model towards learning statistical invariances across the training distributions under the assumption that such invariances hold in unseen test [Table] [Table] domains.
These methods specify both a prior on the types of invariances desired and an algorithm for estimating them from training samples.
Since the speciﬁc statistical invariances these algorithms learn may be highly dependent on the type of data and the network architectures to which they are applied, each technique’s applicability to graph data and GNNs models is unclear.
• Data augmentation methods are a standard component in image classiﬁcation pipelines, but augmentation methods for graph data are not as well studied or prevalent in practice.
We consider two types of graph augmentations: (1) augmentations to node features following an adversarialbased augmentation algorithm [13]; (2) and augmentations to graph structures using the node dropping and edge perturbation operations proposed in [24].
• GNN models and techniques.
The expressive power of the backbone model is also an important factor for generalizing to unseen test data [25].
Some GNN models or techniques may improve OOD generalization performance as they may be robust to certain domain shifts on graphs.
For example, ChebNet [16], a spectral graph convolution model, may perform well in terms of OOD generalization when the domain shift is easier learned in the Fourier domain.
Experiments
OOD generalization performance of GNN models and techniques.
Before evaluating generalization and augmentation algorithms, we want to understand the OOD generalization performance of off-the-shelf GNN models and techniques.
Table 2 summarizes the out-of-distribution (OOD) generalization results of all GNN models and Table 3 shows the corresponding in-distribution (ID) generalization performance.
Here, OOT means out of the 24-hour quota of training time.
The gap between the ID and OOD generalization performance will then be an indicator of the model’s robustness to the domain shift. • Virtual Node, ChebNet, 3WL-GNN, and GSN outperform baselines (GIN, MLP, and GIN-Deep) both ID and OOD.
Virtual Node excels on M OL PCBA, PPA, E NVIRONMENT , when long-range correlation is critical or graphs are large (see Table 7 for statistics). • GSN shows robustness to the domain shifts in M OL HIV: it is not the most performant in-distribution but beats other models on the OOD generalization task.
Similarly, ChebNet is robust to the shifts in graph sizes on G OSSIP C OP .
• Finally, Virtual Node fails on G OSSIP C OP where it must generalize to test domains consisting of larger graphs.
The reported performance 0 .
499 ± 0 .
078 AUC is no better than random guessing ( 0 .
5 AUC).
Performance of domain generalization and augmentation algorithms.
We then evaluate the six domain generalization algorithms and the two graph augmentation methods; see Table 4.
• Augmentation methods, FLAG, and SA, generally work much better than domain generalization algorithms across the datasets (except for MLDG on G OSSIP C OP ).
This implies many of the domain generalization algorithms proposed for tensor data are not well-suited to graph data and GNNs out-of-the-box.
• On M OL PCBA, M OL HIV, and PPA, generalization algorithms (GroupDRO, IRM, DeepCORAL, and MLDG) fail consistently compared to the ERM baselines.
• IRM does not outperform the other algorithms on our graph C OLORED MNIST dataset because of the model selection criteria, similar to what was reported in [25].
Combinations of domain generalization algorithms or augmentation techniques with the best performing GNN models.
We then evaluate the OOD generalization performance of algorithms combined with the best performing models selected from Table 2 for each dataset; see Table 5.
We also report the performance of these combinations when tested in-distribution; see Table 6.
• The differences in performance between domain generalization algorithms in Table 5 are slightly larger than those in Table 4.
And by comparing to Table 4, we see DANN-G outperforms MLDG on G OSSIP C OP , and MLDG outperforms SA on R OTATED MNIST when using ChebNet as the backbone model.
Summary of observations.
We summarize our observations from Tables 2 to 6 as follows:
• Most of the domain generalization algorithms fail when applied to graph distribution shift datasets; as shown in Tables 4 and 5.
Their performance is usually close to the corresponding ERM baseline and can be even lower on the M OL HIV, M OL PCBA, and PPA datasets.
• The best performance on a dataset is usually achieved by the best performing GNN model with an augmentation technique; see Table 5.
This implies that graph augmentations are promising research directions to improve OOD generalization on graphs further.
• Lastly, in Table 6, the gaps between in-distribution and out-of-distribution generalization performance of the best-performing pairs varies across datasets.
Most of the gaps are large, especially on PPA, M OL PCBA, and C OLORED MNIST, addressing that there is still large room to improve the OOD generalization on those datasets by designing tailored algorithms to capture their speciﬁc statistical invariances.
Supplementary Material A Problem Deﬁnitions
Supervised learning on graphs and graph classiﬁcation.
The goal of supervised graph learning is to train a featurizer φ : G → H which maps a graph G =
( V, E ) with node attributes X v for v ∈ V and edge attributes e u,v for ( u, v ) ∈
E to a representation vector h G , which can then be used to predict the label y of graph G through a classiﬁer ω :
H → Y in graph classiﬁcation problems.
In the supervised learning problem of graph classiﬁcation, the training dataset S =
{ ( G i , y i ) } ni [-=
 containsi.i.d.-]=1 contains i.i.d.
samples from the joint probability distribution P ( G, y ) .
We choose a predictor f = ω ◦ φ that minimizes the empirical risk 1 n (cid:80) ni
=1 (cid:96) ( f ( G i ) , y i ) [26], where (cid:96) :
Y × Y → [0 , ∞ ) is the loss function.
The ubiquitous choice of hypothesis class for learning graph representations are Graph
Neural Networks (GNNs), which apply permutation-equivariant operations, e.g. message passing between connected nodes, and a permutation-invariant graph-level pooling function to all nodes representations to obtain h G .
Domain generalization on graphs.
The problem of domain generalization on graphs is an extension of supervised learning where the data distribution is a mixture of D domains { 1 , · · · , D } , each characterized by a dataset S d = { ( G di , y di ) } n d i =1 consisting of i.i.d.
samples from distribution P ( G d , y d ) .
We train on the training domains { 1 , · · · , D train } , and the goal is out-of-distribution (OOD) generalization to D test test domains { D train + 1 , · · · , D train + D test } .
The test domains are not accessible during training, which differs from unsupervised domain adaptation, where unlabeled data from test domains are available during training.
The sample space of graph structure is high dimensional and discrete, exhibiting a different nature than the Euclidean space, R d , of conventional vector features.
An OOD generalization algorithm that learns the statistical invariances of the training distributions on graphs should also respect permutation symmetries, making the challenging OOD generalization problem even harder on graphs.
In the OOD generalization setting, the highly dimensional and discrete nature of graph structure introduces added complexity as compared to structures like 2D grids of continuous pixel values used to represent images.
If we consider connected undirected graphs without loops of size k , there are O (2( k 2 )) different representations of adjacency matrices
A ∈ { 0 , 1 } ( n 2 ) , while there are only O (2( k 2 ) /k !)
distinct graphs.
The fact that a graph structure can be encoded differently implies that the functions acting on graphs should satisfy the permutation invariance given by the permutation group G = Σ k , i.e., the outcomes of a function on any two isomorphic graphs should be identical.
If not mentioned otherwise in this work, we assumed the featurizer φ was a permutation-invariant function (i.e., GNN with graph pooling), and the classiﬁer ω a Multi-Layer Perceptron (MLP).
However, in other cases, the featurizer φ is permutation-equivariant, and the classiﬁer ω is permutation-invariant, where embeddings φ ( G ) are also attribute graphs.
B GDS Datasets
At the heart of our experiments is the design of graph datasets with rich distribution shifts; see Tablefor more information and Fig.
1 for illustrations.
For each dataset, we either prepare or recover the domain information, and as an OOD generalization benchmark, the domain labels d for each sample are included in the datasets. [Table] [Figure] [Table]
[Figure]
[Table]
[Figure]
[Table] [Figure] [Table] Maximum common subgraph: OGB-M OL HIV and OGB-M OL PCBA are two real-world molecular graph datasets adopted from Wu et al.
[2], Hu et al.
Each graph is an abstraction of a molecule, where nodes are atoms and edges are chemical bonds.
The task is to predict molecular properties, e.g., whether the given molecule inhibits HIV virus replication in OGB-M OL HIV.
The domains are deﬁned by scaffolds, the core structures of small molecules [19], which can be translated
[Figure]
into the language of graph theory as the maximal isomorphic subgraph.
In Fig.
1, we see that molecular graphs within a domain share a maximum common subgraph (i.e., the scaffold).
They differ from each other only on a few atoms added to different locations of the scaffold.
The graph edit distances [27] between graphs in the same domain are very small but can be unbounded across domains.
We have many domains in the training, validation, and test splits, as most of the domains contain few samples.
Each of the test domains in the two datasets has only a single sample, which imposes yet another challenge to some OOD generalization algorithms such as DANN [11]; We adopt the two molecule datasets from the OGB paper [7].
As the original paper already splits the train, validation, and test sets using domain information (scaffold), we follow this practice to leave the split unchanged.
Multi-hop neighborhoods: OGB-PPA is a real-world dataset of protein association networks adopted from [3, 7].
The graphs in OGB-PPA are abstractions of protein-protein association relations of 1 , 581 species, and the task is to predict which taxonomic group (out of 37 groups) a graph originates from.
Each graph in OGB-PPA is a subgraph sampled from the 2-hop neighborhood of a protein, where the species of the center protein deﬁnes the domain.
Thus, graphs within a domain share a common type of center node (center proteins are identical species).
Although the center node is always removed in sub-sampling, we believe the sampled neighborhoods around proteins of a speciﬁc species exhibit sufﬁcient similarity to be considered as a coherent domain.
Similarly, as the
OGB-PPAdataset is also adopted from the OGB datasets, we follow the original practice to split the train, validation, and test sets using the species information provided by the dataset.
Size growth: UPFD-G OSSIP C OP is a real-world dataset consisting of news propagation graphs adopted from [4, 20].
The propagation networks are abstractions of social engagement information (retweets between users) collected from Twitter, built according to fact-check information from
Gossipcop and extracted by FakeNewsNet [20].
Given this propagation network, the task is to predict the credibility of news, either real or fake.
We split the entire dataset into ten domains according to graph sizes, where each domain corresponds to a decile (every 10% percentile group).
The training and validation sets are randomly selected from the 80% smallest graphs with a split ratio of 6:2, while the test set is the 20% largest graphs.
Domains constructed to simulate the scenario of training on propagation graphs ﬁltered to a speciﬁc range of sizes but then deploying to real-time fact-checking systems facing an environment with propagation graphs of growing sizes.
On the original dataset, we ﬁnd a multi-layer perceptron (MLP) can achieve 0 .
948 accuracy, which is even higher than the accuracy of a Graph Isomorphism Network (GIN) with the same number of layers and parameters, 0 . 930 .
Thus, we replace all node features (which are user proﬁle information) prepared in Dou et al. [4] with random integers uniformly selected from a vocabulary of size 8 , i.e., { 0 , · · · , 7 } .
We then use the ROC-AUC metric to reﬂect false positives.
Subgraph compositions: SBM-I SOLATION and SBM-E NVIRONMENT are two artiﬁcial graph datasets generated with Stochastic Block Models (SBMs) [21].
A SBM deﬁnes a random graph which modulates the intra- and extra-community connection probabilities.
In SBM-I SOLATION , we randomly select 3 from the 5 total communities C 1 , · · · , C 5 for each graph, with intra-edge probability p 1 = 0 .
5 , · · · , p 5 = 0 .
9 respectively, and we predict the number of communities with relatively smaller extra-edge probability q (cid:48)
= 0 .
1 (with default q = 0 . 3 ).
The domain is characterized by the composition of communities selected, for example, { C 1 , C 3 , C 5 } .
While in SBM-E NVIRONMENT , we deﬁne 7 communities { C 1 , · · · , C 7 } with intra-edge probabilities p 1 = 0 .
3 , p 2 = 0 .
4 , · · · , p 7 = 0 .
9 and set the extra-edge probability q to 0 .
For each graph, we select 3 communities from { C 1 , C 3 , C 5 , C 7 } and 2 communities from { C 2 , C 4 , C 6 } , and we use the former selections to deﬁne the domain and the later selections as the prediction task.
In one sentence, SBM-I SOLATION and SBM-E NVIRONMENT are similar in their distribution shifts: graphs in different domains consist of distinct community subgraphs, although their prediction tasks differ.
When generalizing to unseen test graphs, we have already seen all communities, but we have never seen them appear in the same graph.
Structural distortion: S UPER P IXEL -R OTATED MNIST is a collection of semi-artiﬁcial graphs constructed from images in Rotated MNIST [22] using the super-pixel [23] pipeline.
In Rotated
MNIST, handwritten-digit images are split into 6 equal folds and rotated counter-clockwisely by 0 ° , 15 ° , 30 ° , 45 ° , 60 °, and 75 °, respectively.
Because super-pixels sampled near image boundaries and on digits are not signiﬁcantly affected by rotations, the super-pixel graphs converted from images rotated with different angles can be thought of as graphs that undergo some “structure distortion”; see Fig. 1 for visualizations.
The task is to classify digits with 1-dimensional node features as the super-pixel intensities.
We train and validate on graphs converted from 0 ° , · · · , 60 °-rotated images and generalize to unseen 75 ° ones.
For the generation process of the dataset, after we attain the images which are rotated for certain degrees, we use the SuperPixel [23] image segmentation algorithm to extract nodes that will represent small regions of homogeneous intensity.
Then we construct edges using k -nearest neighbor algorithm
(we set k as 8 here), where distances are computed by the 2D coordinates of each node.
We set the upper bound of the number of super-pixels to 75 and the compactness of the SuperPixel [23] image segmentation algorithm to 0.25.
We adopt the same setting of the SuperPixel algorithm from Dwivedi et al.
Feature shift only:
S UPER P IXEL -C OLORED MNIST is another semi-artiﬁcial graph dataset constructed using Colored MNIST images [9] using the same super-pixel method as above.
In
Colored MNIST, the domain affects the true image-to-label correlations, fooling the algorithms that try to learn this mapping directly.
When converted to graphs, the super-pixel colors are used as node features, and GNNs which only use the node features will fail on the test graphs, where the feature-to-label correlation is reversed.
However, GNNs that only learn the structure-to-label mapping will generalize as if the test graphs are in-distribution.
Domain shifts on S UPER P IXELC OLORED MNIST only affect node features, making it a feature-shift-only dataset.
For the Colored
MNIST dataset, we follow the original paper [9] to generate images with color shifts.
Totally there are three domains, where the ﬁrst two form the train and validation sets, and the last one is the test set.
Firstly a temporary binary label ˜
y is assigned to each image.
For digits 0-4, ˜ y
= 0 ; for digits 5-9, ˜ y
= 1 .
Secondly, ˜ y is randomly ﬂipped with the probability 0.25 to get the label y .
Then, we paint red color to images with label y
= 1 and green to whose y = 0 .
Lastly, the colors are randomly ﬂipped by the probability of 0.1, 0.2, and 0.9 for each domain, respectively.
For graph construction, the pipeline is exactly the same as that for the Rotated MNIST.
The dataset is constructed this way so that the models that rely heavily on the node features will fail.
In-distribution vs. out-of distribution validation set.
An important fact we want to note is that, among the eight datasets we curated, M OL HIV, M OL PCBA, and PPA’s validation sets are out-ofdistribution (i.e., not part of the training domains), while the rest ﬁve datasets’ validation set are just randomly sampled from the training domains.
We understand that both setups are acceptable.
Sampling the validation set from the training domains is just the standard model selection method discussed in [25], while using an out-of-distribution validation set can sometimes improve the OOD test performance as shown in [29].
We did a simple ablation study on M OL HIVand found that using an in-distribution validation set, the OOD test performance of empirical risk minimization with a
Graph Isomorphism Network (GIN) is 0 .
767 , a bit higher than the baseline model 0 .
752 selected using the original out-of-distribution validation set.
Data analytics and visualization.
To gain a qualitative understanding of the distribution shifts deﬁned for some of the datasets above, we visualize the domain distributions via scatter plots of embeddings learned with a Weisfeiler-Lehman (WL) subtree kernel of 3 iterations; see Appendix D for details.
We see form Fig.
2, the graphs (dots) from different domains (represented by colors) in
M OL HIV, M OL PCBA are clearly separated.
However, on PPA and G OSSIP C OP , there are still large overlaps between the distributions of graphs from different domains.
C Baseline Algorithms and Models
We implement a collection of OOD generalization algorithms, graph augmentation methods, and
GNN models and techniques as shown in Table 1.
OOD generalization algorithms.
We implement the following six 2 OOD generalization algorithms
: • Empirical Risk Minimization ( ERM, [26] ) means simultaneously minimizing the loss over all D train datasets S dD train d =1 combined, such that the objective becomes 1 D train 1 n d (cid:80) D train d =1 (cid:80) n d t =1 (cid:96) ( f ( G di ) , y di ) .
• Group Distributionally Robust Optimization ( GroupDRO , [8]) minimizes the empirical risk on the training set while more heavily weighting domains with larger errors.
Let R d =n d (cid:80) n d i (cid:96) ( f (( G di ) , y di )) be the empirical risk over a single domain.
Group Distributionally Robust Optimization (GroupDRO) for domain generalization then solves min { max d ∈ D train R d } , minimizing the worst case expected risk for all domains [8].
• Invariant Risk Minimization ( IRM , [9]) seeks to learn a common parameterization of the linear classiﬁer ω that minimizes the empirical risks in every domain.
It seeks to learn a featurizer ω : G → H and classiﬁer φ : H → Y , composed as predictor φ ◦ ω , such that a single parametrization for φ minimizes the empirical loss in each domain, R d ( φ ◦ ω ) for all domains, φ ∈ arg min φ R d ( φ ◦ ω ) for all d ∈ D train • Correlation Alignment for Deep Domain Adaptation ( DeepCORAL , [10]) matches the mean and covariance statistics of graph representations h G = φ ( G ) (after graph pooling) from different domains.
It penalizes the difference between each domain.
Let C i and C j denote the feature covariance matrices of domain i and j , that is C
i =
[Equation]
D
[Equation]
[Equation]
[Equation]
D i )) , where n i is the number of samples and D i indicates the data example.
The CORAL loss is deﬁned as [Equation]
C i − C j (cid:107) 2 F , d is the number of dimension of data and F is Frobenius norm. • Domain-Adversarial Neural Networks: ( DANN , [11]) use an adversarial network to match the distributions of graph representations.
During the training procedure, DANN encourages the appearance of features, which are discriminative for the main learning task on the source domains and indiscriminate with respect to the transfer between the domains.
As mentioned in Appendix A, we also consider a modiﬁed version, DANN-Graph , where the featurizer φ is a permutationequivariant GNN without graph pooling, and the classiﬁer ω is a permutation-invariant GNN with graph pooling.
We investigate whether performing distribution matching on the graph embeddings before pooling leads to an increase in performance. • Meta-Learning for Domain Generalization ( MLDG , [12]) learns how to generalize across domains using the framework of MAML [32].
Meta-learning for Domain Generalization splits the D train domains datasets S d into subsets meta-train and meta-test .
During training, the loss is ﬁrst computed over the meta-train sets and a parameter update determined, and then the loss is computed over the meta-test subsets with respect to model parameters after meta-train update is applied.
This bi-level optimization is performed using gradient descent, and the ﬁnal parameters are evaluated on the true test domains.
Augmentation methods.
We implement two augmentation methods tailored for graphs
: • Adversarial Augmentation on Graphs ( FLAG , [13]) iteratively augments node features with gradient-based adversarial perturbations.
With image data, rotations and crops are standard examples of semantics preserving transformations, and these types of augmentations are not considered to be domain-speciﬁc.
However, the graphs we consider are abstracted from diverse ﬁelds, and individual nodes, edges, and features are much more semantically relevant than individual pixels in an image - i.e., nodes might represent atoms, and edges represent chemical bonds.
Thus, it is unclear what types of augmentation (in features and connective structure) work well for which types of graphs and the domain shifts they are subject to.
FLAG is one type of graph adversarial-based augmentation algorithm that happens in the node feature space.
Our experimental results show that FLAG is effective, especially when the input node features are discrete categorical features.
In this work, we follow the original paper to do three steps of adversarial training.
We only carry out a rough hyperparameter search on the step size of gradient ascent from { 0 . 01 , 0 . 001 } , which can make the results suboptimal compared with the original paper. • Structural Augmentation ( SA ) performs randomized node dropping and edge perturbation to increase the structural diversity of the graphs sampled from the training domains.
You et al.
proposes a graph contrastive learning framework (GCL) for self-supervised pre-training of GNNs based on minimizing the distance in latent space between two augmented views of the same graph according to a contrastive loss (a formulation of the loss proposed in SimCLR [33], but for graph data).
The self-supervised pre-training plus supervised ﬁne-tuning scheme they implemented for solving transfer learning tasks was the closest application to the domain generalization present in their evaluation.
Inspired by the graph contrastive learning framework (GCL) for self-supervised pre-training of GNNs proposed by You et al.
[24], we select two of the structural augmentation (SA) types that they use to generate contrastive pairs and simply apply them directly to the training data while performing ERM.
We consider Node dropping - randomly discarding a certain portion of vertices V along with their connections, and Edge perturbation - perturbing the structure of G by randomly adding or dropping a certain ratio of edges under the assumption that the graph semantics are preserved under these transformations and that a model should be robust to variance in the edge connectivity.
The choice described in section Appendix C to omit this contrastive pre-training in the main evaluation is motivated by empirical results suggesting that SA with ERM alone is competitive and that GCL does not provide a performance beneﬁt in the domain generalization setting compared to the added computational cost of processing 2 N augmented graphs per epoch of pre-training.
GNN models and techniques.
We implement eight GNN models and techniques reﬂecting a variety of architectural features and inductive biases:
• Standard GNNs: Graph Isomorphism Network ( GIN , [14]) and a 10-layer version GIN-Deep , along with Multi-Layer Perceptron ( MLP ) as the ablation setup without structure learning. • Global-Context Techniques: Virtual Node [15, 34] introduces a “virtual node” that is connected to all the nodes in the graph, which helps to learn correlations at a distance and improves the complexity of graph aggregations. • Spectral Convolution Methods: ChebNet [16] applies spectral ﬁltering in the Fourier domain representations of node features.
• Weisfeiler-Lehman (WL) Expressive Power Hierarchy: 3WL-GNN [17] enjoys guaranteed 3-WL expressiveness, which is strictly stronger than message passing GNNs.
• Structure-Aware GNNs: GSN [18] counts graph substructures isomorphic to some small query graph and enhances the expressive power of message passing GNNs.
D More Experimental Details D.1 Embedding Visualization
We provide some visualization results on the OOD generalization behavior of the DeepCORAL,
DANN, and DANN-G algorithms.
The visualization shows the clustering behavior of the ﬁnal predictions with respect to the predictive labels on the test set.
Results are shown in Fig.3.
We can see that DANN and DANN-G have better performances than DeepCORAL.
[Figure]
D.2 Domain Distributions Visualization of G OSSIP C OP
ChebNet is robust to the shifts in graph sizes on G OSSIP C OP .
Considering scatter plots of the
G OSSIP C OP domain distributions like in Appendix B, but altered where the embeddings are extracted from the graph Laplacian spectrum instead of the WL sub-tree kernel, the comparison between Fig.and Fig.
2d, we see that domain shifts involving graph size may be easier learned in the Fourier representation than the WL kernel space.
[Figure]
D.3 Loss Curves of Virtual Node with ERM on G OSSIP C OP
Virtual node fails on G OSSIP C OP where it must generalize to test domains consisting of larger graphs.
The reported performance 0 .
499 ± 0 .
078 AUC is no better than random guessing ( 0 .
5 AUC); see the learning curves below in Fig.
[Figure]
D.4 Hyperparameter Search
We search from the following hyperparameter sets for different algorithms.
For each algorithm, we choose at most one algorithm-speciﬁc hyperparameter to tune:
• IRM: the weight for IRM penalty loss is chosen from { 1 . 0 , 100 . 0 } , IRM penalty anneals per 500 iterations.
• MLDG: the hyper-parameter beta is chosen from { 0 . 1 , 1 . 0 , 10 . 0 } .
• FLAG: the inner gradient ascent step size for FLAG is chosen from { 0 . 01 , 0 . 001 } .
• SA: the data augmentation ratio for SA is chosen from { 0 . 1 , 0 . 2 , 0 . 3 , 0 . 4 } , the type of data augmentations are randomly chosen from node drop and edge permutation.
• DeepCORAL: Coral penalty weight is chosen from { 1 . 0 , 10 . 0 } .
• DANN: the DANN lambda value is chosen from { 0 . 1 , 1 . 0 , 10 . 0 } .
• DANN-G: similarly, the DANN lambda is chosen from { 0 . 1 , 1 . 0 , 10 . 0 } .
• GSN: the type of subgraph are cycles, and the maximum substructure size is 6.
Tables 8 to 10 present the results for different types of algorithms across different hyper-parameters.
We select the top-performing hyper-parameters from Tables 8 to 10.
We summarize all the important hyper-parameter selections in Table 11.
D.5 Other experiment details
For all the implementations, we use the Adam optimizer, with a weight decay of 0.
We also show the batch size, the number of epochs, learning rate, the number of groups for each batch for every dataset in Table 12.
We use GNN base model GIN with ﬁve convolutional layers, dropout of 0.5, and the
RELU activation function.
The dimension of the hidden layer is 300.
[Table]
[Table] [Table]
0.684 ColoredMNIST 0.
[Table]
[Table] [Table]