GNN D OMAIN A DAPTATION USING O PTIMAL T RANSPORT
Anonymous authors
Paper under double-blind review
A BSTRACT
While Graph Convolutional Networks (GCNs) have recently grown in popularity due to their excellent performance on graph data, their performance under domain shift has not been studied extensively.
In this work, we first explore the ability of
GCNs to generalize to out-of-distribution data using contextual stochastic block models (CSBMs) on the node classification task.
Our results in this area provide the first generalization criteria for GCNs on feature distribution and structure changes.
Next we examine a popular Unsupervised Domain Adaptation (UDA) covariate shift assumption and demonstrate that it rarely holds for graph data.
Motivated by these results, we propose addressing bias in graph models using domain adaptation with optimal transport - GDOT , which features a transportation plan that minimizes the cost of the joint feature and estimated label distribution
P ( X, ˆ Y ) between source and target domains.
Additionally, we demonstrate that such transportation cost metric serves as a good proxy for estimating transferability between source and target graphs, and is better as a transferability metric than other common metrics like maximum mean discrepancy (MMD).
In our controlled
CSBM experiments, GDOT demonstrates robustness towards distributional shift, resulting in 90% ROC AUC (vs. the second-best algorithm achieving < 80 % on feature shift).
Comprehensive experiments on both semi-supervised and supervised real-world node classification problems show that our method is the only one that performs consistently better than baseline GNNs in the cross-domain adaptation setting.
I NTRODUCTION
While Graph Neural Networks (GNNs) (Kipf & Welling, 2017; Velickovic et al., 2017; Hamilton et al., 2017; Chami et al., 2022) had great success in node classification on i.i.d.
data, they are susceptible to deterioration of performance under data distribution shift, where the data used for training and inference come from different distributions (Koh et al., 2021).
Consequently, the behavior of GNNs under distribution shift has become a focus of several recent works, which show that a distribution shift in both graph structure and node features can result in deterioration of GNN performance (Zhu et al., 2021;
Ma et al., 2021a; Wu et al., 2020).
Moreover, Baranwal et al.
(2021) investigated the connection between the out-of-distribution generalization and graph structure change.
They used a graph convolution on a simplified data-generation model called a contextual stochastic block model (CSBM)
(Deshpande et al., 2018).
In this paper, we extend their findings to both graph structural and feature changes in test data.
Our results show when the generalization of a single-layer graph convolution network would be worse than that of a linear classifier.
At the same time, the behavior of classical (i.e. not GNNs) machine learning models under domain shift has been studied extensively.
Common approaches for Unsupervised Domain Adaptation (UDA), where unlabeled target/inference data is available, include learning Domain Invariant Representation
Learning (DIRL)
(Ben-David et al., 2010; Ganin et al., 2016) and Invariant Risk Minimization (IRM) (Arjovsky et al., 2019).
DIRL attempts to align features of source and target data, and IRM assumes that source and target differ in spurious correlations but share the same causal mechanism.
However, the effectiveness of such UDA algorithms on GNNs remains unclear.
Additionally, these approaches require either covariate shift or multiple training environments which are not justified on structured data.
Recently, a new line of work pioneered by Courty et al.
(2017) started to emerge.
The idea is to assume that there is a non-linear transformation from joint feature space in the source domain onto a target domain.
This transformation is not learned directly and instead an optimal transport algorithm optimizes for the best function.
This algorithm does not rely on covariate shift assumption as long as a transportation plan exists.
We explore a UDA setting on CSBM and find that the covariate shift condition ( P s ( Y | X ) =
P t ( Y | X ) ) rarely holds.
Our conclusions confirm those from a recent study by Zhao et al.
(2019), which states that variants of domain invariant representation learning cannot adapt to target graphs with innate deficiency (see also Section 3.2).
Therefore, we come up with an efficient UDA for graph data using optimal transport, resulting in improvements over the DIRL methods.
We prove that such a transportation plan exists between the source and target CSBM graphs and implement an efficient subgraph sampling ( e.g. GraphSAINT (Zeng et al., 2019)) to enable minibatch computation of optimal transport.
Further, we demonstrate that the transportation cost C is a metric that can be used to reason about the transferability between source and target graph data, and is much more indicative than metrics such as MMD (Long et al., 2015) and CMD (Zellinger et al., 2017) that are used successfully for transferability estimation (Ibrahim et al., 2021).
Under both the supervised and semi-supervised distribution shift, our method outperforms existing domain adaptation algorithms designed for Neural Networks and Graph Neural Networks by an average of 1.5% accuracy over the second best approach on 7 datasets.
As far as we know, our approach is the first domain adaptation algorithm in graph learning that does not assume targeted distribution shift, e.g. covariate shift.
Our contribution could be summarized as:
• Based on CSBM, we provide insights on the limits of OOD generalization of GNNs for both feature and structural changes.
We find that the popular covariate shift assumption rarely holds, thus explaining the commonly observed poor performance of DIRL methods.
• We come up with an efficient mini-batch algorithm GDOT for optimal transport-based domain adaptation, and prove that the optimal mapping exists on CSBM.
In addition, we demonstrate that such transportation cost is a superior transferability estimation metric for graph data.
• On synthetic and real datasets, GDOT is able to successfully mitigate the domain shift.
R ELATED W ORK
Unsupervised Domain Adaptation is concerned with situations when training and testing data are drawn from two different distributions, and the goal of UDA algorithms is to transfer knowledge from the source onto target data, obtaining good generalization on target distribution.
In the theoretical foundational work of domain adaptation, Ben-David et al.
(2010) presented an upper bound of target risk using domain discrepancy measure called H -divergence that represents the distance between source and target distributions.
Since then, various domain adaptation algorithms that minimize some definitions of such distance in the latent space were proposed.
To achieve domain invariant representation learning, an additional adversarial head is introduced in Ganin et al.
(2016), with the goal of distinguishing source and target samples in the latent space.
Conditional DANN workCDAN (Long et al., 2018) - incorporates classifier predictions into the adversarial head, either via linear or multilinear conditioning, further improving UDA performance.
There are also methods that seek to optimize some predefined discrepancy measure, which usually exhibits more stable performance and have fewer hyperparameters to tune.
For example, Maximum Mean Discrepancy (MMD) (Gretton et al., 2012; Long et al., 2015) measures the difference of distribution means in
Hilbert kernel space.
More recently, central moment discrepancy metric CMD (Zellinger et al., 2017)
attempts to match higher-order means in non-kernel space instead.
CMD is shown to be less sensitive (than MMD) to the weight with which such regularization is added to the loss.
However, a lot of DIRL algorithms assume a covariate shift: the feature distributions of source and target data are different, but the conditional distribution P ( Y | X ) is the same between the source and target data.
A recent study shows a provable bad generalization of these methods when conditional probability P ( Y | X ) varies across domains (Zhao et al., 2019).
Optimal transportation OT problem in the literature has been used to compute distances between two distributions, e.g. earth mover distance.
Courty et al.
(2017) applied OT to the UDA setting, learning an implicit mapping between source and target samples on joint feature and label distribution.
Instead of solving an optimal transportation plan, the label matching (Le et al., 2021) domain adaptation algorithm aims to mitigate the label shift when
DIRL fails via optimizing the corresponding Wasserstein metric through discriminator training.
Out-of-Distribution Graph Data.
Recently, Gui et al.
(2022) studied the out-of-distribution data challenges in graph learning, which arise from various domain shifts such as graph size, molecular scaffolds etc. .
To address the OOD generalization on graph data, early graph domain adaptation (Zhang et al., 2019) algorithms adopt DIRL approaches.
UDAGCN (Wu et al., 2020), on top of using DIRL, further enforces global and local consistency and extracts cross-domain node embeddings.
Zhu et al.
(2021) proposed to improve the OOD generalization on semi-supervised node classification by instance weighting and minimizing CMD metric between source and target node representations.
Zhang et al.
(2021) tried to capture environment-invariant node properties and explicitly balance the multiple environments to generalize well under distribution shift.
Built upon the DIRL work, Wu et al.
(2022) tried to address the OOD generalization problem by minimizing the mean and variance of risks from multiple training environments that are generated by the environment generators.
In our work, we forego DIRL and explore OT based solutions to graph distribution data shift.
G RAPH C ONVOLUTION R ISK ON D OMAIN A DAPTATION
Below we consider a node classification setup and describe CSBM (Deshpande et al., 2018) used in our analysis.
Definition 3.1 (Contexual Stochastic Block Model (CSBM)) .
CSBM graph is defined as a tuple (A,
X, Y), where A is node adjacency matrix, X describes nodes features and Y defines the nodes labels.
Node labels y
i are random variables drawn from Bernoulli distribution (Ber (0 . 5) ), and entries of adjacency matrix a ij ∼ Ber ( p ) if i,j nodes belong to the same class and a ij ∼ Ber ( q ) otherwise.
Features are drawn according to X i =
y i µ + Z i √ d , y i ∈ {− 1 , 1 } ,
µ ∈ R
d is the feature mean and
Z i ∈ R
d is a gaussian random variable.
The task of node classification takes nodes features X and structure of the graph A to predict labels Y .
We denote the output of graph 1-layer convolution network as Y =
f ( H ) , where H =
D − 1 ( A + I )
Xand linear classification with only node features as Y = f
( X ) .
The classifier f is defined as follows,
[Equation]
Below we introduce the challenges of the generalization, study the problem of domain adaptation and analyze the effectiveness of existing methods.
Then Sec. 4 describes our graph domain adaptation algorithm using optimal transport.
Definition 3.2 (Unsupervised Domain Adaptation (UDA)) .
Given source labeled data { ( x si , y si ) } ni
= containing n samples drawn i.i.d.
from the source domain D S , and a set of unlabeled target data { x tj } mj =1 sampled i.i.d.
from the target domain D T , UDA aims to find a predictive function f that generalizes well on target domain, using the available labeled source and unlabeled target data.
3. OOD GENERALIZATION OF G RAPH C ONVOLUTION N ETWORKS
We define the expected binary classification error ϵ on target data D T as,
[Equation]
Then ϵ h and ϵ x denote the errors on graph convolution network and linear network, respectively.
Theorem 3.1.
Suppose we have training graph G 1 ∼ CSBM ( u, − u, p, q ) and testing graph G 2 ∼
CSBM ( u ′ , − u, p ′ , q ′ ) , D ii is the degree of node i and Φ( | · | ) is the CDF function of multivariate gaussian distribution defined by distance.
Then generalization error of an optimal classifier is given as the following form for a linear layer,
[Equation]
and for a graph convolutional layer,
[Equation]
Proof.
See Appendix A.1.
Corollary 3.1.1 (Generalization performance under structure shift) .
When homophily ratio p/q changes, the generalization of graph convolution is worse than that of a linear layer when number of nodes is fewer than 2 p + q ( p − q ) 2 .
Corollary 3.1.2 (Generalization performance under feature shift) .
Let µ ′∥ be the horizontal component of µ ′ and δ as the relative distance to the origin mean, µ ′∥ =
(1 − δ ) µ .
Then the generalization of graph convolution layer is worse than that of linear layer when,
[Equation]
While we present here the results for one-layer GNNs and linear perceptron, our results can be extended to multi-layer graph convolutions with activations, which gain popularity recently (Baranwal et al., 2022).
We leave this for future work.
3. C HALLENGES OF L EARNING D OMAIN I NVARIANT R EPRESENTATIONS ON G RAPH DATA
We begin with introducing theoretical foundations that led to the creation of various Domain Invariant
Representation Learning (DIRL) methods (Long et al., 2015; Ganin et al., 2016; Long et al., 2018).
DIRL’s key idea is to learn an encoder g : X → Z ∈ R n that minimizes the distribution discrepancy of two domains in the latent space.
Theorem 3.2 (DIRL generalization bounds) .
(Ben-David et al., 2010) g : X → { 0 , 1 } is a hypothesis function from space H , ˆ ε S ( g ) is the empirical risk of g under source domain D S and ε T ( g ) is the true risk of g on target domain.
If VC-dimension of H is d and ˆ D is the empirical distribution each containing n samples, then with the probability at least 1 − δ , ∀ g ∈ H :
[Equation]
(cid:32)(cid:114) [Equation] n [Equation]
λ ∗ = ε T ( h ∗ ) + ε S ( h ∗ ) is the joint optimal risk that can be achieved on both domains by optimal h ∗ .
A small H ∆ H -divergence (second term in the bound, which essentially represents some distance between source and target domains)
P s ( Y | X )
≈ P t ( Y | X )
P s ( Y | X ) ̸
= P t ( Y | X ) leads to invariant or indistinguishable representation between source and target.
DANN (Ganin et al., 2016) is one of the popular DIRL methods that seek to minimize this term by introducing an additional (adversarial) head f : Z → { 0 , 1 } that attempts to predict whether a sample came from the source or target domain.
This encourages the model to learn an embedding g that does not contain domain-specific features and that has similar distributions on the source and target data.
As mentioned before, DIRL methods can mitigate the distributional shift (and assuming the combined error of an ideal joint hypothesis λ ∗ is small) while achieving small ˆ ε S ( h ) when covariate shift assumption P s ( Y | X ) =
P t ( Y | X ) holds.
In graph structure data however, even subtle structural or feature changes can cause large condition shift so P s ( Y | X )
̸
= P t ( Y | X ) .
In the plot on the right, we project the node TSNE embeddings of source and target CSBM graphs.
Two different colors indicate class label, O dots are source data and X are target samples.
When the distribution shift is small and covariate shift assumption holds, DANN can separate different classes well for both source and target domains (top).
However, when there is the conditional shift, the classification accuracy on target is low because it only minimizes discrepancy between representations, and classes end up intermixed.
x ) | h, h ′ ∈ H}
G RAPH UDA WITH O PTIMAL T RANSPORT
In the previous section, we demonstrated the limitations of DIRL on graph data when covariate shift does not hold.
In Transfer Learning literature (Courty et al., 2017) proposed recently an alternative to DIRL: they aim to find a nonlinear transformation between joint (features, labels) space between the source and target domains.
To find this transformation, the optimal transport (OT) problem is formulated.
Definition 4.1 (Optimal Transport (Monge, 1781)) .
Given source and target joint probability distributions P s ( X, Y ) , P t ( X, Y ) respectively, and marginal source and target distributions over feature space X: µ s =
P s ( X ) and µ t =
P t ( X ) , the Monge problem aims to find a mapping T ∗ :
Ω → Ω that transports µ s to µ t as,
[Equation]
In the relaxed transportation problem proposed by Kantorovich (1942), one looks for a discrete transport plan (a joint probability distribution) γ ∈ P
(Ω × Ω) such that,
[Equation]
where u i and v i are pairs ( x i , y i ) from source P s and target P t distributions.
Lemma 4.1.
Let C = (cid:113)(cid:80) ij d ( u i , v j ) γ ∗ ( i, j ) and d be the 2-Wasserstein distance W 22 , then C is a metric on GCN aggregated representation h between source and target CSBM graphs.
We leave the proof in Appendix A.2.
In the proof, we transform joint distribution on ( h i , y i ) into a Gaussian Mixture Model and find the closed-form solution of 2-Wasserstein distance d ( u i , v j ) between pair ( i, j ) .
Furthermore, there exists an optimal transportation plan γ between two graphs created by CSBM due to the fact that an OT problem has a unique solution when d is a metric (Villani, 2021).
Remark.
When used as a metric, the transportation cost C quantifies the discrepancy between source and target CSBM graphs.
C
=
if and only if joint distributions on source and target are the same.
Alternative non-optimal transport metrics that are often used for such purpose in DIRL settings are discrepancy measures like MMD (Long et al., 2015) and CMD (Zellinger et al., 2017).
To demonstrate the value of transportation cost as a discrepancy metric we train a 2-layer graph convolution networks using only source samples and compute their OT discrepancy measure on target data representations.
The results are presented in Figure 3, which demonstrates a clear correlations of transportation cost and testing performance on CSBM graphs.
[Figure]
Now we formally define the learning problem of unsupervised graph domain adaptation with optimal transport and present our algorithm GDOT .
Given source labeled data { ( x si , y si ) } Mi =1 in G s and unlabeled target data { x tj } in G t , we optimize the following loss function,
[Equation]
where ˆ y i and ˆ y j are predictions on the source and target data produced by a graph neural network g respectively.
Both L S and L T are cross-entropy loss 3 .
The loss consists of classification loss on G s and optimal transport cost between source and target graphs; γ ij is the transportation plan between node i in source graph G s and j in target G t , (cid:80) ij γ ij = 1 .
The term in the parenthesis is the realization of distance function d (( x si , y si ) , ( x tj , ˆ y tj )) between source distribution P s ( X, Y ) and estimated target distribution P g t ( X, g
( X, A )) .
Our optimization procedure is similar to mini-batch training of DeepJDOT (Damodaran et al., 2018), apart from the fact that we need to perform neighborhood sampling on graph to obtain source and target subgraph samples for the input of graph neural network g .
We adopt a sub-graph based sampling method - GraphSAINT (Zeng et al., 2019) to obtain batch of nodes from source and target G sb ∼ G s , G tb ∼ G t , respectively.
We describe the training process of GDOT in Algorithm 1.
Finally, we present the domain adaptation bound using optimal transportation with 1-Wasserstein distance in Appendix A.4.
Compared with DIRL UDA algorithms that optimize the bound from
Theorem 3.2, GDOT optimizes the Wasserstein distance bound without covariate shift assumption.
The transportation cost C ≈ W 1 (ˆ P s , ˆ P g t ) corresponds to the domain discrepancy measures in DIRL.
Model Analysis.
The theoretical analysis of our work is based on graph convolutional networks on CSBM.
In practice, the choice of GNN architectures depends on specific problems.
There are two hyperparameters α, β for which we chose the values so that α (cid:80) ij γ ij ∥ x i − x j ∥ 2 and β (cid:80) ij γ ij L T are on the same or smaller scale than the source loss L S .
In each step, let M be the size of mini-batch and N be the size of features x i ∈ R N and K classes, the additional computation cost of our method in each epoch is due to computing the transportation cost matrix
C ∈ R M × M and solving the optimal transportation γ ∗ .
The cost matrix takes O ( M 2 ( N + K )) time and a traditional network of simple algorithms takes O ( M 3 log( M )) .
Therefore, the total time complexity of GDOT is O
( M 3 log( M ) + M 2 ( N + K )) .
To improve the efficiency, one can speed up the computation of optimal via Sinkhorn (Cuturi, 2013), obtaining complexity of O ( M 2 ) .
S YNTHETIC E XPERIMENTS
In this section, we seek to confirm our theoretical insights about the generalization ability and transferability of graph models.
We do this using two different families of synthetic graphs: (1)
CSBM graphs syn-csbm-pq and syn-csbm- δ with structure or feature shift, respectively (2) synthetic graphs constructed from real datasets syn-cora and syn-products proposed by Zhu et al.
Each sample in CSBM graph is composed of a training and testing graph, where the testing graph exhibits either kind of distribution shift in various degrees.
More information about the data can be found in Table 3 in the Appendix, numerical results for all of the figures are also available in the Appendix.
In this section, we compare our method GDOT with standard DIRL algorithms including CMD (Zellinger et al., 2017) and CDAN (Long et al., 2018) using graph convolution network (Kipf & Welling, 2017).
We chose these two algorithms as a point of comparison since
CMD has been shown to outperform a number of other metric-based DIRL algorithms like MMD (Zellinger et al., 2017).
Additionally, CDAN, which uses an adversarial head to impose domain irrelevance and additionally conditions on discriminative information (logits) has been shown to outperform a number of transfer learning techniques (Long et al., 2018).
5. OOD GENERALIZATION OF GCN AND MLP
In our first experiment, we illustrate how distribution shift on structure and features affects neural networks (2-layer MLP), single and double layers of graph convolution networks on a test graph.
We aim to validate the results from theorem 3.1 and show that the generalization of graph convolution networks become worse when the testing graph has more inter-class edges or different feature means.
Similar to Baranwal et al.
(2021), we set the distance of means of two classes as 2 / √ d and p/q = 5 in training graphs, where MLP cannot classify two classes as accurately as graph convolution.
In testing graphs, we keep the density of the graph unchanged ( i.e. average degree) and vary the homophily ratio p/q or feature mean deviation δ as of corollary3.1.1 and corollary3.1.2.
As shown in Figure 4,
GCNs are affected by both structure and feature shift more than MLP (that cannot separate the training data accurately when shift is large).
Calculating the thresholds from corollary 3.1.1, we get 5376 > n =
(p/q=1) and 91 .
5 < n (p/q=2) which aligns well with result in Figure 4a.
In
Figure 4b, we estimate that when δ > 0 .
67 GCN would have worse generalization than linear model (corollary 3.1.2), which is confirmed empirically by ROC_AUC plot.
In addition, we can see deeper
GCNs are more susceptible to feature shift.
Therefore improving OOD generalization is crucial to real-world applications since deeper GNNs are often needed to obtain good performance.
Overall, our experiments are consistent with our findings in theorem 3.1.
[Figure]
5. D OMAIN ADAPTATION ON CSBM
In this experiment, we compare two DIRL algorithms - CDAN and CMD with GDOT on two synthetic CSBM datasets.
We tune the hyperparameters of all three algorithms on validation data obtained from the training graph.
As depicted in Figure 5a and 5b, GDOT yields better results over both baselines under feature and structure distribution shift.
In Appendix A.3, we discuss the theoretical limitations of DIRL algorithms w.r.t.
conditional shift.
We observe that CDAN, which specifically minimizes the second term in the bound, enjoys a minor improvement over vanilla GCNs.
The method CMD does not optimize the H ∆ H -divergence directly and is seemingly less susceptible to conditional shift.
[Figure]
5. D OMAIN ADAPTATION ON EXISTING SYNTHETIC GRAPHS
In our last suite of synthetic experiments, we examine the effectiveness of optimal transport domain adaptation on non-CSBM graphs.
In the literature of low homophily GNN study (Zhu et al., 2020), syn-cora and syn-products are constructed from existing benchmarks via preferential attachment (Barabási & Albert, 1999) in order to control the homophily ratio.
In our theorem A.4, our method should work without the assumption of node features.
We aim to validate this on synthetic graphs generated from real graphs like Cora and ogbn-products.
We train GCN on the same "easy" graph with 1.0 homophily ratio on both datasets and test on target graphs with various homophily ratios ranging from 0.0 to 0.9.
On the multi-class classification task, we compare our method with the same domain adaptation baselines.
The results in Figure 6 demonstrate that similar to the previous experiments, GDOT is able to mitigate the distribution shift regardless of noise levels while other algorithms fail to improve over a standard GCN.
[Figure]
R EAL D ATA E XPERIMENTS
In this section, we will compare our UDA method with domain adaptation algorithms designed for neural networks and graph neural networks in both supervised and semi-supervised learning settings.
In each setting, we will introduce the specific domain adaptation task and how we apply our approach.
Baselines.
In addition to the domain adaptation algorithms used in previous sections, we consider the following methods for comprehensive study under distribution shift: (1) MMD (Long et al., 2015) and (2) DANN (Ganin et al., 2016).
For graph-specific methods, we focus on the task of node classification and choose three most representative methods: (1) UDAGCN (Wu et al., 2020)
couples domain adversarial learning with graph attention mechanism (2) SRGNN-IW (Zhu et al., 2021) proposes to use instance weighting technique on GNN output embeddings under covariate shift (3) Graph-EERM (Wu et al., 2022) proposes to augment training graph for invariance principles in risk minimization.
As for our own ablations, we study two variants of our method where either α or β in Equation 4 is zero indicating optimal transport based on only feature and label distribution distance.
6. S EMI - SUPERVISED N ODE C LASSIFICATION
Zhu et al.
(2021) found biased training data in semi-supervised learning can cause dramatic accuracy loss; they provide the algorithm to generate biased training nodes (refered to as OOD training in Table 1) on three semi-supervised learning benchmarks: Cora, Citeseer and PubMed (Sen et al., 2008).
In semi-supervised classification, source data is a small number of training nodes and target data are all of the remaining nodes in the same graph.
We choose the best-performed GNN architecture from their paper - APPNP (Klicpera et al., 2018) and report the Micro-F1, and Macro-F1 for each method and the accuracy loss compared with IID training data.
We are able to reproduce the performance gap between IID and OOD training data in Table 1.
We begin by noting that most of the general domain adaptation algorithms such as CMD, MMD, and DANN can help improve the performance because covariate shift holds in this problem.
Among these algorithms, we find that directly optimizing discrepancy metrics seems to be more effective and robust (smaller average loss and deviation over 100 runs) than adversarial methods (CDAN and DANN) which often require more tuning.
On three datasets, GDOT always achieves top-2 performances, and our ablation without feature distance ( α = 0 ) in pairwise distance d is usually the second to the best, which illustrates the importance of considering label distribution in OT.
[Table]
6. S UPERVISED N ODE C LASSIFICATION
In a fully-supervised setting
[Table]
transfer learning is often performed across different domains or time periods on graph structure data.
We conduct the domain adaptation experiments on citation graphs provided by ArnetMiner (Tang et al., 2008) with both types of shifts: (1) domain shift, where we adopt two pairs of ACM and DBLP graphs w.r
[Table]
t. graph sizes, (2) time shift, where we use ACM graph prior to 2010 as the source and after 2010 as target data.
The details about statistics and the graph creation process can be found in Appendix C.
[Table]
We use a 2-layer graph convolution network as the base model and adopt mini-batch training introduced in Section 4.
Specifically, we have two graph samplers on both graphs, training nodes in the source batch and all nodes in the target batch are used for solving the optimal transport.
In Table 2, the first observation is that algorithms perform differently under different settings.
For example, CMD outperforms baseline GCN on two tasks while its performance on large-scale domain transfer problems is worse.
GDOT not only performs best among all domain adaptation algorithms but also is consistently better than the baseline model without domain adaptation.
The only other algorithm with this property is SRGNNIW which does not assume the type of distribution shifts as well.
Existing graph domain adaptation algorithms - UDAGCN and EERM exhibit limited or negative transfer improvements.
C ONCLUSION
In this work we establish a theoretical analysis on the effect of graph convolution on out-of-distribution data which illustrates the necessity of domain adaptation when domain shift is large.
However, popular domain invariant representation learning algorithms assume covariate shift which is a strong assumption for graph data.
We present a novel graph domain adaptation framework based on optimal transport to remedy this.
Using a number of synthetic and real data node classification experiments, we demonstrate that our method GDOT results in a robust improvement on different kinds of domain shifts without assuming a covariate shift.
As for future work, we have two notable directions to explore: (1) extend our analysis to multi-layer and other types of graph neural networks (2) demonstrate the effectiveness of GDOT on more tasks such as full graph classification and regression.
A T HEORY DETAILS A. P ROOF OF T HEOREM 3.
In theorem 3.1, we made several simplifications on original CSBM model to investigate its OOD generalization w.r.t.
structure and feature distribution shifts.
The original CSBM ( µ, ν, p, q ) is defined to have two different class means µ and ν .
Given training and testing distribution as
G ∼ CSBM ( µ, ν, p, q ) and G ′ ∼ CSBM ( µ ′ , ν ′ , p ′ , q ′ ) , we let ν = ν ′ =
− µ in CSBM by making ⃗the middle point of original feature mean of two classes.
Without loss of generality, we let two graphs have same amount of nodes n and focus on one class ( i.e. y i = 1 ) of graph G ′ .
Theorem A.1.
Suppose we have training graph G 1 ∼ CSBM ( u, − u, p 1 , q 1 ) and testing graph
G 2 ∼ CSBM ( u ′ , − u, p 2 , q 2 ) , Φ( |·| ) is the CDF function of multivariate gaussian distribution defined by distance.
The generalization error of an optimal classifier is for linear layer,
[Equation]
for graph convolution layer,
[Equation]
Proposition 1.
Through training with hinge loss, the linear model and the linearized graph neural network have the same optimal hyperplane P = { x | w T x + b = 0 } characterized by w ∗ = µ and b ∗ = 0 .
Proof.
In the following, we discuss the classification performances of original and convoluted feature from train graph G .
The training data distribution of two models are,
[Equation]
We restate the generalization error as expected error indicated by linear classifier
[Equation]
The CDF of the standard normal distribution is denoted by the Φ function.
When d = 1 (i.e.
1-dimension case), if we translate the distribution into a standard gaussian N (0 , I ) by moving µ , the classification error regarding the optimal hyperplane { w ∗ , 0 } is the cumulative probability is ϵ x = 1 − Φ( u ′ ) .
[Equation]
In standard multivariate (d > 1) distribution, we define the CDF as a monotonic function regarding the distance to w ∗ and expected classification error is ϵ x = 1 − Φ( | x · µ | ) .
We rescale the gaussian distribution output by graph convolution and the expected error ϵ x and ϵ h become comparable between their means.
[Equation]
On training graph, graph convolution has better linear separability when ϵ h < ϵ
x ( i.e. √ D ii · p − q p + q >due to monotonicity of Φ ).
Note that previous work Baranwal et al.
; Ma et al.
(2021b) also result in similar conclusion.
On testing graph, we assume only one class centroid moves or graph structure (p,q) changes,
[Equation]
Similarly, the error of graph convolutional and linear layer are ϵ h ′
= 1 − Φ( (cid:112) D ′ ii p ′ µ ′ − q ′ µ p ′ + q ′ · µ ) and ϵ x ′ = 1 − Φ( µ ′ · µ ) , which completes the proof.
Now let’s discuss the relative error of a linear layer and graph convolution layer when structure or feature deviates from training, respectively.
When graph structure (p/q) changes on testing graph, the ood generalization is same as on training graph with different p ′ and q ′ .
The degree of node D ii can be estimated as D ′ ii ≈ n ( p ′ + q ′ ).
By solving (cid:112) D ′ ii · p ′ − q ′ p ′ + q ′ < 1 , we have GCN yields larger classification error when n < 2 p ′ + q ′
( p ′ − q ′ ) 2 as of corollary 3.1.1.
When feature mean of the class has shift ( µ ′ in ϵ h ′ and ϵ x ′ ), only horizontal component µ ′∥ of µ ′ = µ ′∥ + µ ′⊥ affects the error ( i.e. cos( µ ′∥ , µ ) = 1 ).
Let µ ′∥
=
(1 − δ ) µ , we consider the following condition of GCN underpeforms linear,
[Equation]
By solving the above equation, we complete the discussion of corollary 3.1.2.
A. P ROOF OF L EMMA 4.
In theorem 3.1, we prove the output of graph convolution for one class is a Gaussian distribution.
We first describes the probability density function of K-class CSBM with k Gaussian distribution ν 1 , ν 2 , ..., ν k and probability vector p 1 , p 2 , ..., p k is drawn from a multinoulli distribution.
[Equation]
Suppose we have u s ∼ P ( ν s , p s ) and v t ∼ P ( ν t , p t ) for source and target graph, a discrete optimal transport problem solves,
[Equation]
Lemma A.2.
Let C = (cid:113)(cid:80) ij d ( u i , v j ) γ ∗ ( i, j ) and d be the 2-wasserstein distance W 22 , then C is a metric on GCN aggregated representation h between source and target CSBM graphs.
Proof.
One important property of CSBM is p 1 , p 2 , ..., p k is from a multinoulli (categorical) distribution, such that the 2-wasserstein distance d ( u i , v j ) is computed on two gaussian distributions.
Assume ν sa ∼ N ( m 0 , Σ 0 ) and ν tb ∼ N ( m 1 , Σ 1 ) are the two gaussian distribution with p sa = 1 and p ta = 1 , we have their 2-wasserstein distance Dowson & Landau (1982) as,
[Equation]
Apparently positivity and symmetry holds on C : C ≥ 0 and C ( u s , v t ) =
C ( v t , u s ) .
It is also obvious that C
=
if and only if d ( u i , v j ) = 0 ∀
i, j , in other words, source and target distribution is identical.
We denote u, u ′ , u ′′ from three different CSBM graphs and p, p ′ , p ′′ as associnated probability vector and prove the triangle inequality of C,
[Equation]
Let γ 01 and γ 12 be the solution of Equation 9 between u, u ′ and u ′ , u ′′ , respectively.
We construct a custom transportation plan γ 02 ∈ Π( u, u ′′ ) as,
[Equation]
We have marginal distribution of γ 02 along i as p ′′ k ,
[Equation]
and similarly (cid:80) k γ 02 ( i, k ) = p i .
Now we start consider the C ( u, u ′′ ) ,
[Equation]
The first inequality is the definition of optimal transport.
The second inequality is because wasserstein distance is a metric and the last inequality is Minkowski inequality.
Now, we have shown the tranportation cost between pair of gaussian mixture distributions from K-class CSBM graphs satisfies all the condition of being a metric.
In our paper, we mainly use 2-class CSBM as a specific case of the lemma.
A. A DDITIONAL DISCUSSION ON DIRL
Theorem A.3 (Limits of learning invariant representations under conditional shift) .
Zhao et al.
Suppose
markov chain
X g −→ Z
h −→ ˆ Y and d JS is the Jensen-Shannon distance,
[Equation]
When P ( Y | X ) is different on source and target, minimizing source risk and H ∆ H -divergence leads to a small JS distance
d JS ( D Z S , D Z T ) .
As a consequence, the marginal label shift d JS ( D Y S , D Y T ) dominating the the lower bound of joint source and target risk.
If convariate shift does not hold, DIRL cannot achieve accurate predictions on target.
A. D OMAIN A DAPTATION B OUND ON O PTIMAL T RANSPORT
We restate the existing bound on target error for optimal transport based domain adaptation with 1-wasserstein distance.
Theorem A.4.
Courty et al.
Suppose g ∈ H is the graph neural network in hypothesis space, the optimal g ∗ is a Lipschitz function with ϕ -probabilistic transfer lipschitzness (PTL) 4 .
Let { γ ∗ | γ ∈ Π( µ s , µ t ) } be the optimal mapping in Equation 4 and W 1 (ˆ P s , ˆ P g t ) corresponds the 1wasserstein distance between two induced empirical distributions from n samples.
If we assume | g ∗ ( x 1 ) − g ∗ ( x 2 ) | <
M, ∀ ( x 1 , x 2 ) and c ′ is the concentration factor of wasserstein distance Bolley et al.
(2007), with the probability at least 1 − δ, ∀ g ∈ H ,
[Equation]
By optimizing GDOT , the optimal transportation cost C approximate the 1-wasserstein distance
W 1 (ˆ P s , ˆ P g t ) in the bound.
Compared with DIRL bound in theorem 3.2, both bound contain the joint optimal risk λ ∗ and the differences are on the assumptions.
DIRL assume covariate shift while optimal transport assume the existence of transportation plan γ ∗
B M ODEL D ETAILS B. I MPLEMENTATIONS
We implement our method and all other baselines using torch-geometric library.
We list the graph neural network specifications used in our experiments,
Synthetic node classification - model architecture: Graph Convolutional Networks Kipf & Welling (2017), hidden dimension: 16, activation: SiLU, number of layers: 2, dropout: 0. 2.
Semi-supervised node classification - model architecture: APPNP Klicpera et al.
(2018), hidden dimension: 32, number of layers:2, dropout: 0.0, 3.
Supervised node classification - model architecture: Graph Convolutional Networks Kipf & Welling (2017), hidden dimension: 128, activation: ReLU, number of layers: 2, dropout: 0.
In GDOT , we use the RandomWalk GraphSAINT Zeng et al.
(2019) sampler and set batch size as 256, step size as 50 and walk length as 2.
We indepdentently run experiments 10 times and report the mean and standard deviation in all table and figures.
The code for each experiment can be found in separate folder in supplementary materials.
B. B ASELINE H YPERPARAMETERS
In our experiments, we use the following baselines with hyperparameters tuning on validation.
Specifically, each baseline has hyperparameters as follow,
For MMD, α ∈ { 0 . 01 , 0 . 1 , 0 . 5 , 1 } controls the weight of regularization.
For CMD, k ∈ { 1 , 3 , 5 , 7 , 10 } determines the number of central moment.
α ∈ { 0 . 01 , 0 . 1 , 0 .
5 , 1 } controls the weight of regularization.
For DANN, α is set in { 0 . 1 , 0 . 5 , 1 } for reverse gradients in backward pass.
β ∈ { 0 . 01 , 0 . 1 , 0 .
5 , 1 } controls the weight of regularization.
For CDAN, λ is a hyper-parameter between source classifier and conditional domain discriminator.
lo ∈ { 0 . 01 , 0 . 1 , 1 } and hi ∈ { 0 . 1 , 1 , 2 } are the initial value and final value of λ .
β ∈ { 0 . 01 , 0 . 1 , 0 .
5 , 1 } controls the weight of regularization.
For UDAGCN, the balance parameters γ 1 and γ 2 are adjusted carefully in the searching space { 0 . 1 , 0 . 3 , 0 . 5 , 0 .
7 , 1 . 0 } , respectively.
The adaptation rate λ is the following schedule: λ = min( 2 1+exp( − 10 p ) − 1 , 0 . 1) , and the p is changing from 0 to 1 within the training process as Wu et al.
For EERM, we search the best learning rate α f ∈ { 0 . 0001 , 0 . 0002 , 0 . 001 , 0 .
005 , 0 . 01 } for GNN backbone, the learning rate α g ∈ { 0 . 0001 , 0 .
01 } for graph editers, the weight β ∈ { 0 . 2 , 0 .
5 , 1 . 0 , 2 . 0 , 3 . 0 } for combination, the number of edge editing for each node s ∈ { 1 , 5 , 10 } , the number of iterations T ∈ { 1 , 5 } for inner update before one-step outer update.
For SRGNN-IW † , the main hyper parameters in the sampler PPR-S are α ∈ { 0 . 01 , 0 . 1 , 0 .
5 , 1 } , γ ∈ { 10 , 50 , 100 , 200 , 500 } .
When the graph is large, ϵ = 0 .
001 is set in the local algorithm for sparse PPR approximation.
λ ∈ { 0 . 1 , 0 .
5 , 1 , 2 } is the penalty parameter for the discrepancy regularizer.
The lower bound for the instance weight B l is in { 0 . 1 , 0 . 2 , 0 . 5 , 1 . 0 } .
Hyperparameters of GDOT α and β are selected between { 0 . 01 , 0 . 1 , 1 } .
[Table] [Table]
C E XPERIMENT D ETAILS C. DBLP-ACM DATASET
We conduct the transfer learning experiments in domain shift and time shift.
These experiments use three sets of citation networks, which are constructed on the datasets provided by ArnetMiner Tang et al.
Specifically, for domain shift, we adopt two sets of ACM-DBLP citation networks of different sizes.
The small set namely ACM-DBLP small is proposed by Wu et al.
It includes the papers extracted from ACMv9 (between years 2000 and 2010) and DBLPv8 (after year 2010).
The large set, ACM-DBLP large is constructed on DBLPv12 (before 2017) and ACMv8 (before 2017).
As to time shift, we utilize ACMv9 across different time periods, specifically, before or after 2010, to build two citation networks, ACM time .
In our experiments, we consider these datasets as undirected graphs and each edge representing a citation relation between two papers.
The papers are classified to some of the predefined categories according to its research topics.
ACM-DBLP small has six categories including“Database”, “Data mining”, “Artificial intelligent”, “Computer vision”, “Information Security” and "High Performance Computing".
For ACM-DBLP large and ACM time , there are five categories including “Database”, “Data mining”, “Artificial intelligent”, “Computer vision”, and “Natural Language Processing".
We evaluate our proposed methods by conducting multi-label classification on these three sets of citation networks.
The dataset statistics are shown in Table 3.
C. CSBM DATASET GENERATION
In this section, we describe the generation process of syn-csbm-pq and syn-csbm- δ .
According to the definition of CSBM 3.1, we fix the size and average degree of graph ( i.e. 128 and 10).
For structure shift, each time we sample a feature mean µ , generate source graph with fixed p/q =and sample a p/q between { 1 , ..., 10 } to generate the target graph.
Such that we ensure the feature of both graph are generated with same gaussian distribution and their homophily ratios are different.
For feature shift, we generate µ ′ by scale µ by 1 − δ and rotate µ ′ from 0 to 60 degrees.
In corollary 3.1.2, we use the same δ to describe the classification error.
When δ is small, feature shift is small and test feature mean µ ′ is close to original feature mean.
The rotation is added to avoid trival adaptation like translation.
The dataset generation code can be found in uploaded code named cSBM_gendata.py.
C. M ORE RESULTS ON OOD GENERALIZATION
In section 5.1, we show the testing ROC AUC between MLP and graph convolution networks in
Figure 4.
Here, we provide its numerical results in Table 4 and Table 5.
Besides, we provide the testing loss of these methods w.r.t.
structure and feature shifts between pair of CSBM graphs.
[Figure] [Figure]
[Table]
C. D OMAIN ADAPTATION ON SYNTHETIC DATASET
In Figure we provide the test logloss plot and numerical results of Figure 5b and 5b.
[Figure]
[Table] [Table]
[Table]