Shift-Robust Node Classification via Graph Clustering Co-training
Abstract
It is widely known that machine learning models only achieve sub-optimal performance when testing data exhibit distribution shift against training i.e. ,
Pr train ( X, Y ) ̸
=
Pr test ( X, Y ) .
Although Graph Neural Networks (GNNs) have become de facto models for semi-supervised learning tasks, they suffer even more from distribution shift because multiple types of shifts origin from not only node features but graph structures.
Existing domain adaptation methods only work for specific type of shifts.
In response, we propose S hift- R obust N ode C lassification (SRNC) - a unified domain adaptation framework for different kinds of distribution shifts on graph.
Specifically, we co-train an unsupervised cluster GNN, which captures the data distribution by graph homophily on target graph.
Then a shift-robust classifier is optimized on training graph and pseudo samples from target graph, which are provided by cluster GNN.
Compared to the existing domain adaptation algorithms on graph, our approach works for both open-set and close-set shifts with convergence guarantees.
In our experiments, the classification accuracy is improved at least 3% against the second-best baseline under open-set shifts.
On time-evolving graph with close-set shift, existing domain adaption algorithms can barely improve the generalization if not worse.
SRNC is still able to mitigate the negative effect ( > 2% absolute improvements) of the shift across different testing-times.
Introduction
Graph Neural Networks (GNNs) [11, 27, 7] have achieved enormous success for node classification, these models experience performance drops if training and testing data are not identically and independently distributed ( i.i.d. ) - distribution shift [13].
In real-world GNN applications, the robustness towards distribution shift has become a pressing topic [39, 18, 32] when distribution shift between structures and features can both deteriorate the performance.
In this work, we consider two different kinds of shifts according to whether label space Y is changed or not - (1) open-set shift: there are emerging new classes in the graph ( e.g. , new COVID variant in community transmission); (2) close-set shift: time-augmented test graph has substantially different class distributions ( e.g. , time-evolving Twitter and Academia graphs).
Despite the popularity and importance of GNNs, only a few domain adaption algorithms are proposed to make GNNs robust to distribution shifts.
Most of them are designed to account for a specific type of distribution shift.
For example, open-set classification [2, 33, 17] can recognize objects of unseen classes during testing.
SRGNN [39] and UDA-GCN [32] are two representative methods adopted from invariant representation learning [15, 16, 35] for close-set shifts.
Nevertheless both methods works for only one kind of close-set shift - covariate shift where class-conditional distributions of hidden representations are same during training and testing.
Robust graph neural networks against distribution shifts are in great need, but there is still no go-to method regarding different kinds of shifts.
[Figure]
Being aware of the aforementioned challenge, we are wondering whether it is possible to design a domain adaption framework with the least amount of assumption of distribution shifts.
To the light of this issue, graph structure enjoys numerous nice topological properties, while none of the existing work incorporate these properties to mitigate the limitation of prior knowledge on distribution shifts.
As a consequence, these algorithms present even worse result than base model without domain adaptation (see results of domain-invariant baselines in Table 1 for more details).
In this paper, we propose a unified domain adaption framework for shift-robust node classification, where unlabeled target (testing) graph is used to facilitate model generalization.
First, we consider the joint probability on node and label Pr( h i , y i ) on graph, where h is the output embedding from a graph neural network.
The modeling of joint probability avoids tackling different assumptions on different marginal distributions
Pr( y i |
h i ) one by one.
Figure 1 gives a toy example that both open-set and close-set can be interpreted in one joint form.
Second, we observe similar degrees of graph homophily [38] ( e.g. , neighbors with same labels) between homophilic source and target graphs.
In other words, most real-world applications ( e.g. time-evolving social networks) would not expect target graph of weak homophily while source graph with strong homophily.
Based on graph homophily , we propose to use graph clustering to identify latent classes [3, 26] by breaking less frequent edges (possibly heterophily) between potential clusters (right side of Figure 1).
If such graph clustering can capture the joint probability on source graph, it would probably generalize well on target graph with comparable degree of homophily.
Therefore, our proposed model, which we call SRNC (Section 4.1), co-trains two graph neural networks (GNNs): an shift-robust classification
GNN Θ for unified domain adaption and a clustering GNN Φ (Section 4.2) for homophily-based clustering - (a)
The clustering memberships inferred by the clustering GNN on source data Q Φ ( C s | X s , A s ) should be close to training conditional distribution Pr train ( Y s | X s , A s ) .
For instance, on the ride of
Figure 2, the KL-divergence between above two probabilities are minimized to push the clustering result consistent with training data; (b) In the other direction, the classifier is optimized on both training graph and target graph for domain adaptation.
The targets samples are sampled from clustering GNN to improve the generalization on target graph.
In SRNC optimization, the two modules improve each other through variational training.
Convergence of the iterative optimization between two modules is theoretically justified in Section 4.3.
Our experiments illustrate that the shift-robust classifier trained in proposed way can detect more than 70% open-set samples on three widely used benchmarks.
For close-set shift, we use early snapshot (prior to 2011) of ogb-arxiv [9] network as training graph and testing on more recent graphs (14-16/1618/18-20).
SRNC is the only method that can consistently reduce the negative effect (outperform base model) of shift in our evaluation.
Meanwhile we show progressively utilizing classifier’s prediction as pseudo labels (our ablation without clustering component) on target graph [33, 17] is much worse than proposed method.
To summarize, we propose the first unified domain adaption GNN framework for multiple types of distribution shift.
We implement the framework by co-training a shift-robust classifier with a variational cluster.
The latter one captures target data distribution and approximates the source classconditional distribution simultaneously.
Extensive experiments show that our method is significantly better out-of-distribution generalization on two different distribution shifts.
Additional case study reveals its potential to discover new classes and improve label space design.
RELATED WORK
Out-of-Distribution Generalization.
In literature, the training and testing distribution is often referred as in-distribution and out-of-distribution when there is a distribution shift.
Many research topics are invented to improve the out-of-distributitaon (OOD) generalization, of which two most well-known ones are domain adaptation and domain generalization.
Domain generalization [30] requires to train a robust and generalizable predictors on unseen domain assuming training data (graphs) from multiple domains are available.
Domain adaption [1] requires only one source domain but the access of the unlabeled target (testing) data.
In terms of node classification, the assumption of domain adaption generally holds because it only requires one training graph and usually whole testing graph is available.
In this paper, we focus our discussion on domain adaptation.
Close-set Distribution Shift.
There has been a recent surge on discussing GNNs generalization [6, 29, 18, 30].
The generalization and stability of GCNs is found to be related with graph filters [29].
The H -divergence [1] and discrepancy measures [19] are invented to measure the generalization bound between source and target domains.
To bridge the gap between source and target, Domain
Invariant Representation (DIR) Learning aims to minimize these discrepancy measures on source and (unlabeled) target data by adversarial learning [5] or regularization ( e.g. CMD [35], MMD [15, 16]).
In graph neural networks [30], DIR is frequently used for domain adaptation.
For example,
DANE [37] adds a domain classifier on top of GNN outputs.
UDA-GCN [32] further proposes a dual-gnn architecture and inter-graph attention to obtain unified representations across source and target graphs.
For covariate shift, SRGNN [39] adopts two shift-robust techniques - CMD and importance sampling for non-IID training data.
SOGA [20] assumes a well-trained GNN model without the access of source graph and enforces maximal mutual information and structural proximity on target.
Besides, generative algorithm DGDA [4] introduces variational graph auto-encoders to recover and disentangle domain and semantic latent variables for adaptation.
We also use unlabeled target data in our framework, but we propose a clustering component to generate pseudo samples specifically for graph structured data.
Open-set Distribution Shift.
Early open set recognition and classification [23, 2] require classifiers to detect unknown objects during testing on image domain.
These studies [8] focus on addressing the overconfidence issue of deep neural networks for unknown classes, such as OpenMax [2] and
DOC [25].
On graph structured data, most of the GNNs work under semi-supervised learning.
Hence, over confidence issue is less likely to occur.
Instead, the topological structure poses challenges for open-set classification.
To this end, OpenWGL [33] employs an uncertainty loss in the form of graph reconstruction loss [12] on unlabeled data.
PGL [17] address the conditional shifts by episodic pseudo labels and domain adversarial regularization with graph neural networks.
Yet, none of open-set learning on graph has explored modeling more than one unseen classes distribution on target graph as ours.
Problem Definition & Preliminary
G = { V, A, E, X } is defined as a graph with nodes V , their features ( X ∈ R | V |×| F | ) and edges E between nodes ( e.g. , adjacency matrix A, A ∈ R | V |×| V | ).
Each GNN layer takes node embeddings H and adjacency matrix A as input, aggregates the neighborhood information and outputs representation
H k .
For example, in GCN [11],
[Equation]
(1) where H
=
X , ˆ A =
A + I
and ˆ D ii =
(cid:80)
j ˆ A ij .
Given a source graph G s and the output of a k-layer graph neural network is h k .
Semisupervised GNNs perform node classification via a cross-entropy loss over labeled source data
D s = { ( h 1 , y 1 ) , ...
( h n , y n ) } .
L = |D s | n (cid:88) i =
N
[Equation]
j
[Equation]
[Equation] [Equation] [Equation]
Classifier
Cluster
[Figure]
D s log& " (( ! |ℎ !
ℎ #" ℎ #$
GNN " !
G s
G t # '( # ') !
# (⋅ |% $% ) ! !
|%" ! !
|%& ! !
B KL(& " ⋅ ℎ !
# ||0 $ (⋅|1 %# )) !
%|!" !
# (⋅ |% &' ) $%&& [Figure]
(⋅|* "# ) log& " (3 , |ℎ ,- )
[Figure]
⋅|
[Figure]
Definition 3.1 (Distribution Shift in Node Classification) .
Assume node representations H t = { h t 1 , ..., h tm } are output from graph neural network on target graph G t with m nodes.
The label space on source and target graph are Y s and Y t , respectively.
Distribution shift occurs if Pr train ( H s , Y s ) ̸ =
Pr test ( H t , Y t ) 1 .
In general, there are two basic types of distribution shifts: (1) open-set shift - new classes arise during test-time, i.e. Y s ⊂ Y t .
(2) close-set shift - Y s =
Y t and joint probability changes between training and testing, Pr train ( H s , Y ) ̸
=
Pr test ( H t , Y ) .
Definition 3.2 (graph homophily ratio [38]) .
The graph homophily ratio τ h
= { ( u,v ):(
u,v ) ∈ E
∧
y u = y v } | E | is the fraction of edges in a graph which connect nodes with the same class label.
Recent study [38] has shown graph homophily is the main reason of the superior performance of graph neural networks on node classification.
Motivated by this phenomenon, we design our shift-robust classifier for domain adaptation on homophilic graphs.
When the graph homophily ratio is low, most of the graph neural networks perform not well [38].
Hence, we leave the heterophilic graph extension of our framework as future work.
Method
In this section, we present our framework for shift-robust node classification.
In Figure 2, SRNC co-trains two GNN modules (1) a semi-supervised classification GNN Θ on the left side (Section 4.1) and (2) an unsupervised clustering GNN Φ (Section 4.2) on the right side.
The classification GNN is optimized on labeled source graph G s and pseudo labels on target graph G t for better generalization against possible distributional shifts.
Meanwhile, the clustering GNN is optimized on target graph G t and regularized with predictions from the classifier on source graph G s .
Finally in Section 4.3 and
Algorithm 1, we summarize how we optimize SRNC via iterative optimization between both modules.
We also provide convergence analysis of the co-training by Expectation-Maximization (EM).
In the remaining of the paper, we call training data as source and testing data as target interchangeably.
4.1 Shift-Robust Node Classification
We first propose the domain adaption loss for semi-supervised node classification towards distribution shifts.
Later in the paper, we always denote θ, ϕ as parameter of categorical distribution which are parameterized via Graph Neural Networks.
For example, Classifier P Θ = P θ ◦ GNN Θ minimize a negative cross-entropy loss on GNN embeddings,
[Equation]
where h i ∈ R N and h i,j is the j-th entry of the vector.
We have source loss L s Θ on training data ( h si , y si ) , and target loss L t Θ | Φ on pseudo samples ( h ti , ˆ y ti ) from target (testing) data.
[Equation]
We denote under-script Θ | Φ as the forward pass relies on both GNNs { Θ , Φ } while backward pass only updates Θ .
For example, in Figure 2,loss term
L s Θ | Φ receives samples from cluster block Q Φ but the back-propagation only update parameters of GNN Θ
The sampling process in the second term first uniformly sample same amount of nodes as training { x ti } |D s |
i =1 from target graph.
Then we obtain their node embeddings { z ti } and pseudo labels { ˆ y ti } through cluster GNN Φ and softmax layer
Q ϕ ( y i | z i ) .
We will discuss how to align the identity of cluster C
= { 1 ..K } and classes Y = { 1 ..N } at the end of clustering co-training (Section 4.2).
In left part of Figure 2, P Θ is first trained on source graph and jointly optimized on pseudo target samples.
Note that source and target graph can be same (semi-supervised) or different (supervised).
Our framework works for both kinds of distribution shifts that we deem essential in the paper.
Specifically, for open-set shift, we set number of cluster larger than known classes, i.e. K > N .
After alignment between clusters and classes, target samples in unmatched clusters are pseudo labeled as new class ˆ y =
| Y | + 1 .
For close-set shift, we simply set number of cluster and classes equal in two modules.
4.2 Graph Clustering Co-training
In this section, we will explain how to approximate the true target data distribution ( h ti , y it ) ∼ P t with Q ϕ and the connection between graph homophily and clustering.
The sufficient condition of uniform sampling from target P t requires an unbiased estimation of P t ( y ti | z ti ) if we uniformly sample node i from target graph.
Although underlying P t ( y ti | z ti ) is unknown, similar nodes likely appear in the same cluster based on graph homophily .
Hence, we decompose the approximation into two steps: (a) homophily-based clustering to group nodes into different clusters (b) co-training mechanism to match the identity of clusters to known label space on source graph.
Modularity [22] measures the strength of division of a graph into clusters by comparing random distributed edges between clusters.
Graph homophily ensures both source and target graph yields large modularity if grouped by ground truth labels.
Given the cluster membership matrix C ∈ { 0 , 1 } | V |× C , the modularity measure [22] S quantifies the divergence between the number of intra-cluster edges and the expected number in a random graph.
By maximizing the modularity, the nodes are densely connected within each cluster:
[Equation]
where c i and c j is the cluster membership of node i and j and d i ∈ R | V | is the degree of the node.
Direct optimizing the binary cluster membership matrix C is NP-Hard.
In GNN Φ , we model node k ’s soft cluster membership c k following [26].
We parameterize Q Φ ( c k | x k , A ) as Q ϕ ◦ GNN ϕ , using output embeddings and a softmax layer - Q ϕ ( c k | z k ) ∼ Categorical ( ϕ ) ,
[Equation]
The optimized cluster distribution is the soft cluster membership output by the GNN in Equation 6.
Given cluster distribution Q t Φ ( c i | x ti , A t ) on target graph, we are interested in its correlations with true P t ( y ti | x ti , A t ) .
In practice, graph homophily -based clustering algorithm would work consistently good on source and target w.r.t.
, that is, Q s Φ ( c i | x si , A s )
≈ P s
[Equation]
y si | x si , [Equation] ) ↔ Q
[Equation] [Equation] | x
[Equation] ≈ P t ( y ti | x ti , A t ) .
Besides the modularity clustering, lots of the graph clustering algorithms either explicitly or implicitly preserves graph homophily .
The choices of clustering algorithms are very flexible as shown in Section D.3.
To approximate the class conditional probability on source, first we need to build a mapping from the graph clusters C to the class labels Y using source graph.
The number of clusters is set to be equal or greater than number of classes N ( K ≥ N ).
Thus, we could build a bipartite mapping between the clusters { c Kk =1 } and the classes { y Nn =1 } .
We use the KL divergence to measure the distance between cluster c k and class y n over source data.
Specifically, we search for the optimal mapping
M ∈ { 0 , 1 } K × N by solving a linear sum assignment cost [14] between them, defined as follows,
[Equation]
Thus we can map some clusters to classes, i.e. , C L → Y, C L ⊆ C, | C L | = N .
After this step, the clusters on source graph are “colored” in Figure 2.
In order to achieve similar class distribution on source data ( Q s Φ ≈ P s ), we add a regularization term on clustering,
[Equation]
If labels are all known in source data, we random sample B nodes and { P s ( y si | h si ) } is a one-hot vector.
Otherwise (semi-supervised), we use current classifier’s inference probability on source data
P θ ( y | h si ) .
4.3 Model Optimization
In this section, we summarize the training objective for classification and clustering modules.
Following Equation 4, the overall loss for classification is,
[Equation]
where the last term (pseudo samples ( x si , ˆ y si ) on unknown source nodes) is added to ensure training convergence.
We have α = 1 for semi-supervised α = 0 for full-supervised source data.
Similarly, the overall loss for clustering also includes both source and target data,
[Equation]
Joint Optimization.
As we can see in the loss function, both modules require inference result (freezing parameters) from the other.
Therefore, we initialize both models with L s Θ and L t Θ | Φ , that is pre-training classification with labeled source graph and clustering with unlabeled target graph.
We then iteratively train the classification module and clustering module w.r.t.
L Θ and L Φ .
In L t Θ | Φ , we sample same amount of nodes as labeled source data D s on target.
In regularization loss L s Φ | Θ , we train clustering GNN Φ for T steps and sample B nodes each step.
Lastly, Algorithm 1 summarizes the joint optimization algorithm.
Convergence.
With the Variational-EM model [21], we now discuss the convergence of graph clustering co-training on source graph.
In the previous sections, we use categorical distribution
P θ , Q ϕ over hidden representation H, Z for two modules.
To analyze the convergence, we need to detail both terms with all of their parameters as P Θ ( y | X s , A s ) =
P θ ( y | GNN Θ ( X s , A s )) and
Q Φ ( c | X s , A s ) =
Q ϕ ( c | GNN Φ ( X s , A s ))
For simplicity, we denote G =
( X s , A s ) as inputs on source data.
In this analysis, clustering GNN serves as a variational distribution to approximate an intractable probability distribution P Φ .
Essentially, we optimize the evidence lower bound (ELBO) of the log-likelihood as follows,
[Equation]
In t-th E-step, regarding the Equation 12, the optimal Q ( t +1)Φ is,
[Equation]
The solution of the above E-step is,
[Equation]
In practice, we achieve this update by sample B nodes on source graph, which is exactly the regularization term L s Φ | Θ (Equation 9).
In t-th M-step, we update the output distribution on source from classifier P ( t +1)Θ with Q ( t +1)Φ ( y | G ) .
To estimate the expectation term in Equation 12, we sample ( x sj , ˆ y sj ) ∼ Q Φ to compute the loglikelihood and update parameter Θ of the classifier.
Given the alignment between cluster and classes in Section 4.2.
Interestingly, we describe the same sampling process on target in L t Θ | Φ and now we add L s Θ | Φ in final loss (Equation 10) of classifier to accomplish the M-step.
Therefore, at each episode, the joint optimization is equivalent to perform variational EM alternatively.
The convergence is proven in the original paper [21].
Discussion.
Micro-graph [36] employs similar clustering and EM optimization between node and motif representations for the purpose of contrastive learning.
The main difference is that we adopt EM between classification and clustering to achieve domain invariant learning under similar homophily ratios.
Limitation of our method are two folds: (a) it requires access of test data as other domain adaptation algorithm (b) the performance of clustering co-training is not guaranteed when testing graph is more heterophily compared with training.
Complexity.
Compared with common domain adaptation algorithms, our main difference in computation comes from graph clustering co-training.
Hereby, we analyze the additional computation cost of Q in each episode.
Let O ( E ) be the time GNNs take to compute a single node embedding and class or cluster membership for each node.
Assuming there are | V t | nodes in target graph and node degree is constant order ( i.e. O ( | V t | ) ̸ = O ( E ) ), the pre-training of clustering takes O ( | V t | · E ) .
During iterative optimization, the KL-divergence regularization costs O (2 ·E · B · T ) , T is the number steps in Algorithm 1.
Overall, the extra complexity O (( | V t | + 2 · B · T ) · E ) is in linear of target graph size | V t | , because usually 2 · B · T ≪ | V t | .
It is the same as most other domain adaption algorithms [33, 32], where target node representations are calculated each episode as well.
Experiments 5.1 Experimental Setting
Datasets.
In our experiments, we perform semi-supervised node classification on four benchmark networks (see Table 4).
These four networks are: Cora, Citeseer, PubMed [24] and ogb-arxiv [9].
We conduct open-set shift experiments on first three datasets and close-set shift on ogb-arxiv, because ogb-arxiv naturally has distribution shift across different time periods.
Parameter Settings and Reproducibility.
In experiments, we use the GCN [11] and add self-loop in graph for all compared algorithms including ours.
The Adam SGD [10] optimizer is used for training with learning rate as 0.01 and weight decay as 0.0005.
The hidden size of all GNN models including
SRNC is 128.
We set cluster number as 16 in cluster GNN Φ for open-set experiment and same as number of classes in close-set experiment.
The pre-training epochs of episode
(Algorithm 1) for classification and clustering are set as 200 and 1000, respectively.
Besides, B is set as the same as number of training nodes and T = 500 in co-training.
Note that none of Cora, Citeseer and Pubmed has 16 classes, which do not favor our method as a fair comparison.
To provide a comprehensive study of SRNC, we design two ablations of our algorithms:
• SRNC w.o Φ : a bootstrapping ablation, where we pseudo label the same amount of data on target using model θ predictions instead of co-trained clustering Φ in Equation 4.
• SRNC Ep.1: we stop the algorithm after episode 1 to verify the effectiveness of iterative optimization. [Table]
5.2 Close-Set Shift on Dynamic Graph
Compared Algorithms.
On close-set shift, we provide the result from multiple close-set domain adaption methods on both supervised and unsupervised GNNs: GCN, DGI [28] with DANN [5],
CMD [35], SRGNN [39], UDAGCN [32] and EERM [34].
The first three methods are model-agnostic, so we applied to both GNNs.
The latter two require end-to-end training witn GNNs.
Evaluation Setting.
On ogb-arxiv, we split the train, validation and test nodes by year.
There are total 12974 (-2011, train), 28151 (2011-2014, validation), 28374 (2014-2016, test set 1),(2016-2018, test set 2) and 48603 (2018-2020, test set 3) paper, respectively.
In testing, each test graph contains all previous nodes such that train graph is a subset of all test graphs.
This dynamic graph setting is close to real world applications, where model is trained on an earlier snapshot (source) and deployed on more recent ones (target).
Results.
Most of close-set domain adaption baselines in comparison ( i.e. DANN, CMD, SRGNN), focus on learning invariant feature representations between source and target.
However, as we stressed before, conditional shifts presumably exist in real-world graphs so regularization (CMD) and adversarial head (DANN) make the performance even worse compared with original model.
Then we are wondering whether unsupervised representation learning is more robust to (close-set) distribution shifts.
We optimize the mutual information objective in DGI on training graph to obtain testing set node embeddings and train a multi-layer perceptron (MLP) classifier with training data.
From
Table 1, we can observe the testing performance of DGI is worse than GCN across different testing periods.
Similarly, existing domain invariant approach only makes the result worse on DGI.
The performance drop is even larger than their GCN counterparts.
It is probably because the distribution shifts happen mostly in GNN encoders and adaption method can not mitigate the shifts in pre-trained graph encoders.
The performance of SRNC is significantly better than other baselines across all test period because we are able to capture the joint distribution Pr test ( X, Y ) on target data.
Besides our method, GCN-UDA is the second best method, it also considers graph structure information via inter-graph attention and therefore, outperforms DANN (domain classifier only) a lot.
EERM reports the similar performance of the original paper on ogb-arxiv dataset, but only present consistent but not superior performance.
As ablations, the bootstrapped pseudo labels (SRNC w.o. Φ ) from classifier can only marginally improve the performance and the improvement on GCN is much smaller as shifts become larger(20182020 split).
SRNC Ep.1 also reports worse results since the algorithm is not converged yet.
Our results demonstrate that our framework is effective under different close-set shift introduced by dynamic graph.
5.3 Open-Set Classification
Compared algorithms.
Since most of the close-set domain adaption algorithms cannot deal with open-set shift, we apply a different set of open-set classification baselines: THS [8], DOC [25],
PGL [17], OpenWGL [33].
[Table]
Evaluation Setting.
We create open-set shift by removing 3 classes on Cora/Citeseer and one class on PubMed from training data.
In testing, nodes from the masked classes are all re-labeled as the unknown class.
For each known class, we random sample 20 nodes for training and report the mean and standard deviation of micro-F1 and macro-F1 for 10 runs.
Besides, we report the performance of GCN and DGI with the unknown label in training data (GCN-IID and DGI-IID) in Table 2, and relative performance drop (of other methods) in Micro ( ∆ F1).
In validation set, we have nodes from the unknown class and use it to select the best hyper parameters all of the baselines and SRNC.
For example, in PGL, we use validation to pick the best threshold α among each episodes { α k } and label nodes with lower probabilities into unknown class.
SRNC has an explicit class for unknown, so we use the Micro-F1 on validation to stop the iterative optimization.
Results.
Compared with reference IID version (GCN-IID and DGI-IID), all methods experience performance drop due to open-set shift.
Among the baselines, threshold based methods GCN-PGL and GCN-THS perform better than the others.
OpenWGL, to our surprise, reports much worse performance in semi-supervised learning , it is probably because the uncertainty difference between known classes and unknown class is not that significant without enough training data.
We also notice the open-set shift is more severe in PubMed, although its unknown ratio (1/3) is lower than the other two.
In literature, when training data is IID (no open-set shift), unsupervised or contrastive learning is even better than semi-supervised GNNs with the same amount of training data.
However, end-to-end supervised GNNs (GCN) is more robust than unsupervised GNNs (DGI) when there is open-set shift.
It is consistent with the close-set observations and for the same reason semi-supervised GNNs can enjoy more improvements from domain adaptation algorithms.
On average, SRNC outperforms all the other representative open-set algorithms for at lease 4 % and 2 % in micro-F1 and macro-F1, respectively.
From the comparison between our own ablations, we find that the graph clustering co-training(SRNC vs. SRNC w.o. Φ ) contributes the most to the performance gains.
This ablation performs bootstrapping that randomly samples pseudo unknown nodes from classifier’s low confident predictions.
In other words, our co-training design is clearly better than bootstrapping and drawing samples from unseen clusters is the key towards better open-set generalization.
Moreover, the iterative brings more improvement (1 ∼ 3% F1) on top of SRNC Ep. ablation since variational EM takes several round to converge in practice.
Conclusion
In this paper, we proposed a general framework, SRNC, to enable graph neural networks for distributional shift.
Different from existing work on either open-set shift or close-set shift , SRNC works for both scenario with a novel graph clustering co-training component.
Accordingly, SRNC employ a latent variable model ( i.e. cluster GNN) to model the latent structure and allows the clustering GNN to improve the generalization of shift-robust classification GNN.
Future work includes applying the model to applications such as medical diagnosis and molecular dynamics, where robustness towards distribution shifts are in critical need.
A Algorithm details
1 /* Episode 0 */ 2 Pre-train the classifier GNN Θ via L s Θ and the cluster GNN Φ via L t Φ ; 3 /* Iterative optimization, episode 1,2...
*/ 4 while Micro-F1 on validation increases do /* update cluster GNN Φ for T steps each episode */ for t = 1 to T do
Sample B nodes ( x si , y si ) from source Φ − ←− ∇ Φ L s Φ + ∇ Φ L t Φ | Θ end /
* update shift-robust classifier GNN Θ */
Sample |D s | nodes ( x ti , ˆ y ti ) from target while L Θ not converge do Θ − ←− ∇ Θ L s Θ + ∇ Θ L t Θ | Φ + ∇ Θ α L s Θ | Φ end 15 end
In addition, we provide the comparison between domain adaptation graph neural networks in Table 3.
[Table]
[Table]
B Experimental Setups B.1 Dataset Details
[Table]
[Table]
B.2 Evaluation Metrics.
In our experiments, we have { ( x t , y t ) } in testing (target) data D t and calculate Micro-F1 and
Macro-F1 on predictions ˆ y from target data,
[Equation]
Notice that in open-set shift, target data D t has one more class than training - the unknown class ( N test =
N train + 1) ).
C Additional Experiments C.1 Unseen Class Discovery
[Table]
Use Case.
DBLP is a computer science bibliography website.
We construct a heterogeneous network upon three types of nodes: (A) Author, (P) Paper and (T) Term.
There are five types of papers labeled by the corresponding venues 2 - Data Mining (DM) , Database (DB) , Artificial Intelligence (AI) ,
Natural Language Processing (NLP) and Computer Vision (CV) .
We sampled 50,000 papers between 2010 and 2015, then use the venue information to label 5567 of them into 5 classes.
After filtering out rare authors and terms, there are 20601 papers, 5158 authors and 2000 terms in total.
In this subset, we observed only 30 % of 20,000 papers are labeled into known categories.
In other words, the majority of nodes in the DBLP graph belong to unseen classes.
This is a real situation where people do not know the label space in advance and it poses an open-set challenge for practitioners.
One biggest advantange of SRNC is the capability of unseen class discovery clustering GNN GNN Φ .
Unlike any other open-set classification algorithms, a graph clustering component can further partition the unknown objects into different groups based on their structural and semantic similarities.
Results.
In Table 5, we show the paper in DBLP dataset that are assigned to the (unmatched) clusters, discovered by our Cluster GNN Φ and Constrained-KMeans + DGI embeddings.
Two methods utilize the same amount of supervision of known classes and we set total number of clusters as 10.
Interestingly, from the five clusters that are supposed to belong to unseen classes, we find two shared clusters about “Computational Mathematics” and “Network & Distributed System” from both algorithms.
The results show ad-hoc semi-supervised clustering on node embeddings produces erroneous cluster membership even for the top-ranked items.
For example, it contains almost half papers from the computer architecture in “Network & Distributed System”, whereas unseen classes suggested by our GNN clusterring are much more consistent.
Traditional clustering on off-the-shelf network embeddings lack rationales of objects within the same clusters.
When our method optimize the ELBO between classification and clustering results on seen classes, it adopts the similar rationales in unseen clusters.
The clustering result also provides insight for designing the new label space or active learning paradigms.
[Table]
D Model Analysis and Parameter Study
D.1 Parameter Sensitivity.
Among all the hyper-parameters, the pre-defined number of clusters | C | seems to be the most crucial in open-set scenario.
Now we provide the performance of SRNC with varied numbers of clusters k , i.e. 7 (optimal) vs. 16 (used in experiment) in Table 6.
Both Micro-F1 and Macro-F1 are further boosted, since seven is the ground truth number of classes.
On the one hand, SRNC is not sensitive to the choice of | C | and better than baselines regardless, especially when there are multiple missing classes (closer to the real-world setting).
On the other, an accurate estimation of | C | can facilitate the model training.
[Figure]
D.2 Effectiveness of iterative optimization.
In Figure 3, we demonstrate the learning procedure on DBLP dataset.
Each column shows the predicted labels from classification GNN (upper) and the cluster assignments from the cluster GNN (lower).
We project the node embeddings onto two dimensions using t-SNE.
In this example, we set number of clusters to six.
In Figure 3a, the pre-trained classification GNN predicts all papers intoseen classes.
Our cluster GNN shows 6 initial clusters.
During the training, the clusters in Figure 3e gradually align with the classifier’s predictions in same color.
The classifier presented in Figure 3b starts to recognize nodes of unseen class with pseudo samples from cluster corresponds to unseen class (yellow) in Figure 3e.
When the training converges, the clusters in Figure 3f are separated clearly.
[Figure]
D.3 Applicability of SRNC framework.
As we discusses in section 4, SRNC is not tailored for one specific GNN architectures or unsupervised graph clustering algorithms.
In this set of experiments, we investigate four different classification models for GNN θ (GCN [11], GraphSAGE [7], GAT [27], SGC [31]).
These four GNN models are widely used for different applications and effective on the Cora dataset we used in this study.
Besides, we investigate two rather different clustering algorithms on graph - DMoN [26] (used in main experiment), DGI+DEC [28] in Figure 4.
Deep embedding clustering (DEC) is a self-trained clustering algorithms.
We use K-means on the DGI node representations to initialize the cluster centers and DEC updates the cluster centers and same soft assignment Q matrix in Equation 7.
We observed that in general, all of these variations are at least (2% on Micro-F1) better than the best performed baseline in the open-set experiments (Table 2).
Specifically, four different GNN architectures yield similar performance except GAT, which is a bit complicated for domain adaptation on small graphs.
Moreover, two different graph clustering algorithms report close numbers.
It validates the convergence of variational EM is empirically reachable as long as the underlying class distributions are well captured by clustering algorithms.