GraphDE: A Generative Framework for Debiased Learning and Out-of-Distribution Detection on Graphs
Abstract
Despite the remarkable success of graph neural networks (GNNs) for graph representation learning, they are generally built on the (unreliable) i.i.d. assumption across training and testing data.
However, real-world graph data are universally comprised of outliers in training set and out-of-distribution (OOD) testing samples from unseen domains, which solicits effective models for i) debiased learning and ii)
OOD detection, towards general trustworthy purpose.
In this paper, we first mathematically formulate the two challenging problems for graph data and take an initiative on tackling them under a unified probabilistic model.
Specifically, we model the graph generative process to characterize the distribution shifts of graph data together with an additionally introduced latent environment variable as an indicator.
We then define a variational distribution, i.e., a recognition model, to infer the environment during training of GNN.
By instantiating the generative models as two-component mixtures, we derive a tractable learning objective and theoretically justify that the model can i) automatically identify and down-weight outliers in the training procedure, and ii) induce an effective OOD detector simultaneously.
Experiments on diverse datasets with different types of OOD data prove that our model consistently outperforms strong baselines for both debiasing and OOD detection tasks.
The source code has been made publicly available at https://github.com/Emiyalzn/GraphDE .
Introduction
For processing ubiquitous graph data, graph neural networks (GNNs)
[17, 43] have emerged as effective approaches for graph representation learning [12], which in turn has spanned a wide range of applications, from drug discovery [23] to image classification [4].
Despite their success existing GNNs are generally built on the assumption that both training and testing data are independently sampled from the identical distribution (i.i.d.), which often does not hold [45, 22, 49, 48].
Real-world training data is often shown universally mixed with outliers [30, 7], which specifically may incur large gradient steps in the wrong direction [38], or be overfitted by the model [3], thereby hurting models’ performance.
Besides, testing samples from unseen domains are also unavoidable when the model is deployed in the wild [2, 19], which may easily be classified into a wrong class with a high probability [13].
Inevitably, graph data also involve these two common problems.
As illustrated in Fig.
1, we indeed find that vanilla GNNs perform poorly when encountering large portion of outliers and out-of-distribution (OOD) testing samples, which solicits effective models for i) debiased learning and ii)
OOD detection, towards general trustworthy purpose.
0.2 Biased Ratio
T e s t
A cc u r a cy ( % )
SAGPool
SAGPool GraphDE-a
TopKPool
TopKPool GraphDE-a (a) Debiasing results on SPMotif
OOD Score
F r e qu e n c y F r e qu e n c y
OOD Score
Max SoftmaxScore
Variational Inferred Score
[Figure]
We argue that training outliers and OOD testing samples can be conceptually unified as they both represent data drawn from distributions other than the major in-distribution (ID) part of training data.
However, they are inherently different as the training outliers also catch label-related distribution shifts, e.g. noisy labels [1], while OOD testing samples are solely determined by their features [14] since we cannot access their ground-truth labels.
This difference also explains why many existing works tend to tackle the two problems independently.
In light of the intrinsic connection between these two types of data we present GraphDE, a probabilistic generative framework for Graph DE biased learning and OOD DE tection.
Specifically, we model the generative process to characterize the distribution shifts of graph data, along with an environment variable as an indicator.
Utilizing variational inference [51], we propose to infer the environment variable during the training procedure of GNN.
Our framework consists of three modules: the recognition model that infers the environment variable based on the input graph data; the structure estimation model by which we extract an OOD detector on testing dataset; the classification model which is our target GNN.
Following the idea of data resampling [38, 37], we model the latter two modules by two-component mixtures, with one component for ID data, and the other component robustifying the model by isolating outliers.
Based on this, we derive a tractable learning objective for our model and theoretically justify its effectiveness for the target tasks.
The contributions are:
• We focus on two critical but largely unexplored problems for graph data: debiased learning and OOD detection, find their fundamental correlations and define them under a probabilistic framework.
• We propose GraphDE with a novel learning objective and theoretically justify that: 1) it can identify and down-weight outliers during training; 2) it can provide an OOD detector on the testing set; 3) its constituent modules are optimized in a mutually-promoting manner.
• We evaluate GraphDE with different GNN backbones on diverse datasets of different OOD types, where GraphDE achieves consistent performance improvements over the baselines.
Specifically, as shown in Fig.
1, for the debiasing task, it yields up to 14.6% accuracy improvement on SPMotif; for the OOD detection task, it outperforms the strongest baseline by 9.31% on MNIST-75sp.
Problem Formulation and Related Works
In typical graph learning problems, given a set of labeled graphs D tr = { ( G i , y i ) } N tr i =1 , we need to train a model f θ : G → ˆ y to predict the labels on the testing dataset D te = { G i } N te i =1 .
Specifically, the input graph G =
( A, X ) is composed of an adjacency matrix A
= { a vu | v, u ∈ V } and node features
X = { x v | v ∈ V } , where V is the node set.
Define G as a random variable of the input graph and y as a random variable of the graph label, the data-generating process can then be characterized as p ( G , y | e )
= p ( G | e ) p ( y | G , e ) where we introduce an environment variable e as a potential unknown indicator for data from particular distributions.
The biased data (we use the form interchangeable to outliers) in the training set and out-of-distribution (OOD) data in the testing set can both be treated as data drawn from different distributions to the distribution generating the majority of the training set and testing data of our interests, i.e., the in-distribution (ID) data so-called.
Hence, we can model them by a binary environment variable e , with e = 1 denoting the ID portion and e = 0 for the biased/OOD portion.
Then the training and testing datasets can be split into D tr = D trin ∪ D trout and D te =
D tein ∪ D teout accordingly.
Note that i) the partitions of training/testing data are only for illustration purpose and are strictly unknown in practice; ii) the testing data and distribution are also strictly unknown during the training process; iii) here e is mainly used for distinguishing the ID part with the OOD counterpart of datasets.
In fact, the
OOD part ( e = 0 ) may consist of data from multiple distributions 2 .
Denote the loss on sample ( G, y ) by L ( y, f θ ( G )) , e.g., cross-entropy loss for classification problems.
Then the standard empirical risk minimization (ERM) objective can be written as
[Equation]
This objective works based on the i.i.d.
assumption across training and testing data.
As we aim at promoting the ID test accuracy (i.e. the performance on D tein ), however, the updating gradient by training data drawn from p ( G , y | e = 0) will hurt model performance drastically [3].
On the other hand, for testing data from unseen distributions D teout , the model would incline to produce imprecise results as p tr ( e ) ̸
= p teout ( e ) , which is undesired and should be avoided for the trustworthy purpose.
We next formulate the two problems of our interests.
Task I: Debiased learning for training data.
The target is to mitigate outliers’ negative effects during training, thereby enhancing the performance on ID testing data.
We modify the ERM objective:
[Equation]
where L ( y, f
[Equation] [Equation]
=
[Equation]
where the loss function for biased training data is fixed as L 0 , so the debiasing objective is equivalent to minimize E ( G,y ) ∼ p
( G , y | e =1) [ L ( y, f θ ( G ))] , i.e., the training loss for the ID portion D trin .
Task II: OOD detection for testing data.
We aim to obtain an OOD detector g composed of a trained model f θ : G → ˆ y , a scoring function s and a threshold τ :
[Equation]
As we are willing to induce an OOD detector during the training procedure of GraphDE, we leverage the trained model f θ from task
I and what we need is to find the optimal score function s ∗ that can best separate out ID and OOD data (e.g. maximize the AUROC).
Related works on debiased learning.
A series of strategies have been proposed to debias the training dataset in general machine learning [38, 5] and computer vision [30, 7].
Many works propose to preprocess the dataset by filtering out the outliers [5, 25].
However, these methods cannot capture label-related distribution variations such as noisy labels [1] and also cannot be easily adapted to deal with graph data.
Another popular strategy is data resampling [38, 37], which relies on the training loss to identify and down-weight outliers in the training set.
However, these methods only capture bias in the mapping from features to labels, but ignore the variation in feature space.
For the first time to our best knowledge, we formally define and deal with the graph debiased learning problem.
Related works on OOD detection.
A trustworthy learning system should not only produce accurate predictions for ID data, but also distinguish and reject OOD examples without further prediction.
[Figure]
ID:
OOD:
OOD: inferred e
Generative Models mutually boost
Generative Models infer infer
ID:
OOD:
ID:
OOD: computed = + +
Recognition Model #
(%|',))
# 3 (+|',))
Training Stage
[Figure]
Deep learning based OOD detection has achieved promising progress in vision [27, 13].
Besides, node and edge-level anomaly detection on graphs are also widely studied, some of which are based on classical network science [26], while others utilize the power of GNNs [36, 9].
In comparison, only limited works [52, 29] attempt to solve graph-level OOD detection.
Specifically, OCGIN [52] designs a one-class GNN objective to distinguish between ID and OOD data.
GLocalKD [29] utilizes joint random distillation of graph and node representations to learn rich global and local pattern information of graph-structured data.
However, both of them need to be trained individually for the detection task.
In this paper, we extract an OOD detector during the training procedure of the target
GNN, while outperforming all the baselines on 7 out of 9 benchmarks.
Appendix B gives more comprehensive description on related works on GNNs and Trustworthy GNNs.
Methodology 3.1 Probabilistic Generative Learning for Debiased Learning and OOD Detection
To improve the ID test accuracy, we want to make the model more robust by automatically identifying potential outliers during training.
Thus, we take a generative perspective on the graph data and adopt variational inference to infer the environment variable during training.
Generative Process.
We model the joint distribution p ( X , y , A , e ) to capture graph-level distribution shifts.
In line with the homophily principle from random graph theory [33], i.e., linking probability depends on some inherent properties of nodes, we assume that the graph structure is generated based on the node features and the environment variable.
The graph labels are then generated based on the graph structure, node features, and the environment.
We also assume p ( e ) and p ( X ) to be independent distributions for environments and node features, respectively.
We do not specify the form of p ( X ) , since everything will be conditioned on X in this paper, similar to [28, 44].
We have:
[Equation]
where the generative models p ( A | X , e ) and p ( y | X , A , e ) could be modeled by some flexible parametric distributions p θ ( A | X , e ) and p θ ( y | X , A , e ) with parameter θ .
These two distributions will later be instantiated as the structure estimation model and classification model, respectively.
We define the generative models with two-component mixtures.
For example, p θ ( A | X , e ) is composed of an ID component p θ ( A | X ) , and an OOD component p
( A | X ) (the subscript 0 implies a fixed distribution).
The environment variable e ∈ { 0 , 1 } indicates whether the graph should be modeled by the ID ( e = 1 ) or the OOD ( e = 0 ) component.
Specifically, these two terms are decomposed as:
[Equation]
Recognition Model.
As the variables X , A and y are all observed during training, what we need is to infer the sample-wise environment variable e conditional on these observed variables.
Thus, we resort to variational inference [51] and introduce a recognition model q ϕ ( e | A , X , y ) parameterized with ϕ to approximate the true posterior p θ ( e | A , X , y ) .
Learning Objective.
We train the model parameters θ and ϕ by optimizing the Evidence Lower
BOund (ELBO) of the observed data tuple ( A, X, y ) :
[Equation]
Eq.
6 shows that by maximizing the ELBO, we are maximizing the log-likelihood of observed data log p θ ( A, y | X ) while minimizing the KL divergence between the recognition model and the true posterior p θ ( e | A, X, y ) .
Finally, the learning objective for the training dataset can be written as:
[Equation]
[Table]
[Equation]
Justification of GraphDE.
We provide theoretical analysis here to reveal the rationale of GraphDE.
Without loss of generality, we do not take account into the KL loss here for simplicity.
The proofs of the propositions are postponed to Appendix A.
Proposition 1. 1)
The learning objective for GraphDE is in a re-weighted form when q ϕ ( e | A , X , y ) is instantiated as a Bernoulli distribution, with q ϕ ( e i
= 1 | A
i , X i , y i ) acting as a weight for the i -th sample; 2) Given the ideal recognition model q ∗ ϕ that gives q ∗ ϕ ( e = 1 |
( A, X, y ) ∈ D in ) =
1 and q ∗ ϕ ( e = 1 | ( A,
X, y ) ∈
D out )
= 0 , the generative models can learn to best fit the ID data.
Prop. 1 reveals how the inferred posterior takes effect on the learning objective.
Typically, if q ϕ ( e | A , X , y ) can assign a higher probability to ID data, our model can thereby learn to ignore the negative effects of biased training data and better fit the target distribution (i.e. ID).
Proposition 2. 1)
Assuming the generative models fit to the ID data, i.e. p θ ( A | X ∈ D in ) ≥ p θ ( A | X ∈ D out ) and p θ ( y | ( A, X ) ∈ D in ) ≥ p θ ( y | ( A, X ) ∈
D out ) , the recognition model will learn to predict q ϕ ( e = 1 | ( A, X
, y ) ∈ D in ) ≥ q ϕ
( e = 1 | ( A,
X, y ) ∈
D out ) ; 2) Given optimal generative models that best fit the ID data and perform randomly on outliers, there exists a recognition model q ∗ ϕ which yields the minimal objective while ideally predict the environment variable.
Prop. 2 proves GraphDE’s effectiveness from the recognition model’s perspective.
To summarize, the above two propositions show that q ϕ ( e | A , X , y ) , p θ ( y | A , X ) and p θ ( A | X ) can mutually promote each other during training.
As deep networks tend to learn simple shared (i.e. ID) patterns first [3], this positive feedback can guide GraphDE to identify outliers in early epochs and thereby learn better representations of ID data.
Besides, the propositions also suggest that given either of the recognition model or the generative model optimal, the other can be trained to be optimal, which suggests the existence of the global optimum.
Based on these, we can further understand the implications of the three loss terms in Eq.
7: i)
L cls is the re-weighted version of the original classification loss; ii) L reg acts as graph regularization and will impact the gradient of q ϕ ( e | A , X , y ) , weighting ID data higher than the outliers; iii) the third term L kl is the KL divergence between the learned posterior and the fixed prior distribution, ensuring the posterior not go too complex.
Inference on Testing Dataset.
For new data, we need to infer the posterior conditional distribution of e and y given ( A , X ) , i.e. p θ ( e | A , X ) and p θ ( y | A , X ) , respectively.
To compute p θ ( e | A , X ) is just to do OOD detection .
For binary e , we can compute its posterior analytically:
[Equation]
The posterior can be used as the score function s for the OOD detector, once we have the well-trained p θ ( A | X , e ) .
Subsequently, we can calculate p θ ( y | A , X ) by the total probability theorem:
[Equation]
where p θ ( y | A , X , e ) can also be obtained after training.
3.2 GraphDE Instantiations
We proceed to specify the distributions p θ ( A | X , e ) , p θ ( y | X , A , e ) , p ( e ) and q ϕ ( e | A , X , y ) .
We assume p ( e ) =
Bernoulli ( α ) where α ∈ [0 , 1] represents our prior belief about the cleanliness of the dataset.
A higher α implies a larger portion of ID data in D tr .
It is fixed as a scalar during training.
3.2.1 Instantiations of the Recognition Model
We propose two variants for the recognition model q ϕ ( e | A , X , y ) .
In the first way, we directly assign a learnable scalar for each sample in the training dataset, i.e., q ϕ ( e i
|
A i , X i , y i ) =
Bernoulli ( α i ) , where α i ∈ [0 , 1] is a trainable parameter jointly optimized with generative models through the ELBO in Eq.
We call this variant GraphDE-v(ariational) as we optimize directly over α i .
In the second way, notice that e is a binary variable by definition, we can compute the posterior analytically by
[Equation]
We call this variant GraphDE-a(nalytical) and the optimization procedure in this setting is essentially an EM algorithm [31]: in the E-step, we evaluate the posterior distribution of the latent variable e by Eq.
10; in the M-step, we optimize the model parameters θ using gradient descent w.r.t.
the objective in Eq.
In comparison, GraphDE-v runs more efficiently as it does not need to calculate the posterior, while GraphDE-a can give a tight approximation log p θ ( A, y | X )
=
L ELBO .
We compare the space/time complexity of the algorithms in Appendix F.3.
3.2.2 Instantiations of the Structure Estimation Model
Recall that p θ ( A | X , e ) is defined by a two-component mixture (Eq. 5), we need to specify p θ ( A | X ) and p
( A | X ) separately for the structure estimation model.
We follow the common practice and assume that the edges are conditionally independent [28].
Then the conditional probability of A can be further factorized as p θ ( A | X )
= (cid:81) v,u ∈ V p
θ ( a vu | X ) .
We next consider two versions for p θ ( a vu | X ) .
First, we adopt the latent space models (LSM) [15] where the probability of a vu only depends on the representations of nodes v and u :
[Equation]
Eq.
11 can be viewed as a logistic regression model where σ ( · ) denotes the Sigmoid function, U is a linear transformation matrix, and w is a learnable vector.
Second, we propose a variant of LSM referred as CosLSM which utilizes the cosine similarity function to measure the similarity between the node representation pair ( x v , x u ) :
[Equation]
where we harness multi-head [42] linear transformations { U i } mi =1 for better capacity.
Note that if we train the structure estimation model completely based on the observed edges, the model will yield trivial solutions which predict all edges’ probabilities as 1.
To avoid this issue, we randomly sample one disconnected node pair labeled as a vu = 0 for each existing edge during training.
Finally, for p 0 ( a | X ) , we simply fix it as p 0 ( a vu = 1 | X ) = 1 / 2 .
Although more complex distributions can be used for improving the capacity, our experiment results empirically show that this simple distribution is enough for a wide range of debiasing and detection tasks on graphs.
3.2.3 Instantiations of the Classification Model
We next specify p θ ( y | A , X ) and p 0 ( y | A , X ) for the classification model.
Similar to the structure estimation model, we fix p
( y | A , X ) =
K − 1 where K denotes the total number of classes.
Typically, a GNN takes ( A , X ) as input and outputs the probability of y .
So for p θ ( y | A , X ) , we adopt two types of GNN models: the global pooling models [17, 43] and hierarchical pooling models [21, 18].
Denote node v ’s neighborhoods by N v , specifically, common GNN convolutional layers execute recursive feature propagation along G :
[Equation]
[Equation]
) where W ( l ) denotes weight matrix in the l -th layer and AGG ( l ) is a permutation-invariant function that aggregates node representations.
In hierarchical pooling GNNs, each convolutional layer is followed by a pooling layer to filter out nodes for the next convolution operation:
[Equation]
(14) where idx is the selected node indices obtained by operations such as selecting the top- k score (calculated by a learnable function) nodes.
Finally, for both types of GNNs, a global pooling layer (e.g. max or mean operation over node features) is adopted to extract a graph-level representation h to be fed into the classifier Φ whose probability is obtained by an output softmax layer:
[Equation]
where n denotes the total number of convolutional layers.
Experiments
In this section, we conduct extensive experiments to answer the following questions:
• Q1:
How effective is GraphDE in debiased learning and OOD detection?
• Q2:
How do the composed modules and hyperparameters impact GraphDE’s performance?
Besides, how does GraphDE take effect during the training procedure?
4.1 Experiment Setup
Datasets.
One synthetic and three real-world datasets of different OOD types are used for debiasing and OOD detection tasks.
Here we briefly introduce them, leaving details of statistics to Appendix E.
• Spurious-Motif [50] (SPMotif) is a synthetic dataset in which each graph is composed of one base (denoted by S ) and one motif (denoted by C ).
The graph label y is solely determined by the motif C .
To simulate distribution shifts, we manually construct spurious correlations between S and y .
We use datasets with different spurious correlations to represent ID and OOD data, respectively.
This dataset is unsuitable for typical OOD detection task, as the distribution shift is dominated by label information, opposed to that the OOD detectors focus on detecting input graphs with abnormal structures or node features.
• MNIST-75sp [18] converts the MNIST images into super-pixel graphs with at most 75 nodes.
For OOD data, we add Gaussian noise to node features to simulate distribution shift.
• Collab [47] is a social network derived from 3 public collaboration datasets.
We use the graph size to simulate distribution shift, with graphs of no more than 45 nodes denoted as ID data.
• DrugOOD [16] aims to benchmark the OOD generalization performance in the field of AI-aided drug discovery.
We use the provided dataset curator to generate a dataset with different scaffolds , and treat the largest 6000 molecules as ID data.
[Figure]
[Table]
Note that during the debiased training stage of GraphDE
[Table]
we need to mixed a certain number of OOD data into the training dataset.
To prevent potential data leakage [14] during the evaluation of OOD detection, the OOD testing dataset should be drawn from a different distribution
w.r.t.
the OOD data mixed into the training dataset.
We denote these two sources of OOD data as the OOD testing dataset and the OOD mixed dataset, respectively.
We leave more details of dataset splits to Appendix E. GNN backbones.
We adopt two types of GNN backbones for debiasing and OOD detection tasks.
• Global pooling backbones.
We adopt two popular GNNs: GCN [17] and GAT [43].
To extract the global graph representation for graph classification, we use a global mean pooling or global max pooling layer after the GNNs to summarize the node representations.
• Hierarchical pooling backbones.
This architecture is comprised of blocks each of which consists of a graph convolutional layer and a pooling layer.
Typically, TopKPool [18] selects local parts of input graph for propagation by a learnable score function; SAGPool [21] uses the self-attention mechanism to calculate attention scores, and retain important nodes for propagation.
Debiasing settings.
For the debiasing task, we mix the training set with certain ratio of outliers and compare models’ performance on ID testing data.
We evaluate GraphDE’s debiasing performance on the four datasets.
Besides, As far as we know, there are little existing works concerning debiased learning on graph data.
For demonstrating the empirical superiority of our approach, we adopt two recently proposed methods that claim to enhance the general-purpose robustness of GNNs as competitors: 3 : DropEdge [39] randomly removes a number of edges from the input graph in each training epoch, and GRAND [11] uses a random propagation strategy to perform graph augmentation.
We study the negative effect of outliers by varying the biased ratio ( |D trout | / |D tr | ) of training data.
OOD detection settings.
We evaluate GraphDE’s OOD detection performance on three real-world datasets (with equal number of ID and OOD testing samples), using three widely used metrics [14]: area under the receiver operating characteristic curve ( AUROC ), area under the precision-recall curve ( AUPR ), and the false positive rate at 95% true positive rate ( FPR95 ).
Besides, we consider three types of baselines: first, we use the backbones’ max softmax score (MSP) [13] as a vanilla baseline; second, we adopt two-stage graph kernel baselines in [52] (specifically, we adopt WL [40] and PK [34] kernels, LOF [5] and OCSVM [6] detectors); finally, we use OCGIN [52] and GLocalKD [29] as two deep learning based detection baselines.
[Table]
4.2 Main Results (Q1)
We train and evaluate each model under the settings as described in the last section (more hyperparameter and training/testing details in Appendix D).
All the experiments are run over 5 different random seeds.
More experimental results can be found in Appendix F.
We make the following observations:
Debiasing results.
Tab. 1 reports GraphDE’s debiasing performance on SPMotif.
In this setting, we treat the robust GNN baselines and two variants of our GraphDE as plug-in modules and apply them to all the four backbones for comparison.
The results show that GraphDE consistently (for all the backbones) outperforms the baselines by a large margin along with low variance.
Specifically, for the TopKPool backbone, GraphDE-a surpasses the backbone by 10.11% and the strongest baseline (GRAND) by 5.29%.
Such improvements strongly suggest that GraphDE can achieve better debiasing performance for various backbones.
Besides, the results also show that the GraphDE-a generally outperforms GraphDE-v, as the EM algorithm can give a tight approximation for the log-likelihood.
Fig.
3 illustrates GraphDE’s performance on three real-world datasets.
In this setting, we mainly study the debiasing performance w.r.t.
different biased ratios, so the backbone for robust GNN baselines and
GraphDE are all fixed as GAT.
As shown in the figure, GraphDE consistently outperforms baselines on the three datasets for all the biased ratios.
The backbones’ performance drops drastically as the biased ratio gets higher, and the robust GNN baselines can mitigate this effect to some extend.
In comparison, GraphDE performs much more stable.
This advantage is more pronounced when the biased ratio is large, e.g. GraphDE surpasses all the other baselines by nearly 5% on Collab when the biased ratio is 0.2.
An interesting observation is that GraphDE can also achieve a better test accuracy when the biased ratio is 0, implying that the pure training set also violates the i.i.d.
assumption.
Detection results.
Tab. 2 reports GraphDE’s detection performance on three real-world datasets.
As can be seen from the table, the max softmax score (MSP) achieves low detection performance.
It suggests that vanilla GNNs are poorly calibrated [35] and it is of critical need to develop new graphlevel OOD detectors.
The results show that GraphDE performs the best on 7 out of 9 benchmarks.
Specifically, on MNIST-75sp, GraphDE surpasses the strongest baseline by approximately 10% for
AUROC, while also pulls down the false positive rate (FPR95).
4.3 Further Study (Q2)
Impact of prior ratio.
We select different prior ratios 1 − p ( e ) to study its impact on debiasing performance.
As shown in the Fig. 4(a), the ID test accuracy doesn’t fluctuate much from 0.05 to 0.7, but drops drastically when prior ratio goes higher than 0.7.
Besides, we also find that the test accuracy peaks at prior ratio 0.1 for biased ratio 0.1, and 0.2 for biased ratio 0.2, exactly when L kl becomes 0.
Ablation on L reg .
We remove L reg in the objective to study the structure estimation model’s impact on debiasing performance.
As illustrated in Fig. 4(b), we can find that the test accuracy drops obviously at all biased ratios.
Specifically, The test accuracy drops over 6% at biased ratio 0.25.
The results prove the structure estimation model’s capability of down-weighting outliers in training data.
Ablation on L cls .
We remove L cls to study the influence of the classification model on the OOD detection performance.
As shown in Fig. 4(c), we plot the detection AUROC on the testing dataset w.r.t.
different biased ratios.
The figure illustrates that the AUROC generally drops as the biased ratio gets higher, since the outliers will hurt the fitting performance of the structure estimation model.
Typically, we can see that the blue line (without L cls ) is consistently lower than the red line, with the
[Figure] [Figure]
gap getting larger as the biased ratio goes higher.
Specifically, the performance gap is approximately 3% when biased ratio is 0.3.
The results prove that the classification model can help down-weight training outliers, thereby promoting the detection performance of the structure estimation model.
Training procedure.
We plot GraphDE’s test accuracy after every training epoch on ID and OOD (i.i.d. to training outliers) datasets.
As shown in Fig.
5(a), the ID test accuracy simply fluctuates, rises, and converges.
In comparison, the trend of OOD test accuracy can be divided into two phases.
As separated by the green dashed line, we can see the red line first increases then decreases.
Generally, in the increasing phase, GraphDE is optimized to better fit the outliers in the training data.
However, in the decreasing phase, as the outliers are down-weighted by a large extend, GraphDE focuses on fitting ID data.
This transition prevents overfitting of outliers and promotes the ID test accuracy.
Visualization of inferred probability.
We visualize the recognition model’s inferred probability q ϕ ( e = 1 |
A, X, y ) on the training dataset.
As shown in Fig.
5(b), ID data is assigned with a higher probability on both datasets.
In particular, most outliers are assigned with probability at around 0 on
SPMotif.
This directly indicates that the outliers are down-weighted during the training procedure.
Conclusion and Outlook
In this paper, we have provided a viable approach to the problem of debiased learning and OOD detection over graph data.
Specifically, a generative process is modeled to capture distribution shifts of graph data.
By introducing a variational recognition model to infer the environment variable and two-component mixed generative models, the learning objective for GraphDE is derived, which can identify and down-weight outliers during training, as well as inducing an effective OOD detector on new data.
While our framework is mainly designed for graph data where both the debiased learning and OOD detection remain relatively under-explored, it can be potentially generalized to other data formats like images or texts, which we leave for future studies.
Acknowledgement
This work was partly supported by National Key Research and Development Program of China (2020AAA0107600), National Natural Science Foundation of China (61972250, 72061127003), and
Shanghai Municipal Science and Technology (Major) Project (2021SHZDZX0102, 22511105100).
Checklist
For all authors... (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?
[Yes] (b) Did you describe the limitations of your work?
[Yes] See Appendix G. (c) Did you discuss any potential negative societal impacts of your work?
[Yes] See Appendix G. (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes] 2.
If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
[Yes] (b) Did you include complete proofs of all theoretical results?
[Yes] See Appendix A. 3.
If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[Yes] See Appendix E for dataset information.
See the supplementary for the codes.
They will be released public upon publication.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
[Yes] See Appendix D and Appendix E. (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[Yes] See experiment section.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
[Yes] See Appendix D. 4.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
[Yes] See Appendix E. (b) Did you mention the license of the assets?
[N/A] (c) Did you include any new assets either in the supplemental material or as a URL?
[N/A] (d) Did you discuss whether
and how consent was obtained from people whose data you’re using/curating?
[N/A] (e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content?
[N/A] 5.
If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?
[N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
[N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
[N/A]