E NERGY - BASED O UT - OF -D ISTRIBUTION D ETECTION FOR G RAPH N EURAL N ETWORKS
Anonymous authors
Paper under double-blind review
A BSTRACT
Representation learning on semi-structured data, e.g., graphs, has become a central problem in deep learning community as relational structures are pervasive in real situations and induce data inter-dependence that hinders trivial adaptation of existing approaches in other domains where the inputs are assumed to be i.i.d.
sampled.
However, current models in this regime mostly focus on improving testing performance of in-distribution data and largely ignores the potential risk w.r.t.
out-of-distribution (OOD) testing samples that may cause negative outcome if the model is overconfident in prediction on them.
In this paper, we identify a provably effective OOD discriminator based on an energy function directly extracted from a graph neural network trained with standard supervised classification loss.
This paves a way for a simple and efficient OOD detection model for GNN-based semi-supervised learning on graphs, which we call GNN-Safe .
It also has nice theoretical properties that guarantee an overall distinguishable margin between the detection scores for in-distribution and OOD samples, which, more critically, can be further strengthened by a non-learning-based structured propagation scheme.
Extensive experiments over five real-world datasets validate the practical efficacy of the proposed model for detecting various OOD instances that are inter-connected in a graph with up to 17 . 0% improvement on average AUROC over competitive peer models and without sacrificing in-distribution testing accuracy.
I NTRODUCTION
Real-world applications often require machine learning systems to interact with an open world, violating the common hypothesis that testing and training distributions are identical.
One critical challenge that stands in the way for trustworthy AI systems is how to equip the deep learning models with enough capability of realizing what they don’t know , i.e., detecting out-of-distribution (OOD) data on which the models are expected to have low confidence (Amodei et al., 2016; Liang et al., 2018).
This is fairly important when it comes to safety-critical applications such as medical diagnosis (Kukar, 2003), autonomous driving (Dai & Van Gool, 2018), etc.
A surge of recent works have explored various effective methods for OOD detection in vision where data points are considered to be independently sampled (Hendrycks & Gimpel, 2016; Bevandi´c et al., 2018; DeVries & Taylor, 2018;
Hein et al., 2019; Hsu et al., 2020; Sun et al., 2021; Bitterwolf et al., 2022).
The issue, however, is that many of real-world data are inherently inter-dependent: the data generation of each instance may involve the dependence on others in a system.
The inter-dependence can sometimes be embodied by observed structures that connect instance nodes as a graph (Belkin et al., 2006), which requires the model to capture the inter-connecting nature of data-generating distributions for reasonable OOD detection (Ma et al., 2021; Wu et al., 2022).
The structural interconnections between data points on graphs would nevertheless make OOD data become more obscure and difficult to distinguish than independently sampled image data studied in vision tasks.
Another obstacle for graph-based OOD detection is that the learning problems are often encountered with scarce labeled data and (relatively) abundant unlabeled data, i.e., in a semi-supervised learning setting (Weston et al., 2012; Yang et al., 2016; Kipf & Welling, 2017).
In contrast, current methods for OOD detection often train a neural classifier with sufficient labeled data, and the well-trained classifier can produce good representation enabling the OOD discriminator to make informative decision on given inputs.
However, the scarcity of labeled training data in graph domain requires the model to exploit useful information from the observed structures that may reflect the underlying geometry behind data for better generalization (Bronstein et al., 2017; Chong et al., 2020).
In this paper, we investigate into out-of-distribution detection for semi-supervised learning on graphs, where the model needs to handle data inter-dependence induced by the graph and learn from limited labeled data in the training set.
Our methodology is built on current state-of-the-art models, which follow the pioneering work (Kipf & Welling, 2017) adopting graph neural networks (GNNs) as an encoder for node representation/prediction that accommodates the structural information, yet purely focus on in-distribution testing performance.
As an important step to enhance the reliability of GNN models against OOD testing instances, we identify an intrinsic OOD discriminator from a GNN classifier trained with standard learning objective.
The key insight of our work is that standard GNN classifiers possess inherent good capability for detecting OOD samples (i.e., what they don’t know ) from unknown testing observations, with details of the model as described below.
◦ Simplicity and Generality: The out-of-distribution discriminator in our model is based on an energy function that is directly extracted through simple transformation from the predicted logits of a GNN classifier trained with standard supervised MLE loss on in-distribution data.
Therefore, our model can be efficiently deployed in practice, i
e., it does not require training a graph generative model for density estimation or any extra OOD discriminator.
Also, the model keeps a general form, i
e., the energy-based detector is agnostic to specific GNN models and can in principle enhance the reliability for arbitrary off-the-shelf GNNs against OOD data.
◦ Theoretical Soundness: Despite simplicity, our model can be provably effective for yielding distinguishable scores for in-distribution and OOD inputs, which can be further reinforced by an energy-based belief propagation scheme, a non-learning-based approach for boosting the detection consensus over graph topology.
We also discuss how to properly incorporate an auxiliary regularization term when training data contains additional OOD observation as outlier exposure, with double guarantees for preserving
in-distribution learning and enhancing out-of-distribution reliability.
◦ Practical Efficacy: We apply our model to extensive node classification datasets of different properties and consider various OOD types.
When training with standard cross-entropy loss on pure in-distribution data and testing on OOD detection at inference time, our model consistently outperforms SOTA approaches with an improvement of up to 12 .
9% on average AUROC; when training with auxiliary OOD exposure data as regularization and testing on new unseen OOD data, our model outperforms the strong competitors with up to 17 . 0% improvement on average AUROC.
B ACKGROUND
Semi-supervised learning on graphs.
We consider a set of instances with indices i ∈ { 1 , 2 , · · · , N } = I whose generation process involves inter-dependence among each other.
The instance-level inter-dependence is represented by an observed graph G =
( V, E ) where
V =
{ i | 1 ≤ i ≤ N } denotes the node set containing all the instances and E = { e ij } denotes the edge set.
The observed edges induce an adjacency matrix A
= [ a ij ] N × N where a ij = 1 if there exists an edge connecting nodes i and j and 0 otherwise.
Moreover, each instance i has an input feature vector denoted by x i ∈ R D and a label y i ∈ { 1 , · · · , C } where D is the input dimension and
C denotes the class number.
The N instances are partially labeled and we define I s (resp.
I u ) as the labeled (resp. unlabeled) node set, i.e., I
=
I s ∪ I u .
The goal of semi-supervised learning on graphs is to train a node-level classifier f with ˆ Y = f
( X, A ) , where X =
[ x i ] i ∈I and ˆ Y =
[ˆ y i ] i ∈I , that predicts the labels for in-distribution instances in I u .
Out-of-distribution detection.
Besides decent predictive performance on in-distribution testing nodes (sampled from the same distribution as training data), we expect the learned classifier is capable for detecting out-of-distribution (OOD) instances that has distinct data-generating distribution with the in-distribution ones.
More specifically, the goal of OOD detection is to find a proper decision function G such that for any given input x :
[Equation]
where G x = ( { x j } j ∈N x , A x ) denotes the ego-graph centered at node x , N x is the set of nodes in the ego-graph (within a certain number of hops) of x on the graph and A x is its associated adjacency matrix.
Our problem formulation here considers neighbored nodes that have inter-dependence with the centered node throughout the data-generating process for determining whether it is OOD.
That being said, the OOD decision for each input is dependent on other nodes in the graph, which is fairly different from existing works in vision that assume i.i.d.
inputs and independent decision for each.
Energy-based model.
Grathwohl et al.
(Grathwohl et al., 2020) found that a neural classifier for i.i.d.
inputs is intrinsically an energy-based model (EBM) (Ranzato et al., 2007).
Assume a classifier as h ( x ) : R D → R C which maps an input instance to a C -dimensional logits, and a softmax function over the logits gives a predictive categorical distribution: p ( y | x ) =
[Equation]
] , where h ( x )
[Equation]
c
[Equation]
queries the c -th entry of the output.
The energy-based model builds a function E ( x , y ) :
R D
→ R that maps each input instance with an arbitrarily given class label to a scalar value called energy , and induces a Boltzmann distribution: p ( y | x ) =
[Equation]
x
[Equation]
( x ) , where the denominator (cid:80) y ′
e − E ( x ,y ′ ) is often called the partition function which marginalizes over y .
The equivalance between an EBM and a discriminative neural classifier is established by setting the energy as the predicted logit value E ( x , y )
=
− h
( x ) [ y ] .
And, the energy function E ( x ) for any given input can be
[Equation]
The energy score E ( x ) can be an effective indicator for OOD detection due to its nice property: the yielded energy scores for in-distribution data overall incline to be lower than those of OOD data (Liu et al., 2020).
However, to the best of our knowledge, current works on energy-based modeling as well as its application for OOD detection are mostly focused on i.i.d.
inputs and the power of the energy model for modeling inter-dependent data has remained under-explored.
P ROPOSED M ETHOD
We describe our new model for out-of-distribution detection in semi-supervised node classification where instances are inter-dependent.
We will first present a basic version of our model based on energybased modeling from a data-generating perspective that involves non-i.i.d.
generation (Section 3.1).
Then we will introduce an advanced version with a novel energy-based belief propagation method for boosting OOD detection performance at inference time (Section 3.2).
Finally, we extend the model training to leverage auxiliary OOD training data (if available) for further enhancing the OOD detection capability (Section 3.3).
The proofs for all theoretical results are deferred to Appendix A.
3. E NERGY - BASED OOD D ETECTION WITH D ATA D EPENDENCE
In semi-supervised learning on graphs, one critical point is how to leverage the data inter-dependence as extra information for solving node-level prediction.
The most commonly adopted model class follows the classic idea of Graph Neural Networks (GNNs) (Scarselli et al., 2008) which iteratively aggregate neighbored nodes’ features for updating centered nodes’ representations.
Such an operation is often called message passing that plays a central role in modern GNNs for encoding topological patterns and accommodating global information from other instances.
In specific, assume z ( l ) i as the representation of instance node i at the l -th layer and the Graph Convolutional Network (GCN) (Kipf & Welling, 2017) updates node representation through a layer-wise normalized feature propagation:
[Equation]
where ˜ A is an adjacency matrix with self-loop on the basis of A , D is its associated diagonal degree matrix, W ( l ) is the weight matrix at the l -th layer and σ is a non-linear activation.
With L layers of graph convolution, the GCN model outputs a C -dimensional vector z ( L ) i as logits for each node, denoted as h θ ( x i , G x i )
=
z ( L ) i , where θ denotes the trainable parameters of the GNN model h .
Induced by the GNN’s updating rule, here the predicted logits for instance x are denoted as a function of the L -order ego-graph centered at instance x , denoted as G x .
The logits are used to derive a categorical distribution with the softmax function for classification:
[Equation]
By connecting it to the EBM model that defines the relation between energy function and probability density, we can obtain the energy form induced by the GNN model, i.e., E ( x , G x , y ; h θ )
= − h
θ
( x , G x ) [ c ] .
More importantly, without changing the parameterization of the graph neural network, we can express the free energy function E ( x , G x ; h θ ) in terms of the denominator in Eqn. 4:
[Equation]
The non-probabilistic energy score given by Eqn.
5 accommodates the information of the instance itself and other instances that have potential dependence based on the observed structures.
Following the works that target in-distribution testing performance, e.g. Kipf & Welling (2017);
Velickovic et al.
; Wu et al.
(2019), the GNN model is often trained in a supervised manner by minimizing the negative log-likelihood of labeled training data
[Equation]
where D in denotes the distribution where the labeled portion of training data is sampled.
We next show that, for any GNN model trained with Eqn. 6, the energy score can provably be a well-posed indicator of whether the input data is from in-distribution or out-of-distribution.
Proposition 1.
The gradient descent on L sup will overall decrease the energy E ( x , G x ; h θ ) for any in-distribution instance ( x , G x , y ) ∼ D in .
The proposition shows that the induced energy function E ( x , G x ; h θ ) from the supervisedly trained
GNN guarantees an overall trend that the energy scores for in-distribution data will be pushed down.
This suggests that it is reasonable to use the energy scores for detecting OOD inputs at inference time for safety consideration.
However, as mentioned before, due to the scarcity of labeled data, the supervised training may not guarantee good generalization on new data, even those samples from D in .
That being said, the model might still mistakenly output low (resp. high) energy for out-of-distribution (resp.
in-distribution) samples from new observations.
3. C ONSENSUS B OOSTING WITH E NERGY - BASED B ELIEF P ROPAGATION
The crux for enhancing the generalization ability lies in how to leverage the unlabeled data exposed to the model.
In principle, the unlabeled data could help to restructure the model hypothesis and recognize the geometric structures behind data (Belkin et al., 2006).
Inspired by the label propagation (Zhu et al., 2003), a classic non-parametric semi-supervised learning algorithm, we propose energy-based belief propagation that iteratively propagates estimated energy scores among inter-connected nodes over the observed structures.
We define E (0) =
[ E ( x i , G x i ; h θ )] i ∈I as a vector of initial energy scores for nodes in the graph and consider the propagation updating rule
[Equation]
where 0 < α < 1 is a parameter controlling the concentration on the energy of the node itself and other linked nodes.
After K -step propagation, we can use the final result for estimation, i.e., ˜ E
( x i , G x i ; h θ ) =
E ( K ) i , and the decision criterion for OOD detection can be
[Equation]
where τ is the threshold and the return of 0 indicates the declaration for OOD.
The rationale behind the design of belief propagation is to accommodate the physical mechanism in data generation with instance-wise interactions.
Since the input graph could reflect the geometric structures among data samples with certain proximity on the manifold, it is natural that the generation of a node is conditioned on its neighbors, and thereby connected nodes tend to be sampled from similar distributions 1 .
Given that the energy score of Eqn.
5 is an effective indicator for out-of-distribution, the propagation in the energy space essentially imitates such a physical mechanism of data generation, which can presumably reinforce the confidence on detection.
Proposition 2 formalizes the overall trend of enforcing consensus for instance-level OOD decisions over graph topology.
Proposition 2.
For any given instance x i , if the averaged energy scores of its one-hop neighbored nodes, i.e., (cid:80) j a ij E ( k − 1) j (cid:80) j a ij , is less (resp.
more) than the energy of its own, i.e., E ( k − 1)
i , then the updated energy given by Eqn.
7 yields E ( k )
i
[Equation]
Proposition 2 implies that the propagation scheme would push the energy towards the majority of neighbored nodes, which can help to amplify the energy gap between in-distribution samples and the OOD.
This result is intriguing, suggesting that the energy model can be boosted by the simple propoagation scheme at inference time without any extra cost for training.
We will verify its effectiveness through empirical comparison in experiments.
3. E NERGY -R EGULARIZED L EARNING WITH B OUNDING G UARANTEES
The energy used for OOD detection at inference time is directly extracted from a GNN classifier trained with the supervised classification loss, and it is not explicitly optimized for separating indistribution and OOD samples.
The potential issue is that the classifier might still learn a complex mapping from inputs to labels that minimizes the empirical risk to zero yet would lead to overconfidence on OOD inputs outside the training support.
To resolve this problem, we can add hard constraints on the energy gap through a regularization loss L reg that bounds the energy for indistribution data, i.e., labeled instances in I s , and OOD data, i.e., another set of auxiliary training instances I o from a distinct distribution.
The obstacle, however, is that the introduction of an extra loss term may interfere with the optimization for the original objective L sup .
In other words, the goal of energy-bounded regularization might be inconsistent with that of supervised learning, and the former may change the global optimum to somewhere the GNN model gives sub-optimal fitting on the labeled instances.
The ideal choices for
L reg need to satisfy two requirements: i) the energy scores induced by the GNN model minimizing
L reg , denoted as ˜ E ( x , G x ; h θ ∗ ) , can desirably separate in-distribution and OOD inputs; ii) the predicted logits associated with ˜ E ( x , G x ; h θ ∗ ) also minimizes L sup , i.e., the two objectives L sup and
L reg consistently facilitate in-distribution learning.
In the next proposition, we present a general result as a cornerstone for the energy regularization in our model.
Proposition 3.
For any energy regularization L reg , satisfying that the GNN model h θ ∗ minimizing
L reg yields the energy scores upper (resp.
lower) bounded by t in (resp.
t out ) for in-distribution (resp.
OOD) samples, where t in < t out are two margin parameters, then the corresponding softmax categorical distribution p ( y | x , G x ; h θ ∗ ) also minimizes L sup .
The implication of this proposition is important, suggesting that we can construct a proper L reg whose optimal energy scores are guaranteed for detecting OOD inputs and stay consistent with the supervised training of the GNN model.
This motivates us to consider the final training objective as
L sup + λ L reg , where λ is a trading weight, and the regularization term L reg that explicitly constrains the absolute energy values can be defined as
[Equation]
The above loss function will push the energy values within [ t in , t out ] to be lower (resp.
higher) for in-distribution (resp.
out-of-distribution) samples, and this is what we expect.
We thereby obtain a new training objective which strictly guarantees a desired discrimination between in-distribution and OOD instances in the training set, and in the meanwhile, does not impair model fitting on in-distribution data.
Admittedly, one disadvantage of the energy-regularized learning is that it requires additional
OOD data exposed during training.
As we will show in the experiments, nonetheless, the energy regularization turns out to be not a necessity in quite a few cases in practice.
E XPERIMENTS
The goal of our experiments is to test the reliability of the model against out-of-distribution data from testing set.
The model performance can be measured by its capability of discriminating OOD testing samples from the in-distribution ones.
In the meanwhile, we do not expect the degradation of the model’s performance on in-distribution testing instances.
That being said, an ideal model should desirably handle OOD detection without sacrificing the in-distribution accuracy.
4. S ETUP
Implementation details We call our model trained with pure supervised loss as GNN-Safe and the version trained with additional energy regularization as GNN-Safe-r .
We basically set the propagation layer number K as 2 and weight α as 0.5.
For fair comparison, the GCN model with layer depthand hidden size 64 is used as the backbone encoder for all the model.
We defer to more details for hyper-parameter settings, architectures and training to Appendix B.
Datasets and Splits Our experiments are based on five real-world datasets that are widely adopted as node classification benchmarks: Cora , Amazon-Photo , Coauthor-CS , Twitch-Explicit and ogbn-Arxiv .
Following a recent work on OOD generalization over graphs (Wu et al., 2022), we generally consider two ways for creating OOD data in experiments, which can reflect the realworld scenarios involving OOD instances inter-connected in graphs. 1) Multi-graph scenario:
the
OOD instances come from a different graph (or subgraph) that is dis-connected with any node in the training set.
2) Single-graph scenario: the OOD testing instances are within the same graph as the training instances, yet the OOD instances are unseen during training.
Specifically, ◦ For Twitch , we use the subgraph DE as in-distribution data and other five subgraphs as OOD data.
The subgraph ENGB is used as OOD exposure for training of GNN-Safe-r .
These subgraphs have different sizes, edge densities and degree distributions, and can be treated as samples from different distributions Wu et al.
◦
For Arxiv that is a single graph, we partition the nodes into two-folds: the papers published before 2015 are used as in-distribution data, and the papers after 2017 are used as OOD data.
The papers published at 2015 and 2016 are used as OOD exposure during training.
◦
For Cora , Amazon and Coauthor that have no clear domain information, we synthetically create OOD data via three ways.
Structure manipulation: use the original graph as in-distribution data and adopt stochastic block model to randomly generate a graph for OOD data.
Feature interpolation: use random interpolation to create node features for OOD data and the original graph as in-distribution data.
Label leave-out: use nodes with partial classes as in-distribution and leave out others for OOD.
For in-distribution data in each dataset, only partial nodes’ labels are exposed for supervised training and another subset of nodes is used for validation.
Other nodes are used as in-distribution testing data.
We provide detailed dataset and splitting information in Appendix B.1.
Metric We follow common practice using AUROC, AUPR and FPR95 as evaluation metrics for
OOD detection.
The in-distribution performance is measured by the Accuracy on testing nodes. In
Appendix B.3 we provide more information about these evaluation metrics.
Competitors We compare with two classes of models regarding OOD detection.
The first family of baseline models focus on OOD detection in vision where inputs (e.g., images) are assumed to be i.i.d.
sampled: MSP (Hendrycks & Gimpel, 2016), ODIN (Liang et al., 2018),
Mahalanobis (Lee et al., 2018), OE (Hendrycks et al., 2019b) (using OOD exposure for training), Energy and Energy FineTune (Liu et al., 2020) which also uses OOD exposure for training.
We replace their convolutional neural network backbone with the GCN encoder used by our model for processing graph-structured data.
The second family of baseline models are particularly designed for handling OOD data in graph machine learning and we compare with two SOTA approaches: the Graph-based Kernel Dirichlet
GCN method GKDE (Zhao et al., 2020) and Graph Posterior Network GPN (Stadler et al., 2021).
We provide more implementation details for these model in Appendix B.2.
4. C OMPARATIVE R ESULTS
How does GNN-Safe(-r) perform compared with competitive methods?
In Table 1 and 2 we report the experimental results of GNN-Safe(-r) in comparison with the competitive models.
We
[Table]
[Figure] [Figure]
These ablation results demonstrate that the proposed energy propagation and regularized training can both be helpful for enlarging the energy margin between in-distribution and OOD inputs.
found that GNN-Safe-r consistently outperform all competitors by a large margin throughout all cases, increasing average AUROC by 12 .
8% (resp.
and reducing average FPR95 by 44 .
8% (resp.
21 . 0% ) on Twitch (resp.
Cora-Structure ) and maintain comparable in-distribution accuracy.
This suggests the superiority of our energy-based model for OOD detection on graphs.
Furthermore, without using the energy regularization, GNN-Safe still achieves very competitive
OOD detection results, increasing average AUROC by 12 .
9% on Cora-Structure .
In particular, it even outperforms OE on all datasets and exceeds Energy FT in most cases on Cora , Amazon and Coauthor , though these two competitors use extra OOD exposure for training.
This shows a key merit of our proposed model: it can enhance the reliability of GNNs against OOD data without costing extra computation resources.
To quantitatively show the efficiency advantage, we compare the training and inference times in Table 8 in Appendix C. We can see that GNN-Safe is much faster than the SOTA model GPN during both training and inference and as efficient as other competitors.
These results altogether demonstrate the promising efficacy of GNN-Safe(-r) for detecting OOD anomalies in real graph-related applications where the effectiveness and efficiency are both key considerations.
How does the energy propagation improve inference-time OOD detection?
We study the effectiveness of the proposed energy-based belief propagation by comparing the performance of
Energy and GNN-Safe (the Energy model in our setting can be seen as ablating the propagation module from GNN-Safe).
Results in Table 1 and 2 consistently show that the propagation scheme indeed brings up significant performance improvement.
Furthermore, in Fig.
1 we compare the energy scores produced by GNN-Safe (the middle column) and Energy (the left column).
The figures show
[Table]
[Figure]
that the propagation scheme can indeed help to separate the scores for in-distribution and OOD inputs.
Such results also verify our hypothesis in Section 3.2 that the structured propagation can leverage the topological information to boost OOD detection with data inter-dependence.
How does the energy-regularized training help OOD detection?
We next investigate into the effectiveness of the introduced energy-regularized training.
In comparison with GNN-Safe, we found that GNN-Safe-r indeed yields better relative performance across the experiments of Table 1 and 2, which is further verified by the visualization in Fig.
1 where we compare the energy scores of the models trained w/o and w/ the regularization loss (the middle and right columns, respectively) and see that GNN-Safe-r can indeed produce larger gap between in-distribution and OOD data.
Furthermore, we notice that the performance gains achieved in Table 2 are not that considerable as
Table 1.
In particular, the AUROCs given by GNN-Safe and GNN-Safe-r are comparable on Amazon and Coauthor .
The possible reason is that the OOD data on the multi-graph dataset Twitch and large temporal dataset Arxiv is introduced by the context information and the distribution shift may not be that significant, which makes it harder for discrimination, and the energy regularization can significantly facilitate OOD detection.
On the other hand, the energy regularization on auxiliary OOD training data may not be necessary for cases (like other three datasets) where the distribution shifts between OOD and in-distribution data are intense enough.
4. A NALYSIS OF H YPER - PARAMETERS
We next analyze the impact of using different encoder backbones and hyper-parameters, to shed more lights on the practical efficacy and usage of our proposed model.
Will different encoder backbones impact the detection results?
Fig.
2 compares the performance of MSP, Energy, Energy FT, GNN-Safe and GNN-Safe-r w.r.t.
using different encoder backbones including MLP, GAT (Velickovic et al., 2018), JKNet (Xu et al., 2018), MixHop (Abu-El-Haija et al., 2019) on Cora with three OOD types.
We found that different backbones indeed impact the detection performance and in particularly, GNN backbones bring up significantly better AUROC than MLP due to their stronger expressiveness for encoding topological features.
However, we observe that the relative performance of five models are mostly consistent with the results in Table 2 where we use
GCN as the backbone and two GNN-Safe models outperform other competitors in nearly all the cases.
In the right figure, even with MLP backbone, two GNN-Safe models perform competitively as the counterparts with GNN backbones, which also validates the effectiveness of our energy propagation scheme that helps to leverage topological information over graph structures.
How do the propagation depth and self-loop strength impact the inference-time performance?
Fig. 3(a) plots the results of GNN-Safe-r with propagation step K increasing fromto 64 and self-loop strength α varing between 0.1 and 0.9.
As shown in the figure, the AUROC scores first go up and then degrade as the propagation steps increase.
The reason is that a moderate number of propagation steps can indeed help to leverage structural dependence for boosting OOD detection yet too large propagation depth could potentially overly smooth the estimated energy and make the detection scores indistinguishable among different nodes.
Similar trend is observed for α which contributes to better performance with the value set as a moderate value, e.g., 0 . 5 or 0 .
This is presumably because too large α would counteract the effect of neighbored information and too small α would overly concentrate on the other nodes.
How do the regularization weight and margins impact the training?
We then discuss the performance variation of GNN-Safe-r w.r.t.
the configurations of t in , t out and λ in Fig.
3(b) and (c).
We found that the performance of GNN-Safe-r is largely dependent on λ which controls the strength of energy regularization and its optimal values also vary among different datasets, though we merely tune it with a coarse searching space { 0 . 01 , 0 . 1 , 1 . 0 } .
Besides, a proper setting for the margin t in and t out yields stably good performance while too large margin would lead to performance degradation.
R ELATED W ORKS
Out-of-distribution detection for neural networks Detecting OOD samples on which the models should have low confidence has been extensively studied by recent literature.
One line of works define OOD uncertainty scores based on pre-trained models (optimized with supervised loss), e.g., softmax confidence score (Hendrycks & Gimpel, 2016) or maximum logits (Hendrycks et al., 2019a).
While these models are observed to sometimes produce unsatisfactory results (Hein et al., 2019), they are free from either fine-tuning or training with auxiliary regularization loss for OOD detection.
In contrast, another line of works train a robust model against OOD anomalies via additionally regularizing the classifier towards yielding low confidence on auxiliary OOD training data (Bevandi´c et al., 2018; Liu et al., 2020; Mohseni et al., 2020).
Different from training a discriminative classifier as is done by the above-mentioned models, the third category of models turn to generative modeling for detecting OOD examples (Ren et al., 2019).
However, the aforementioned methods mostly target the situations where instances (e.g., images) are assumed to be i.i.d.
generated, and largely ignore the widely existing problem scenario that involves inter-dependence of data generation.
Graph neural networks & out-of-distribution learning on graphs Graph neural networks come into the spotlight due to its provable and practical power for learning effective representation of inter-connected data (Kipf & Welling, 2017).
The issue, however, is that many recent evidence suggests that GNN models are sensitive to distribution shifts and could perform undesirably on OOD testing data (Koh et al., 2021), which urges the community to focus on the reliability of GNN learning.
One research path orthogonal to ours is to enhance the generalization capability of GNNs on OOD testing data (Bevilacqua et al., 2021; Ma et al., 2021; Wu et al., 2022), while our work explores another important problem, i.e., how to make GNNs realize what they don’t know at training time and capable for detecting OOD data at inference time.
The major difference from existing OOD detection methods, e.g., (Zhao et al., 2020; Stadler et al., 2021; Bazhenov et al., 2022), is that the
OOD discriminator in our model is directly extracted from the predicted logits of GNNs trained with standard supervised MLE loss, which keeps a simple and general form that is agnostic to GNN architectures.
Also, our model is built on off-the-shelf discriminative GNN classifiers and is free from training complicated graph generative models for density estimation or any extra OOD discriminator.
C ONCLUSIONS
We show in this paper that graph neural networks trained with supervised loss can intrinsically be an effecitve OOD discriminator for detecting OOD testing data on which the model should avoid prediction.
Our model is simple, general, and well justified by theoretical analysis.
We also verify its practical efficacy through extensive experiments over real-world datasets with various OOD data type.
We hope our work can shed new insights on out-of-distribution learning on graph data and also serve as a practically useful approach for enhancing the reliability of GNNs as well as empowering the applicability of GNN models to real applications where safety considerations matter.
A P ROOFS FOR T ECHNICAL R ESULTS A. P ROOF FOR P ROPOSITION
Proof.
The gradient of L nll w.r.t.
θ is given by ∂ L nll
[Equation]
i ∈I tr [Equation] We can see from the above equation that the training procedure overall minimizing the first-order gradient of L nll will decrease the energy score E ( x i , G x i , y i ; h θ ) and increase the energy E ( x i , G x i , c ; h θ ) .
In a general sense, the energy value produced by the trained model tends to become lower for any in-distribution instance ( x , G x , y ) ∼ D in .
A. P ROOF FOR P ROPOSITION
Proof.
If
[Equation]
, we can re-write the update in Eqn.
7 as the scalar-form for each instance:
[Equation]
The opposite result can be similarly proved for the case (cid:80) j a ij E ( k − 1)
j (cid:80) j a ij > E ( k − 1) i .
A. P ROOF FOR P ROPOSITION
Proof.
We define E ( x , G x ; h θ ∗ ) as the energy yielded by the model h
θ ∗ minimizing the regularization loss L reg .
And, h θ † denotes the model yielding the optimal predictive softmax distribution that minimizes the supervised classification loss L sup , i.e.,
[Equation]
Notice that E ( x , G x ; h θ ) =
− log (cid:80)
Cc =1 e h θ ( x , G x ) .
Therefore we have
[Equation]
The above relationship suggests that the energy E ( x , G x ; h θ ∗ ) is secretely equivalent to the energy induced by the predicted logits h θ † ( x , G x ) − E ( x , G x ; h θ ∗ )
+ E ( x , G x ; h θ † ) .
We can thereby denote the predictive softmax distribution of E ( x , G x ; h θ ∗ ) as
[Equation]
The above predictive distribution exactly minimizes L sup as defined by Eqn.
We thus have proven that the optimal energy that minimizes L reg intrinsically induce the predictive softmax distribution that minimizes L sup .
On the other hand, given our assumption that the optimal energy function can well distinguish the scores for in-distribution and OOD inputs, we thus obtain proper choices for the regularization loss L reg whose optimal energy satisfies the two requirements mentioned upfront in Section 3.3.
B E XPERIMENTAL D ETAILS
We supplement experiment details for reproducibility.
Our implementation is based on Ubuntu 16.04.6, Cuda 10.2, Pytorch 1.9.0 and Pytorch Geometric 2.0.3.
Most of the experiments are running with a NVIDIA 2080Ti with 11GB memory, except that for cases where the model requires larger
GPU memory we use a NVIDIA 3090 with 24GB memory for experiments.
B. D ATASET I NFORMATION
The datasets used in our experiment are all public available as common benchmarks for evaluating graph learning models.
For ogbn-Arxiv , we use the preprocessed dataset and data loader provided by the OGB package 2 .
For other datasets, we use the data loader provided by the Pytorch Geometric package 3 .
Cora (Sen et al., 2008) is a citation network where each node denotes a published paper and each edge indicates the citation relationship.
The goal is to predict each paper’s topic as the label class.
This dataset contains 2,708 nodes, 5,429 edges, 1,433 features and 7 classes.
We use three different ways for synthetically creating OOD data as mentioned in Section 4.1.
For training/validation/testing splits on in-distribution data, we use follow the semi-supervised learning setting by Kipf & Welling (2017) and use its provided splits.
Amazon-Photo (McAuley et al., 2015) is an item co-purchasing network on Amazon where each node denotes a product and each edge indicates that two linked products are frequently purchased together.
The node label denotes the category of the product.
This dataset contains 7,650 nodes, 238,162 edges, 745 features, and 8 classes.
Similar to Cora , we use three ways for synthetically creating OOD data.
And, for in-distribution data, we follow the common practice (Kipf & Welling, 2017) and use random splits with 1:1:8 for training/validation/testing.
Coauthor-CS (Sinha et al., 2015) is a coauthor network of computer science.
Nodes denote authors and they are connected as edges if the two authors co-authored a paper.
The goal is to predict the respective field of study for authors based on their papers’ keywords as node features.
This dataset contains 18,333 nodes, 163,788 edges, 6,805 features, and 15 classes.
We also use the aforementioned three ways to construct OOD data and for in-distribution data, we follow the common practice using random splits with 1:1:8 for training/validation/testing.
Twitch-Explicit (Rozemberczki & Sarkar, 2021) is a multi-graph dataset, where each subgraph corresponds to a social network in one region.
Nodes in this dataset represent game players on Twitch and edges indicate two users follow each other.
Node features are embeddings of games played by the Twitch users, and the label class indicates whether a user streams mature content.
The node numbers of the subgraphs are ranged from 1,912 to 9,498, with edge numbers from 31,299 to 153,138 and shared feature dimension 2,545.
As mentioned in Section 4.1 we use subgraph DE as in-distribution data where we use random splits with 1:1:8 for training/validation/testing.
Besides, subgraph ENGB is used as OOD exposure and ES, FR, RU are used for OOD testing data.
ogbn-Arxiv (Hu et al., 2020) is a relatively large graph dataset that records the citation information from 1960 to 2020.
Each node denotes a paper with a subject area as the label to predict.
Edges indicate citation relationship and each node has a 128-dimensional feature vector obtained by the word embeddings of its title and abstract.
As described in Section 4.1 we use the time information for partitioning in-distribution and OOD data.
For the in-distribution portion, we also use 1:1: random splits for training/validation/testing.
Notice that here our partition for the in-distribution and
OOD data disables us to use the original splitting by Hu et al.
(2020) which used time information for splitting training and testing nodes.
In our setting, the random splitting also guarantees that the training, validation and testing nodes (of in-distribution data) come from an identical distribution that is distinct from that of OOD data.
B. I MPLEMENTATION D ETAILS
Architectures As mentioned in Section 4.1, our basic encoder backbone is set as a GCN, and we also consider other models such as MLP, GAT, JKNet and MixHop for further discussions.
We summary the architectural information of all the encoder backbones used in our experiments as follows.
• GCN: 2 GCNConv layers with hidden size 64, ReLU activation, self-loop and batch normalization.
• MLP: 2 fully-connected layers with hidden size 64 and ReLU activation.
• GAT: 2 GATConv layers with hidden size 64, ELU activation, head number [2 , 1] and batch normalization.
• JKNet: 2 GCNConv layers with hidden size 64, ReLU activation, self-loop and batch normalization.
The jumping knowledge module is using max pooling.
• MixHop: 2 MixHop layers with hop number 2, hidden size 64, ReLU activation, and batch normalization.
Hyper-parameters As described in Section 4.1, we consider the following hypermeters as a default setting: the propagation step K = 2 and the propagation self-loop weight α = 0 .
For GNN-Safe, we only consider grid-search for the learning rate within { 0 . 1 , 0 . 01 , 0 .
001 } using validation set.
For GNN-Safe-r, we use the validation set for additionally search for margin hyper-parameters t in ∈ {− 9 , − 7 , − 5 } , t out ∈ {− 1 , − 2 , − 3 , − 4 } and the regularization weight λ ∈ { 0 . 01 , 0 . 1 , 1 . 0 } .
Training Details
The model GNN-Safe is trained with the supervised classification loss on labeled training data, and GNN-Safe-r is trained with the supervised loss plus the regularization loss on the
OOD training data.
We use the supervised classification loss (i.e., cross-entropy) on validation set as the indicator of epoch selection for reporting the testing result.
More specifically, in each run, we train the model with 200 epochs as a fixed budget and report the testing performance produced by the epoch where the model yields the lowest classification loss on validation data.
Competitors For competitor models, we use their implementations provided by the original papers and do some adaptation for our setting.
As mentioned in Section 4.1, we use the same encoder backbone GCN (with the same hidden size 64 and layer number 2) for baselines MSP, ODIN,
Mahalanobis, Energy and Energy FT.
For GKDE and GPN, we use their public implementation, refer to their hyper-parameter settings reported in the paper and do some fine-tuning for different datasets.
B. E VALUATION M ETRICS
We provide more details concerning the evaluation metrics we used in our experiment.
We follow the common practice of OOD detection literature and use AUROC, AUPR and FPR95 for evaluating the performance for detecting OOD data in the testing set.
These metrics are independent of the threshold τ .
Specifically, AUROC refers to the Area Under the Receiver Operating Characteristic (ROC) curve.
The ROC curve presents a trade-off between the true positive rate and the false positive rate under different thresholds in 0 ∼ 1 .
Intuitively, the AUROC can be understood as the probability that for any pair of a positive example (i.e., in-distribution sample) and a negative example (out-of-distribution sample), the positive one has a larger estimated score than the negative.
However, AUROC is not always an ideal metric, especially when the positive class and negative class are imbalanced.
Under such a case, the AUPR adjusts for different positive/negative base rates.
AUPR refers to the Area
Under the PR curve which presents the trade-off between the precision and recall under different thresholds in 0 ∼ 1 .
Furthermore, FPR95 is also a widely used metric which refers to the False
Positive Rate at 95% true positive rate (TPR).
FPR95 can be interpreted as the probability that a out-of-distribution sample is misclassified as in-distribution when the TPR is as high as 95%.
C A DDITIONAL E XPERIMENT R ESULTS
We supplement more experimental results in this section.
In specific, we report detailed OOD detection performance on each OOD dataset (different subgraphs for Twitch and different years for
Arxiv ) in Table 3 and 4 as complementary for Table 1 in the maintext.
Besides, in Table 5, 6 andwe report the AUROC/AUPR/FPR95 and in-distribution testing accuracy on Cora , Amazon and
Coauthor as complementary for Table 2 in the maintext.
In Table 8 we compare the training time per epoch and inference time of our model and the competitors.
Moreover, we compare the training time per epoch and inference time of GNN-Safe(-r) and the competitors in Table 8.
Also, we present more hyper-parameter analysis results in Fig.
3 including α , K , t in , t out and λ .
[Table] [Table]
[Table] [Table]
95.72 Energy FT 98.
GNN-Safe 99.
95.76 GNN-Safe-r 99.
[Table] [Table]
0.252 Energy FT 0.
0.218 GNN-Safe 0.
0.219 GNN-Safe-r 0.
[Figure]