A Graph Is More Than Its Nodes: Towards Structured Uncertainty-Aware Learning on Graphs
Abstract
Current graph neural networks (GNNs) that tackle node classification on graphs tend to only focus on nodewise scores and are solely evaluated by nodewise metrics.
This limits uncertainty estimation on graphs since nodewise marginals do not fully characterize the joint distribution given the graph structure.
In this work, we propose novel edgewise metrics, namely the edgewise expected calibration error (ECE) and the agree/disagree ECEs, which provide criteria for uncertainty estimation on graphs beyond the nodewise setting.
Our experiments demonstrate that the proposed edgewise metrics can complement the nodewise results and yield additional insights.
Moreover, we show that GNN models which consider the structured prediction problem on graphs tend to have better uncertainty estimations, which illustrates the benefit of going beyond the nodewise setting.
Introduction
Learning from graph-structured data has gained increasing attention from the deep learning community and numerous graph neural networks (GNNs) have been introduced to cope with this new paradigm [7, 25, 17, 28].
Commonly used GNN models like Graph Convolution Network (GCN) [7] and Graph Attention Network (GAT)
[25] produce nodewise scores from which good classification results can be achieved.
On the other hand, learning on graphs poses particular challenges for uncertainty estimation.
Given the graph structure, the node predictions are no longer independent samples.
Instead, we have a structured prediction problem [15] and should also account for their interdependencies.
In other words, high-quality uncertainty estimation for GNNs requires reliable estimation of the joint output distribution.
Hence considering only nodewise marginal output distributions is insufficient.
From this structured prediction perspective, we see a considerable limitation of existing works on uncertainty-aware learning for GNNs, namely that they only employ nodewise metrics as evaluation criteria.
Given the abundant existing work on uncertainty-aware learning for standard multi-class classification [20, 3, 19], many ideas have been adapted to the GNN case.
This includes approaches based on Bayesian formulations [29, 4], evidential learning [30, 22], as well as post-hoc calibration methods [24, 27, 5].
However, for estimating the quality of predictive uncertainty, these approaches directly use the nodewise metrics from the multi-class classification literature, which are intended for i.i.d.
test samples and ignore the graph structure.
To tackle this issue, we investigate novel uncertainty estimation metrics that consider the interdependencies in the graph structure.
Specifically, we propose the edgewise expected calibration error
3( p 1 ; p 2 ; p 3 )
Nodewise
Edgewise
Chain
Cycle (0; 1; 1)   (2 / 3; 2 / 3; 2 / 3) 1 /1 /(0 .
results and offer best our knowledge have not yet been proposed for graph learning
Equipped with the proposed edgewise metrics, we compare uncertainty estimation of GNN models which simply produce nodewise scores, e.g., GCN and GAT, with models which explicitly tackle the structured prediction problem, like Graph Markov Neural Network (GMNN) [17] and Explicit
Pairwise Factorized Graph Neural Network (EPFGNN) [28].
While naive mean field [15] can approximate predictions based on nodewise scores to the joint output distribution, we observe in practice that approaches treating graph learning as structured prediction problems tend to have better uncertainty estimation.
This demonstrates the benefit of going beyond the nodewise setting, especially for the uncertainty estimation of GNNs.
Uncertainty estimation on graphs beyond nodewise criteria
Given a graph G = ( N , E ) with c output classes for each node, the task of node classification aims at predicting the labels y U
= { y i } i ∈ U of the test
nodes U ⊆
N given the labels y L
=
{ y i } i ∈
L of the training nodes L = N \ U and the input features x = { x i } i ∈N .
Ideal uncertainty estimation is achieved when the predicted joint distribution ˆ p ( y U | x, y L ) matches the true joint distribution p ( y U | x, y L ) .
However, this condition cannot be verified in practice, for the following two reasons:
It is impossible to estimate p ( y U | x, y L ) with a single set of ground-truth labels y ∗ U ; 2.
The joint distributions are intractable given their exponential size w.r.t.
the node count.
Instead, one can consider weaker but more tractable alternatives based on marginal distributions.
Existing works [24, 27, 5] consider the confidence calibration of nodewise marginal distributions and measure the quality of uncertainty estimation using their expected calibration error (ECE) as defined by Guo et al.
[3]: Denote (ˆ y i ) i ∈ U = argmax y
U ˆ p ( y U | x, y L ) the class predictions and ˆ c i = max l ˆ p ( y i =
l | x, y L ) the confidence of node i , we partition the test nodes into m bins ( B n 1 , . . .
, B nm ) with B nk = { j ∈ U | k − 1 m <
ˆ c j ≤ km } , the nodewise ECE is defined as ECE
[Equation]
k
[Equation]
| [Equation] | U | (cid:12)(cid:12) acc ( B nk ) − conf ( B nk ) (cid:12)(cid:12) , with (1) acc ( B nk ) =
|
B nk | (cid:88)
i ∈
B nk
(ˆ y i =
y ∗ i ) and conf ( B nk ) =
|
B nk | (cid:88) i ∈
B nk ˆ c i .
2.1 Edgewise expected calibration error
An important drawback of the nodewise ECE is that it neither reflects the dependency w.r.t graph structure nor the interdependencies between the nodes.
To remedy this, we consider the confidence calibration of edgewise marginals, and propose the following edgewise expected calibration error:
Denote E U
=
( i, j
) ∈
E| i, j ∈ U } the subset of edges constrained to test nodes, and for edge ( i,
j ) ∈
E U denote
ˆ c
i,j = max l,m ˆ p ( y i =
l, y j
= m | x, y L ) its confidence, we construct a similar edge binning ( B e 1 , . . .
, B em ) with B ek = { ( i,
j ) ∈
E U | k − 1 m <
ˆ c i,j ≤ km } and define the edgewise
ECE as
[Equation]
Figure 1 provides a concrete example that shows the dependency of edgewise ECE w.r.t graph structures.
Similar issues also exist for other nodewise metrics like negative log-likelihood (NLL) and Brier score.
Their edgewise extensions are also defined in Appendix B.
Along with the quantitative results of edgewise ECE, we can also plot its corresponding reliability diagram [3] for edgewise marginal predictions.
They provide additional insights by displaying the over/under-confident tendency of the predictions at each confidence level.
It is possible to go beyond the edgewise setting and calibrate marginal distributions of larger subgraphs.
We focus on the edgewise metrics in this work because it is a minimal extension that accounts for the graph structure and has manageable complexity.
Moreover, edge marginals have a special significance for probabilistic graphical models: they determine the exact joint distribution of a tree and correspond to the Bethe approximation in the general case [26].
2.2 Agree and disagree ECEs
While the edgewise ECE estimates the general quality of the predicted edge marginals, it can be interesting to consider some additional subsets.
Especially, since homophily plays an important role in graph learning [31], we find it insightful to separate the homophilous and heterophilous cases and consider the subsets of agreeing E A
= { ( i,
j ) ∈
E U | y ∗ i =
y ∗ j } and disagreeing edges
E D
= { ( i,
j ) ∈
E U | y ∗ i ̸
=
y ∗ j } .
This allows us to define the agree ECE ( ECE a ) and the disagree ECE ( ECE d ) similar to Equations (3) and (4) but computed from E A and E D respectively instead of E U :
• To compute the agree ECE, we construct the binning ( B a 1 , . . .
, B am ) for the agreeing edge set E A with B ak = { ( i, j
) ∈
E A | k − 1 m <
ˆ c i,j ≤ km } .
The agree ECE is defined as
[Equation]
• Analogously, we construct the binning ( B d 1 , . . .
, B dm ) for the disagreeing edge set E D with
[Equation]
d
[Equation]
j [Equation]
D [Equation] m <
ˆ c i,j ≤ k m } .
The disagree ECE is defined as
[Equation]
1 (cid:0) ˆ y j = y ∗ j (cid:1) and conf ( B dk
[Equation] = | [Equation]
(cid:88) i ∈ [Equation] [Equation]
Experiments
We conduct a series of experiments to empirically assess the proposed edgewise metrics and analyze the uncertainty estimation of GNNs.
We focus on solving the node classification problem on commonly used benchmark datasets Cora [18], Citeseer [18], and Pubmed [13].
In terms of GNN learning frameworks, we compare commonly used GCN [7] and GAT [25] which directly produce nodewise output scores against GMNN [17] and EPFGNN [28] which explicitly address the structured prediction problem formulation.
We analyze the quality of their predictive uncertainty using the nodewise, edgewise, and agree/disagree ECE.
[Table]
Calculation of the edgewise metrics requires estimating the edgewise marginal distributions.
GMNN [17] and EPFGNN [28] model the joint output distribution from which the edge marginals can be inferred.
For GCN [7] and GAT [25], we assume that they represent the joint output distribution as the product of nodewise marginal distributions, i.e., a naive mean field [15] assumption is made.
This means that their predicted edgewise marginals are simply products of the neighboring node marginals.
Further details about the experimental setup are provided in Appendix C.
We summarize the results in Table 1.
Additional results, including the corresponding test accuracies and plots of reliability diagrams, are included in Appendix D.
Nodewise and edgewise expected calibration errors Firstly, we consider the nodewise and edgewise ECE results.
Both metrics provide indications for the general quality of uncertainty estimation.
From the results in Table 1, we observe two noticeable findings:
• GNNs which address node classification as structured prediction (GMNN and EPFGNN) tend to have better uncertainty estimation, in terms of both nodewise and edgewise ECEs, compared to models like GCN and GAT which simply output nodewise scores.
• There is a significant correlation between nodewise and edgewise ECEs.
This is not surprising, as good uncertainty estimations should make both metrics small.
That said, the edgewise ECE has a different focus w.r.t.
the nodewise ECE, resulting the performance ranking of the Pubmed at odds with each other.
Agree and disagree ECEs To further analyze the edgewise probabilistic predictions, we partition the edge set into agreeing and disagreeing subsets.
We evaluate the quality of their predictive uncertainty separately using agree and disagree ECEs.
• Structured prediction approaches achieve better agree ECEs and worse disagree ECEs.
Given the fact that Cora, Citeseer, and Pubmed are all assortative graphs where neighbors tend to agree with each other, these methods seem to focus on calibrating the agreeing edges to reach good overall uncertainty estimation.
• GAT consistently yields the best disagree ECEs.
This might be an effect of its attention mechanism which allows for distinct treatments of the agreeing and disagreeing edges.
• Reliability diagrams from Appendix D indicate that predictions on agreeing edges tend to be under-confident.
On the contrary, the predictions on
disagreeing edges tend to be over-confident given that GNNs fail to produce correct high-confidence predictions.
Discussions
In this work, we demonstrate the benefits of going beyond the nodewise setting for better uncertaintyaware graph learning.
We discuss the need for structured evaluation metrics and propose edgewise metrics that offer additional insights.
And our experiments show that tackling the structured prediction on graphs can help the uncertainty estimation.
Our experiments also highlight the well-known challenge of graph learning in heterophilous scenarios.
This is currently an active area of research [31] and existing work is less established.
We hope our proposed edgewise metrics can help the related research, and in general bolster the development and evaluation of new approaches for uncertainty-aware learning on graphs.
Acknowledgments and Disclosure of Funding
This work was supported by the Munich Center for Machine Learning (MCML) and by the ERC
Advanced Grant SIMULACRON.
The authors would like to thank the anonymous reviewers for their constructive feedback.
A Nodewise and edgewise ECEs for concrete examples
To compare nodewise and edgewise ECE and show the dependency of edgewise ECE w.r.t.
the graph structure, we do a case study for two simple graphs shown in Figure
Left: a chain and a cycle, both having three binary nodes ( N = U = { 1 , 2 , 3 } ) with labels represented by their white or blue color ( { w, b } ), and both having the same ground-truth labels ( y ∗ 1 , y ∗ 2 , y ∗ 3 ) =
( w, b, b ) .
We use the naive mean field approximation and factorize the predicted joint distribution ˆ p ( y 1 , y 2 , y 3 )
=
ˆ p ( y 1 )ˆ p ( y 2 )
ˆ p ( y 3 ) into product of nodewise predictions, and denote p i
= ˆ p ( y i =
b ) the predicted probability of node i having blue label.
In the following, we compute the nodewise and edgewise
ECEs using a single bin on the two graphs for the predictions listed in Figure 1 Right.
As both graphs have the same node settings, their nodewise ECEs are identical for any given prediction.
( p 1 , p 2 , p 3 ) =
This is the perfect forecast where the true labels are correctly predicted with full certainty, and we reach zero nodewise and edgewise ECEs for both graphs; ( p 1 , p 2 , p 3 ) = ( 23 , 23 , 23 ) :
This is a uniform prediction case that is nodewise calibrated.
For both graphs, the label predictions are (ˆ y 1 , ˆ y 2 , ˆ y 3 ) =
( b, b, b ) and the nodewise accuracy and confidence terms are computed as
[Equation]
Thus the nodewise ECE for both graphs is
[Equation]
For the chain case, the edgewise predictions are wrong for edge (1 , 2) and right for (2 , 3) , and both edgewise predictions have confidence 23 × 23 = 49 , thus its edgewise ECE is
[Equation]
For the cycle case, all three edges (1 , 2) , (1 , 3) , (2 , 3) have confidence 49 and only the edge (2 , 3) has correct label prediction, thus its edgewise ECE is
[Equation]
( p 1 , p 2 , p 3 ) =
This is a non-uniform prediction that has zero edgewise ECE on the chain and nonzero edgewise ECE on the cycle.
For both graphs, the label predictions are again (ˆ y 1 , ˆ y 2 , ˆ y 3 )
= ( b, b, b ) and their common nodewise ECE is computed as
[Equation]
For the chain case, one out of the two edges is correctly predicted, and its edgewise ECE is
[Equation]
while for the cycle case, one out of its three edges is correctly predicted and its edgewise
ECE is computed as
[Equation]
The above examples highlight the difference between nodewise and edgewise ECE.
Particularly, the edgewise ECEs depend on the graph structure while nodewise ECEs are structure agnostic and only depend on the node settings.
B Definitions of other edgewise metrics
Apart from ECE, other edgewise metrics can also be formulated.
We provide the formulations for some common metrics in this section.
Accuracy While the commonly used nodewise accuracy is defined as
[Equation]
we can also define the edgewise accuracy
[Equation]
which computes the ratio of edges with correct label predictions.
Negative log-likelihood The nodewise negative log-likelihood (NLL), which recovers the joint NLL divided by | U | under the naive mean field assumption, is defined as
[Equation]
Similarly, we can define the edgewise NLL as
[Equation]
Brier score The nodewise Brier score has the definition
[Equation]
20) and we can formulate the edgewise Brier score as Brier
[Equation]
= [Equation] Note that the agree/disagree variants of the above edgewise metrics can be derived analogously by replacing E U with E A and E D respectively.
Moreover, while we focus on the ECE metric from Guo et al.
[3], other variants of ECE exist [10] and their edgewise and agree/disagree variants can be formulated in a similar manner.
C Details of experimental setup C.1 Dataset statistics
[Table]
We evaluate our models on common citation networks: Cora [18], Citeseer [18] [Table] and Pubmed [13].
A summary of the dataset statistics is shown in Table 2.
The edge homophily ratio from Zhu et al.
[31] defines the fraction of edges in a graph which connects nodes with the same class label y ∗ .
As we measure the homophily level of the test nodes U , the edge homophily ratio is equivalent to the agreeing edges E A over test edges E U .
Its complement is the disagreeing edges that we want to predict.
[Equation]
We define E U by constraining the connected nodes that need to be both test nodes. If a test node does not have a neighbor which belongs to the test nodes, it does not contribute to the edgewise metrics. We design an index to indicate the percentage of test nodes kept in E U :
[Equation] of test nodes which are kept in E [Equation] K ( E U ) close to 1 implies fewer nodes are omitted.
Similarly the index can be applied to E A and E D .
C.2 Details for model training
We randomly split the nodes to 15% as observed and 85% to unobserved set.
We follow a training framework as [9, 5] to divide the observed set into three-fold where the two portions ( 10% ) are the training set and the other portion ( 5% ) is the validation set.
For all the datasets, we do 5 random data splits, three-fold cross-validation per split, and together with 5 random model initializations.
This results in 75 runs for each experiment.
The models and training framework are implemented in
PyTorch [16] and PyTorch Geometric [2].
We choose the commonly used GCN [7] and GAT [25] as the baseline models which do not consider the structured dependency between output nodes.
For GCN, we use 2 layers with 64 hidden units.
For GAT, we use 2 layers with 8 attention heads, and each consists of 8 hidden units.
Dropout [21] is applied between layers with dropout rate 0 .
We train both models by minimizing NLL using
Adam optimizer [6].
The learning rate is set to 0 .
01 with the weight decay 5e-4.
We set the maximum epochs as 2000 and stop the training if the model does not improve on the validation set over the past 100 epochs.
For the structured prediction models we choose GMNN [17] and EPFGNN [28].
Both models are trained with the EM framework [14] to tackle the partially observed scenario of node classification on graphs and maximize the evidence lower bound instead of the intractable observed log-likelihood.
GMNN consists of one GCN as GNN backbone and another GCN to approximate the Conditional
Random Field (CRF) [11].
Both GCNs have the same hyperparameters as the baseline GCN.
We follow the training pipeline as the original paper.
We first pre-train the GNN backbone forepochs For M step we maximize the pseudolikelihood function [1] and update the approximated CRF with 100 epochs.
For E step we update the GNN backbone for 100 epochs to approximate the true joint distribution.
We observe the approximated CRF can achieve better performance than the GNN backbone by drawing multiple samples.
In practice we draw 32 samples and compute the mean as the prediction for evaluation.
EPFGNN consists of one GCN backbone and a Markov Random Field (MRF) [8] which models the label dependency explicitly using a shared compatibility matrix.
The GCN backbone produces as output the unary energies of the MRF.
Same as GMNN, we first pretrain the GCN backbone for 200 epochs.
For M step, we train the GCN backbone and MRF jointly by maximizing piecewise training objective [23].
Following Wang et al.
[28], the training stops if the model does not improve its accuracy on the validation set over 10 epochs.
For E step, we update a proposal distribution using mean field to approximate the true joint distribution.
To get edge marginals from the MRF, we perform loopy belief propagation [12] with 100 iterations for evaluation.
D Additional experimental results
In this section [Table] we collect additional experimental results.
Table 3 summarizes the classification results in terms of nodewise, edgewise, agree, and disagree accuracies.
And Figures 2, 3 and 4 display the reliability diagrams for GNN predictions on Cora, Citeseer and Pubmed. [Table]
Note the nodewise accuracies reported in Table 3 have significant discrepancies compared to the ones reported in Wang et al.
This is caused by two main factors:
• At test time we employ the loopy belief propagation inference which also estimates edgewise marginals, while in Wang et al.
[28] naive mean field is used instead; • We have a different experimental setup and splitting (see Appendix C).
We report the corresponding results for EPFGNN using naive mean field in Table 4, which are more in line with the ones from Wang et al.
We see that switching from naive mean field to loopy belief propagation yields better ECEs but worse accuracies.
The decrease in accuracies might be explained by the mismatch of the inference method since naive mean field is used doing training
[Table]
[Table] [Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]