What Makes Graph Neural Networks Miscalibrated?
Abstract
Given the importance of getting calibrated predictions and reliable uncertainty estimations, various post-hoc calibration methods have been developed for neural networks on standard multi-class classification tasks.
However, these methods are not well suited for calibrating graph neural networks (GNNs), which presents unique challenges such as accounting for the graph structure and the graph-induced correlations between the nodes.
In this work, we conduct a systematic study on the calibration qualities of GNN node predictions.
In particular, we identify five factors which influence the calibration of GNNs: general under-confident tendency, diversity of nodewise predictive distributions, distance to training nodes, relative confidence level, and neighborhood similarity.
Furthermore, based on the insights from this study, we design a novel calibration method named Graph Attention
Temperature Scaling (GATS), which is tailored for calibrating graph neural networks.
GATS incorporates designs that address all the identified influential factors and produces nodewise temperature scaling using an attention-based architecture.
GATS is accuracy-preserving, data-efficient, and expressive at the same time.
Our experiments empirically verify the effectiveness of GATS, demonstrating that it can consistently achieve state-of-the-art calibration results on various graph datasets for different GNN backbones.
Introduction
Graph-structured data, such as social networks, knowledge graphs and internet of things, have widespread presence and learning on graphs using neural networks has been an active area of research.
For node classification on graphs, a wide range of graph neural network (GNN) models, including GCN [14], GAT [40] and GraphSAGE [9], have been proposed to achieve high classification accuracy.
This said, high accuracy is not the only desideratum for a classifier.
Especially, reliable uncertainty estimation is crucial for applications like safety critical tasks and active learning.
Neural networks are known to produce poorly calibrated predictions that are either overconfident or under-confident [7, 41].
To mitigate this issue a variety of post-hoc calibration methods [7, 19, 43, 38, 8] have been introduced over the last few years for calibrating neural networks on standard multi-class classification problems.
However, calibration of GNNs, in the context of node classification on graphs, is currently still an underexplored topic.
While it is possible to apply existing calibration methods designed for multi-class classification to GNNs in a nodewise manner, this does not address the specific challenges of node classification on graphs.
Especially, node predictions in a graph are not i.i.d.
but correlated, and we are tackling a structured prediction problem [25].
A uniform treatment when calibrating node predictions would fail to account for the structural information from graphs and the non i.i.d. behavior of node predictions.
Our contribution.
In this work, we focus on calibrating GNNs for the node classification task [14, 40].
First, we aim at understanding the specific challenges posed by GNNs by conducting a systematic study on the calibration qualities of GNN node predictions.
Our study reveals five factors that influence the calibration performance of GNNs: general under-confident tendency, diversity of nodewise predictive distributions, distance to training nodes, relative confidence level, and neighborhood similarity.
Second, we develop Graph Attention Temperature Scaling (GATS) approach, which is designed in a way that accounts for the aforementioned influential factors.
GATS generates nodewise temperatures that calibrate GNN predictions based on the graph topology.
Third, we conduct a series of GNN calibration experiments and empirically verify the effectiveness of GATS in terms of calibration, data-efficiency, and expressivity.
Related work
For standard multi-class classification tasks, a variety of post-hoc calibration methods have been proposed in order to make neural networks uncertainty aware: temperature scaling (TS) [7], ensemble temperature scaling (ETS) [43], multi-class isotonic regression (IRM) [43], Dirichlet calibration [19], spline calibration [8], etc.
Additionally, calibration has been formulated for regression tasks [17].
More generally, instead of transforming logits after training a classifier, a plethora of methods exists that modify either the model architecture or the training process itself.
This includes methods that are based on Bayesian paradigm [12, 1, 6, 22, 42], evidential theory [33], adversarial calibration [37] and model ensembling [20].
One common caveat of these methods is the trade-off between accuracy and calibration, which oftentimes do not go hand in hand.
Post-hoc methods like temperature scaling, on the other hand, are accuracy preserving.
They ensure that the per node logit rankings are unaltered.
Calibration of GNNs is currently a substantially less explored topic.
Nodewise post-hoc calibration on GNNs using methods developed for the multi-class setting has been empirically evaluated by
Teixeira et al.
They show that these methods, which perform uniform calibration of nodewise predictions, are unable to produce calibrated predictions for some harder tasks.
Wang et al.
[41] observe that GNNs tend to be under-confident in contrast to the majority of multi-class classifiers, which are generally overconfident [7].
Based on their findings, Wang et al.
[41] propose the CaGCN approach, which attaches a GCN on top of the backbone GNN for calibration.
Some approaches improve the uncertainty estimation of GNNs by adjusting model training.
This includes Bayesian learning approaches [45, 10] and methods based on the evidential theory [46, 35].
Problem setup for GNN calibration
We consider the problem of calibrating GNNs for node classification tasks: given a graph G =
( V , E ) , the training data consist of nodewise input features { x i } i ∈V ∈ X and ground-truth labels { y i } i ∈L ∈ Y = { 1 , . .
, K } for a subset L ⊂ V of nodes, and the goal is to predict the labels { y i } i ∈U ∈ Y for the rest of the nodes U
=
V \ L .
A graph neural network tackles the problem by producing nodewise probabilistic forecasts ˆ p i .
These forecasts yield the corresponding label predictions ˆ y i :=
argmax y
ˆ p
i ( y ) and confidences ˆ c i := max y
ˆ p
i ( y ) .
The GNN is calibrated when its probabilistic forecasts are reliable, e.g., for predictions with confidence 0 .
8 , they should be correct 80% of the time.
Formally, a GNN is perfectly calibrated [41] if
[Equation]
In practice, we quantify the calibration quality with the expected calibration error (ECE) [27, 7].
We follow the commonly used definition from Guo et al.
[7] which uses a equal width binning scheme to estimate calibration error for any node subset N ⊂ V : the predictions are regrouped according to M equally spaced confidence intervals, i.e. ( B 1 , . .
. , B M ) with B m = { j ∈ N | m − 1 M < ˆ c j ≤ mM } , and the expected calibration error of the GNN forecasts is defined as
[Equation]
acc ( B m ) − conf ( B m ) (cid:12)(cid:12)(cid:12) , with (2) acc ( B m ) =
| B m | (cid:88) i ∈
{+B m 
+}
( y i =
ˆ y i ) and conf ( B m ) =
| B m | (cid:88) i ∈ B m ˆ c i .  Factors that influence GNN calibration To design calibration methods adapted to GNNs, we need to figure out the particular factors that influence the calibration quality of GNN predictions.
For this we train a series of GCN [14] and GAT [40] models on seven graph datasets: Cora [32], Citeseer [32], Pubmed [23], Amazon
Computers [34], Amazon Photo [34], Coauthor CS [34], and Coauthor Physics [34].
We summarize the dataset statistics in Appendix A.1.
Details about model training are provided in Appendix A.2 for reproducibility.
To compare with the standard multi-class classification case, we additionally train
ResNet-20 [11] models on the CIFAR-10 image classification task [16] as a reference.
Our experiments uncover five decisive factors that affect the calibration quality of GNNs.
In the following we discuss them in detail.
4.1 General under-confident tendency
[Figure]
confident predictions. In Figure 1 we plot the reliability diagrams [24] for results on different graph datasets using GCN.
Similar to Wang et al.
[41], we see a general trend of under-confident predictions for GNNs.
This is in contrast to the standard multi-class image classification case which has overconfident behavior.
Also, it is interesting to see that this under-confident trend can be more or less pronounced depending on the dataset.
For Coauthor Physics, the predictions are well calibrated and have no significant bias.
Results using GAT models lead to similar conclusions and are provided in Appendix B.1.
4.2 Diversity of nodewise predictive distributions
[Figure]
Contrary to the standard multi-class case [Figure] GNN outputs can have varying roles depending on their positions in the graph, which means that their output distributions could exhibit dissimilar behaviors.
This is empirically evident in Figure 2, where we visualize the entropy distributions of GCN output predictions v.s. the standard multi-class results (GAT results are available in Appendix B.2).
We see that the entropies of GNN outputs have more spread-out distributions, which indicates that they have distinct roles and behaviors in graphs.
In terms of GNN calibration [Figure] this observation implies that uniform node-agnostic adjustments like temperature scaling [7] might be insufficient for GNNs
[Figure]
whereas nodewise adaptive approaches could be beneficial
[Figure]
4.3 Distance to training nodes
[Figure]
observe that training nodes and their neighbors tend to be better A graph provides additional structural information for its nodes.
One insightful feature is the minimum distance to training nodes.
We discover that nodes with shorter distances, especially the training nodes themselves and their direct neighbors, tend to be better calibrated.
To evaluate the calibration quality nodewise
[Figure]
we propose the nodewise calibration error , which is based on the binning scheme used to compute the global expected calibration error (ECE) [27, 7]: for each node
[Figure]
we find its corresponding bin depending on its predicted confidence [Figure] and the calibration error of this bin is assigned to be its nodewise calibration error. Using this nodewise metric
[Figure] in Figure 3 we visualize the influence of minimum distance to training nodes on the nodewise calibration quality (c [Figure] f. Appendix B.3 for GAT results).
We see that nodes close to training ones typically have lower nodewise calibration error.
This suggests that minimum distance to training nodes can be useful for GNN calibration
[Figure]
4.4 Relative confidence level
[Figure] We observe that nodes which are less confident than their neighbors tend to have worse calibration.
Another important structural information is the neighborhood relation.
We find out that the relative confidence level δ ˆ c i of a node i , i.e., the difference between the nodewise confidence ˆ c i and the average confidence of its neighbors
[Equation]
has an interesting correlation to the nodewise calibration quality.
In Figure 4 we show the relation between the relative confidence level of a node and its nodewise calibration error (c.f. Appendix B. for GAT results).
Especially, We observe that nodes which are less confident than their neighbors tend to have worse calibration, and it is in general desirable to have comparable confidence level w.r.t. the neighbors.
For GNN calibration, the relative confidence level δ ˆ c i can be a useful node feature to consider. 5 Node homophily 0. C a li b . e rr o r
Cora 5 Node homophily 0.
C a li b . e rr o r
Citeseer 5 Node homophily 0. C a li b . e rr o r Pubmed 5 Node homophily 0.
C a li b . e rr o r
Computers 5 Node homophily 0. C a li b . e rr o r Photo 5 Node homophily 0.
C a li b .
e rr o r
CS 5 Node homophily 0.
C a li b . e rr o r Physics Furthermore, we find that different neighbors tend to introduce distinct influences.
For assortative graphs which are the focus of this work, we find out that calibration of nodes are affected by node homophily , i.e., whether a node tends to have the same label prediction as its neighbors.
For a node with n a agreeing neighbors and n d disagreeing ones, we measure the node homophily as
[Equation] [Equation]
where positive values indicate greater ratio of agree neighbors and vice versa.
Figure 5 summarizes the variation of nodewise calibration error w.r.t.
the node homophily for different graph datasets (c.f. Appendix B.5 for GAT results).
We find out that nodewise calibration errors tend to decrease significantly for nodes with strongly agreeing neighbors.
This suggests that neighborhood predictive similarity should be considered when doing GNN calibration.
Graph attention temperature scaling (GATS)
Based on the findings in Section 4, we design a new post-hoc calibration method, named Graph
Attention Temperature Scaling (GATS) , which is tailored for GNNs.
5.1 Formulation and design of GATS
To obtain a calibration method that is adapted to the graph structure G = ( V , E ) and reflects the observed influential factors in Section 4, the graph attention temperature scaling approach extends the temperature scaling [7] method to produce a distinct temperature T i for each node i ∈ V .
T i is then used to scale the uncalibrated nodewise output logits
z i and produce calibrated node predictions ˆ p i
[Equation]
Formulation of T i .
The nodewise temperature T i should address the five factors discussed in
Section 4.
We achieve this via the following considerations:
• We introduce a global bias parameter T 0 to account for the general under-confident tendency; • To tackle the diverse behavior of node predictions, we learn a nodewise temperature contribution τ i based on the predicted nodewise logits
z i ; • To incorporate the relative confidence w.r.t.
neighbors, we introduce δ ˆ c i from Eq.
4 as an additional contribution term scaled by a learnable coefficient ω ; • To model the influence of neighborhood similarity, we use an attention mechanism [39] to aggregate neighboring contributions τ j with attention coefficients α i,j depending on the output similarities between the neighbors i and j ; • Distance to training nodes is used to introduce a nodewise scaling factor γ i to adjust the node contribution and the aggregation process.
It is learnable for training nodes and their direct neighbors and fixed to 1 for the rest: [Equation] neighbor of training node
[Equation] [Figure]
where node 2 is a training node and node 1 is the target node which aggregates information from the the five factors discussed in Section produces
Putting together the above components, the nodewise temperature T i has the following expression:
[Equation]
Here we have a multi-head formulation where h indicates the h -th attention head, and ˆ n ( i ) denotes the neighbors of node i including self-loop.
We uses 8 heads ( H = 8 ), which works well in practice.
(c.f. Section 6.3.)
Defining τ hi .
Nodewise contribution τ hi are computed as the outputs of parameterized linear layers ϕ h ( · ; θ ) which take transformed nodewise output logits ˜
z
i as input
[Equation]
The transformed nodewise output logits ˜
z i are produced as follows: we first normalize the original logits z i to range [0 , 1] , then sort the classwise logits for each node.
This makes the linear layers ϕ h focus on the general logit distributions rather than class predictions.
A similar idea has been explored by Rahimi et al.
[30], where they show that intra order-preserving functions improve the model calibration.
Here we find out that this sorting-based transformation helps GATS to learn useful representations of nodewise contributions τ hi .
(c.f. Section 6.3.)
Defining α
i,j .
The attention coefficients α i,j are defined based on the neighbor similarity, which is determined by the inner product between rescaled nodewise logits
z i /γ i and z j /γ j .
Inspired by
Veli ˇ ckovi ´ c et al.
[40], we compute the attention coefficients α i,j as follows:
[Equation]
5.2 Calibration properties of GATS
Zhang et al.
[43] propose three desiderata for calibration methods: accuracy-preserving, data-efficient, and expressive.
GATS fulfills all of them:
• GATS is accuracy-preserving: since all node predictions are scaled by inverse temperatures 1 /T i which are positive scalars, the order of output logits is preserved; • GATS is data-efficient.
It is a parametric calibration model with C · H + 4 learnable parameters ( T 0 , ω, γ t , γ n , ( θ h ) 1 ≤ h ≤ H ).
According to Zhang et al.
[43], parametric methods are already data-efficient; • GATS is expressive, as it produces nodewise temperatures
T i adapted to the graph structure.
Experiments in Section 6.2 empirically confirm the data-efficiency and expressivity of GATS.
5.3 Comparison with CaGCN
It is interesting to compare our proposed GATS approach to the CaGCN method proposed by Wang et al.
[41], since both approaches aim at calibrating GNNs, and both make use of the graph structure to produce nodewise temperatures.
While CaGCN uses a GCN to generate nodewise temperatures straightforwardly, GATS uses an attention mechanism which differentiates the influence from various neighbors.
GATS also integrates a series of careful designs following the insights from the study in
Section 4.
Experiments in Section 6.1 shows that GATS tends to produce better calibration results compared to CaGCN.
Experiments
To evaluate the performance of GATS on GNN calibration and understand the effects of its designs, we conduct a series of experiments for baseline comparison and ablation study.
We use two representative
GNNs: GCN and GAT, which are trained on the seven aforementioned graph datasets plus a larger graph, CoraFull [2], for post-hoc calibration.
We display the ECE results with M = 15 bins. We use the expected calibration error (ECE) [27, 7] with M = 15 bins as an evaluation metric, and follow an experimental protocol similar to Kull et al.
[19, 18]: For all the experiments, we randomly split the labeled/unlabeled ( 15% / 85% ) data five times, and use three-fold internal cross-validation of the labeled data to train the GNNs and the calibrators.
We also utilize five random initializations, resulting in 75 total runs for each experiment. We provide detailed experimental settings in Appendix A.
6.1 Performance comparison
We benchmark GATS against existing baselines on a variety of GNN calibration tasks.
We compare
GATS with the following baselines:
• Temperature scaling (TS) [7] simply uses a global temperature to scale the logits.
• Vector scaling (VS) [7] scales the logits separately over the class dimension and additionally introduces a classwise bias for the recalibrated output logits.
• Ensemble temperature scaling (ETS) [43] learns a mixture of uncalibrated, TS-calibrated, and uniform probabilistic outputs.
• GCN as a calibration function (CaGCN) [41] is specifically designed for calibrating GNNs.
It uses a GCN on top to generate nodewise temperatures
Additionally, we also report the ECEs of uncalibrated predictions as a reference.
Among the above baselines, TS, VS, and ETS are calibration methods designed for standard classification cases and operate on nodes uniformly.
CaGCN on the other hand performs separate nodewise adjustments and uses the graph structure, similar to our proposed GATS approach.
For the post-hoc calibration experiments, we fix the weight of the trained GNN backbones and adjust the parameters of the calibration methods on the validation set.
Negative log-likelihood is chosen as the objective for the calibration process.
We provide details of method configurations and calibration settings in Appendix A.3.
Table 1 summarizes the calibration results.
Overall, we observe that GATS consistently produces well-calibrated predictions for all graph datasets and GNN backbones.
Except for the GAT model trained on Pubmed (3rd best) and the GCN model trained on Amazon Computers (2nd best), GATS achieves the highest calibration quality in all cases.
Also, it is interesting to see that for all cases the best result is achieved by methods which use the graph structure and produce adapted adjustments for different nodes.
This demonstrates the necessity of designing calibration methods that address the unique challenges posed by GNN calibration.
Although CaGCN can get the best results for Pubmed using GAT and Amazon Computers using
GCN, we see that its performance is rather unstable for different scenarios, and sometimes it can even produce worse calibration results than the uncalibrated baseline.
Using their proposed marginbased loss did not help in our settings.
We suspect that CaGCN might have an overly complex architecture for the task, and it cannot differentiate neighborhood influences with the common normalized adjacency matrix.
Our proposed GATS model does not have this issue.
It has consistent and good calibration performance in all cases.
[Table] We also observe that the results tend to have high variations, since GNN backbones tend to predict highly varying results when trained with different initial weights and random splits [34].
We ensure the reliability of the results by averaging over a total of 75 runs with various initial weights and random splits for each case.
6.2 Data-efficiency and expressivity of GATS
[Figure]
Furthermore, we analyze the data-efficiency and the expressivity of GATS for GNN calibration.
For this we reuse the GNN models trained on the
CoraFull dataset, and consider the influence of calibration sample size on the GNN calibration performance.
For comparison we also report the corresponding results using ensemble temperature scaling and CaGCN.
Figure 7 visualizes the results with GCN backbone.
The results for GAT backbone are summarized in Appendix D.
Overall, we see that GATS is both data-efficient and expressive.
It requires few calibration samples to get decent calibration performance.
This is in contrast to CaGCN which needs more than 5% of nodes for calibration to get acceptable results.
Compared to ETS, GATS is more expressive and has a considerably lower calibration error for
CoraFull, which is a large graph dataset.
6.3 Ablation study
To empirically analyze the effect of various GATS design choices, we conduct a series of ablation study experiments in this section.
Overall, we notice that all designs are advantageous and removing any of them leads to a general decrease in performance.
[Table]
Effect of global bias T 0 .
We consider the effect of global bias T 0 by comparing to a GATS variant without it (i [Table] e., setting T 0 = 0 in Eq.
Its results are collected in column “w/o T 0 ” of Table 2.
Overall we see that a learnable bias T 0 is beneficial in most cases.
Effect of nodewise scaling factor γ i .
The nodewise scaling factors γ i provide custom adjustment for training nodes and their neighbors.
Removing its influence can be done by fixing all γ i to one in
Eqs.
(8) and (10).
The results of the variant without γ i are recorded in column “w/o γ i ” of Table 2.
They are worse in general, suggesting that nodewise scaling factors γ
i are indeed helpful.
Effect of relative confidence level δ ˆ c i .
To evaluate the impact of introducing nodewise relative confidence level δ ˆ c i , we create a GATS variant where ω in Eq.
(8) is fixed to zero, which effectively removes the influence of the relative confidence level δ ˆ c i .
In column “w/o δ ˆ c i ” of Table 2 we have the results corresponding to this variant, which is in general slightly worse than the standard GATS which includes δ ˆ c i .
Effect of attention-based aggregation.
To understand the role played by the attention mechanism, we create a GATS variant which completely removes the attention related term ( (cid:80)
j ∈
ˆ n ( i ) α
i,j γ j τ hj ) from Eq.
Its performance is shown in column “w/o attention” of Table 2.
Again, we observe that removing the attention-based component tends to worsen the calibration performance.
Effect of logit sorting.
GATS uses normalized and sorted logits ˜
z i as input to generate nodewise temperature contributions τ hi .
And we find that this sorting transform is essential for learning good representations of τ hi .
In column “w/o sorting” of Table 2 we have the results for GATS variants which uses the logits without sorting to compute τ hi .
And we observe that this deteriorates the calibration performance.
[Table]
Effect of attention head count H .
Finally [Table] we analyze the influence of multi-head count H on the
GNN calibration results.
For this we run a series of experiments using GATS models with 1, 2, 4, 8, and 16 attention heads.
The results are collected in Table 3.
We see that for GCN backbones, GATS models with more attention heads tend to get better results.
However, for GAT backbones, usingheads results in worse performance compared to 8 heads.
Accounting also for the fact that doubling the attention head count effectively doubles the computational requirements, 8 attention heads is a decent general setting for GATS.
Conclusion
In this work, we tackle the GNN calibration problem.
We conduct a systematic study to analyze the calibration properties of GNNs predictions.
Our study reveals five influential factors and manifests the unique challenges raised by GNN calibration.
Based on the insights from our studies, we propose a novel calibrator, GATS, which accounts for the identified factors and is tailored for calibrating GNNs.
GATS is accuracy-preserving, data-efficient, and expressive at the same time.
Our experiments demonstrate that GATS achieves state-of-the-art performance for GNN calibration on various graph datasets and for different GNN backbones.
Our work focuses on the node classification tasks for assortative graphs, where neighbors tend to agree with each other.
It is thus important to realize that the validity of the conclusions from Sectionis limited to the assortative case, and might no longer hold for disassortative graphs [47, 29].
It can be an interesting future work to conduct similar studies for GNN calibration in the heterophilous case, especially when more established GNN architectures are available.
More generally, devising calibration methodologies for other graph learning tasks such as link prediction [44] and graph classification [4] could also be an interesting direction for future research. Acknowledgments and Disclosure of Funding This work was supported by the Munich Center for Machine Learning (MCML) and by the ERC
Advanced Grant SIMULACRON.
The authors would like to thank Nikita Araslanov for proofreading and helpful discussions, as well as the anonymous reviewers for their constructive feedback.
Checklist
For all authors... (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?
[Yes] (b) Did you describe the limitations of your work?
[Yes] c.f. Section 7.
(c) Did you discuss any potential negative societal impacts of your work?
[N/A] (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes] 2.
If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
[N/A] (b) Did you include complete proofs of all theoretical results?
[N/A] 3.
If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[Yes] (b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
[Yes] (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[Yes] (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
[Yes] 4.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
[Yes] (b) Did you mention the license of the assets?
[N/A]
(c) Did you include any new assets either in the supplemental material or as a URL?
[Yes] (d) Did you discuss whether and how consent was obtained from people whose data you’re using/curating? [N/A] (e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content?
[N/A] 5.
If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?
[N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable? [N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
[N/A]
We consider eight real-world graph datasets including citation networks Cora [32], Citeseer [32], Pubmed [23], Coauthor CS [34], Coauthor Physics [34], and CoraFull [2] together with Amazon co-purchase networks Computers [34] and Photo [34].
Table 4 summarizes their statistics.
[Table]
We report the homophily index proposed by Pei et al.
[29], which provides a global view of the neighborhood similarity for a graph.
Given a graph G =
( V , E ) , the homophily is defined as [Equation] i [Equation]
Number of node i ’s neighbors who have the same label as i
Number of i ’s neighbors .
A.2 Details of model training setup
We follow the setting of Shchur et al.
[34] to define GCN [14] and GAT [40] models.
Both models consist of 2 layers and the hidden dimension is fixed to 64.
For the multi-head layer in GAT, the number of attention heads is fixed to 8 with 8 hidden units per head.
We implement the models and training pipelines in PyTorch [28] and PyTorch Geometric [5].
All models are trained for a maximum of 2000 epochs, using early stopping with a patience of 100 epochs.
We choose Adam [13] as the optimizer with initial learning rate 0.01.
We add a weight decay of 5e-4 for Cora, Citeseer, and
Pubmed, and 0 for the rest.
We use stratified sampling to randomly select 15% of the nodes as observed set, mask out the output labels of the rest 85% of the nodes for test prediction, and ensure that the nodes with the same label are split proportionally.
Following Kull et al.
[19, 18], we further divide the labeled set with three-fold cross-validation.
The bigger portions (10%) are used as training sets and the rest (5%) are used as validation sets.
The GNN models (GCN and GAT) are trained on the training set, then used to predict the masked-out test set.
Figure 8 illustrates the aforementioned data partition in our experiments.
In total, we use 5 random data splits, three-fold cross-validation for each split, and 5 random model initializations per data partition, resulting in 75 total runs for each experiment.
[Figure]
A.3 Details of model calibration setup
We compare GATS with four scaling-based calibrators: temperature scaling (TS) [7], vector scaling (VS) [7], ensemble temperature scaling (ETS) [43] , CaGCN [41].
The calibrators are trained on the validation set using the negative log-likelihood (NLL) loss and validated on the training set for early stopping and hyperparameter search.
The optimizer configuration and the training schedule are the same as Section A.2.
We observe TS and VS using Adam with weight decay 0 achieves better performance than using L-BFGS [21] in the original implementation 3 .
For ETS, we follow the official implementation that uses Sequential Least SQuares Programming (SLSQP) [15].
For
CaGCN, we use a two-layer GCN with 16 hidden units and choose the hyperparameters following the original paper.
For GATS, we utilize one message passing layer and initialize T 0 , γ t , and γ n toand ω to 0.
We find the best hyperparameter using cross-validation.
Table 5 shows the search space for GATS hyperparameters.
[Table]
B Additional plots
Here we include additional plots which shows the corresponding factors influencing the calibration of GAT models (c.f. Section 4).
Overall, we reach the same conclusions as the GCN case.
B.1 General under-confident tendency for GAT
Figure 9 summarizes the GAT results.
We see a general tendency of under-confident predictions (plots above the diagonal) except for the Physics dataset, which differs from the overconfident behavior of multiclass image classification using CNNs.
[Figure]
B.2 Diversity of node distributions for GAT
Figure 10 shows the GAT results.
Compared to the standard classification case, predictions of GAT also tend to be more spread out
[Figure]
[Figure]
B.3 Effect of distance to training nodes for GAT
GAT result are shown in Figure 11.
We also see that training nodes and their neighbors tend to be better calibrated
[Figure]
B.4 Relative confidence level for GAT
The plots for the GAT case are shown in Figure 12.
Similar to the GCN case, we observe that nodes which are less confident than their neighbors tend to be poorly calibrated and it is in general desirable to have a comparable confidence level w
[Figure]
r
[Figure]
t. the neighbors.
[Figure]
B.5 Neighborhood similarity for GAT
Figure 13 shows tha GAT results.
Analogue to the GCN case, nodes with strongly agreeing neighbors tend to have lower calibration errors
[Figure]
C Additional calibration results
This section includes supplementary results with additional metrics and calibration methods.
C.1 Results using additional metrics
Variants of ECE Even though ECE [27, 7] is the most commonly used metric for measuring calibration, it has some limitations: (1) ECE only considers top-1 probabilistic output but can not reflect classwise calibration.
The binning-based estimator [27] is dependent on the choice of binning scheme.
To alleviate these disadvantages, we evaluate the trained calibrators using the classwise-ECE [19] and the kernel density estimation (KDE) based KDE-ECE [43] which is a binning-free metric:
• Classwise-ECE measures the gap between the classwise average prediction conf ( B m,k ) and the actual frequency of that class freq ( B m,k ) in M equally spaced bins across all classes k ∈ K :
[Equation]
where ˆ p
i,k denotes the probability of predicting class k for sample i in bin B m,k .
We compute classwise-ECE using M = 15 bins in our implementation.
• Instead of using a binning-based estimator, KDE-ECE uses a kernel function K h to estimate the accuracy ˜ π ( c ) given confidence prediction c and the marginal density function f ( c ) of the predictive confidence:
[Equation]
where i ∈ N ⊂ V denotes the evaluated node, and h is the bandwidth of the kernel function.
We follow the official implementation 4 of KDE-ECE, where the the Triweight
Kernel K h ( u ) = (1 /h ) 3532 (1 − ( u/h ) 2 ) 3 [3] on [ − 1 , 1] is chosen as the kernel function and bandwidth is calculated as h = 1 .
σ |N| − 1 / 5 [31] with σ being the standard deviation of the confidence.
The classwise-ECEs are summarized in Table 6, and the KDE-ECEs are collected in Table 7.
In general, we observe similar conclusions as in the confidence ECE case (c.f. Section 6.1): Overall
GATS achieves the state-of-the-art calibration results.
[Table]
[Table] Non calibration metrics Although not calibration metrics, we also report the results in terms of negative log-likelihood (Table 8) and Brier score (Table 9) for reference. C.2 Results for additional baselines
While in the main paper we focus on “temperature scaling style” methods which directly rescale the output logits
[Table]
here we compare with the following additional calibration methods which have different principles.
These methods are all designed for multi-class classification and do not consider the structural information of the graph. • Multi-class isotonic regression (IRM) [43] is a multi-class generalization of the nonparametric isotonic regression method; [Table] [Table]
• Calibration using spline (Spline) [8] fits the calibration function with splines; • Dirichlet calibration (DIR) [19] uses the Dirichlet distribution to model the distribution of probabilistic outputs.
It also employs an off-diagonal and intercept regularization (ODIR); • Order invariant calibration (OI) [30] is the order-invariant intra order-preserving model.
It uses sorted output logits as calibration input and builds up a neural network with special structures to preserve the accuracy and the intra order of the predicted logits
[Table]
The authors of spline calibration specify how to calibrate a specific class or a chosen top- r class
[Table]
and in their implementation 5 they focus on calibrating the top- 1 class.
However, it is not clear how to adjust the rest of the predictions to ensure valid probabilistic predictions after calibration.
We adopt a heuristic which proportionally rescales the non top- 1 output probabilities so that the calibrated probabilistic output sums up to one.
Also, the authors wrongly claimed that calibrating the topscore “does not alter the classification accuracy” [8].
{+In practice, the score after calibration might no longer remain top- 
+}
and the predictions could be altered.
For Dirichlet calibration, we find out that the scaling factors λ, µ of ODIR affect the performance and need to be tuned depending on the dataset.
Thus we do a hyperparameter search for each dataset with search space (0 . 01 , 0 .
Table 10 summarizes the calibration results in terms of ECE.
Note that we do not include the results for Dirichlet calibration on CoraFull because it fails to calibrate the GNN backbones and significantly deteriorates the predictive accuracies ( < 10% v.s. > 60% before calibration).
Overall we observe that GATS still achieves the state-of-the-art performance compared to the additional baselines for GNN calibration.
[Table]
C.3 Accuracies of calibration methods
Since many baseline calibration methods are not accuracy-preserving, in Table 11 we additionally report their test accuracies.
Accuracy preserving methods (GATS, TS, ETS, CaGCN, OI) have the same accuracies as the uncalibrated case, which is also reported for reference.
D Data efficiency and expressiveness of GATS: GAT results
Figure 14 shows the results for the GAT case.
We see that GATS is also data efficient and expressive when calibrating GAT models.
E CaGCN results discussion
While the ECEs of CaGCN in its original paper are promising [41], we observe that the ECEs of
CaGCN are often unstable and sometimes even worse than that of the uncalibrated model in our experiments.
One possible reason is that we use a different splitting from the CaGCN paper, where they follow a fixed splitting from Kipf and Welling [14].
A significant difference is that the splitting from Kipf and Welling [14] has more validation nodes than the training nodes.
This differs from typical real-world applications, where the larger fold would often be used to train a good classifier [26], and only the smaller fold is available for fitting the calibrator.
[Table]
[Figure]
In our splitting, the validation sets of Cora and Citeseer are substantially smaller than those in Kipf and Welling [14].
We observe that CaGCN yields suboptimal calibration results (see Section 6.1) and predictions with higher confidence tend to be over-confident in the reliability diagram in Figure 15.
By contrast, the validation set in Pubmed is relatively large since it has more nodes.
We notice that CaGCN achieves competitive results in Pubmed and the confidence-accuracy curve almost lies on the diagonal.
We observe that CaGCN also produces suboptimal calibration results in CoraFull, even though the validation set is large.
We suspect that this is caused by the class imbalance of the
CoraFull data.
Class imbalance is known to be a challenge for many calibration methods [36].
F GATS weight visualization
GATS learns τ hi from sorted logits.
We discover that the absolute value of the learned weights in the linear layer generally follows the ranking of the logits across the class.
That is to say, logits with higher value have stronger influence to τ hi .
In Figure 16 we visualize the weights θ = ( θ 1 , . . . , θ H ) of the linear layers ϕ h in GATS. Here, it is interesting to see that the weights θ h from different heads
Confidence 0.
A cc u r a c y
Cora 0.
Confidence 0.
A cc u r a c y
Citeseer 0. Confidence 0. A cc u r a c y Pubmed 0.
Confidence 0.
A cc u r a c y
Cora 0.
Confidence 0.
A cc u r a c y
Citeseer 0.
Confidence 0.
A cc u r a c y
Pubmed h have slight variations.
Combining multiple heads in the attention with sorting could be considered as a form of ensemble without the being overly parameterized
[Figure]
G Analysis of correlations between the factors
In this section we visualize the correlation between the local-view factors: the distance to training nodes, the relative confidence level, and the neighborhood similarity.
Each plot shows how the factor on the y-axis varies when the factor on the x-axis is fixed to a given value.
Two factors are independent when we observe a horizontal line in the plot.
As the relative confidence level is a model dependent factor, GCN and GAT will have different correlation plots when it is involved.
G.1 Distance to training nodes – relative confidence level
Figures 17, 18, 19, and 20 show the correlation plots between the distance to training nodes and the relative confidence level.
In Figure 17 and 19 we see that regardless of the distance to the training nodes, the averaged relative confidence level stays around zero.
[Figure]
[Figure]
[Figure]
[Figure]
G.2 Relative confidence level – neighborhood similarity
Figures 21, 22,
[Figure]
and 24 show the correlations between the relative confidence level and the neighborhood similarity.
We observe some partial correlation between these two factors, especially in the negative region of the node homophily.
[Figure]
[Figure]
[Figure]
homophily
[Figure]
R e l .
c
o n f i d e n c e
Cora
[Figure]
homophily
[Figure]
R e l .
c
o n f i d e n c e
Citeseer 5 Node homophily
R e l .
c
o n f i d e n c e
Pubmed 5 Node homophily
R e l .
c
o n f i d e n c e
Computers
[Figure]
homophily
R e l .
c
o n f i d e n c e
Photo
[Figure]
homophily
R e l .
c
o n f i d e n c e
CS
[Figure]
homophily
[Figure]
R e l .
c
o n f i d e n c e
Physics
[Figure]
Figures 25 and 26 show the correlation between the distance to training nodes and neighborhood similarity.
Note that these two factors are not model-dependent and thus GCN and GAT share the same results.
We observe that these two factors have a less significant correlation
[Figure]
[Figure]
H Factor node count analysis
In this section we plot the number of test nodes depending on the three local view factors: distance to training nodes, relative confidence level
[Figure]
and neighborhood similarity.
H.1 Node count for distance to training nodes
Figure 27 summarizes the node count results of the distance to training nodes.
We see the majority of nodes can be connected to the training nodes by one or two hops.
[Figure]
H.2 Node count for relative confidence level
Figure 28 and 29 are the node count results of the relative confidence level for GCN and GAT respectively.
We observe that most of the nodes are concentrated around the zero relative confidence level.
[Figure] [Figure]
H.3 Node count for neighborhood similarity
Figure 30 shows that the majority of nodes lie in the positive homophily region.
[Figure]