T RAINING , A RCHITECTURE , AND P RIOR FOR D ETERMINISTIC U NCERTAINTY M ETHODS
Anonymous authors
Paper under double-blind review
A BSTRACT
Accurate and efficient uncertainty estimation is crucial to build reliable Machine
Learning (ML) models capable to provide calibrated uncertainty estimates, generalize and detect Out-Of-Distribution (OOD) datasets.
To this end, Deterministic
Uncertainty Methods (DUMs) is a promising model family capable to perform uncertainty estimation in a single forward pass.
This work investigates important design choices in DUMs: (1) we show that training schemes decoupling the core architecture and the uncertainty head schemes can significantly improve uncertainty performances.
(2) we demonstrate that the core architecture expressiveness is crucial for uncertainty performance and that additional architecture constraints to avoid feature collapse can deteriorate the trade-off between OOD generalization and detection.
(3) Contrary to other Bayesian models, we show that the prior defined by DUMs do not have a strong effect on the final performances.
I NTRODUCTION
Safety is critical to the adoption of deep learning in domains such as autonomous driving, medical diagnosis, or financial trading systems.
A solution for this problem is to create reliable models capable to estimate the uncertainty of its own predictions.
Different uncertainty types are divided in aleatoric uncertainty quantified by the inherited noise in the data, thus irreducible; epistemic uncertainty quantified by the modeling choice or lack of data, thus reducible; predictive uncertainty, a combination of aleatoric and epistemic (Gal, 2016).
In practice, high quality uncertainty estimates must be calibrated and able to detect Out-Of-Distribution (OOD) data like anomalies while preserving good Out-Of-Distribution (OOD) generalization performances like on dataset shifts.
Recently, a family of methods for uncertainty estimation named Deterministic Uncertainty Methods (DUMs) have emerged (Postels et al., 2022).
Contrary to uncertainty methods such as Ensembles (Lakshminarayanan et al., 2017), MC Dropout (Gal & Ghahramani, 2016) or other Bayesian neural networks on weights (Blundell et al., 2015), which require multiple forward passes to make predictions, DUMs only require a single forward pass, thus making them significantly more computationally efficient.
Generally, DUMs are composed of three components with high potential impact on their performances: the training procedure which is supposed to optimize the model toward high predictive and uncertainty performances, the core architecture which is supposed to define informative embeddings used to make predictions, and the prior which is supposed to define the default uncertain predictions.
In this work, we investigate the role of these three components on the quality of DUMs uncertainty estimates by evaluating calibration performances, OOD detection, and OOD generalization.
Our main contributions are:
• Training : We show that decoupling the learning rates of the core architecture and uncertainty heads of DUMs, jointly training the core architecture and the uncertainty head of DUMs, and pretraining with more data and higher data quality improve uncertainty performances.
• Architecture : We demonstrate that the expressiveness of the core architecture defined by the architecture type , architecture size , and dimension of the latent space is crucial for both predictive and uncertainty performances.
Further, we show that applying additional regularization constraints to avoid feature collapse does not find better trade-off between OOD detection and generalization, even sometimes degrading performances.
• Prior :
In contrast to Bayesian neural networks on weights where the choice of prior is critical (Wenzel et al., 2020; Fortuin et al., 2022; Noci et al., 2021; Kapoor et al., 2022), we empirically show that the choice of prior defined in the training loss or in the uncertainty head of DUMs has a relatively small effect on the final uncertainty performances.
D ETERMINISTIC U NCERTAINTY M ETHODS
We consider a classification task where the goal is to predict the label y ( i ) ∈ { 1 , . . .
, C } based on the input feature x ( i ) ∈ R D .
In this case, a DUM generally performs predictions in two main steps: (1) A core architecture f ϕ maps the input features x ( i ) ∈
R D to a latent representation z
( i ) ∈ R H , i.e. f ϕ ( x ( i ) )
=
z ( i ) .
In practice, important design choices are the latent dimension H and the architecture f ϕ which should be adapted to the task (see section 4).
Further, multiple works proposed to apply additional bi-Lipschitz or reconstruction constraints to enrich the informativeness of the latent representation
z ( i ) (see section 4).
An uncertainty head g ψ maps the latent representation
z ( i ) to a predicted label
ˆ y
( i ) and an associated (aleatoric, epistemic, or predictive) uncertainty estimate u ( i ) , i.e. g ψ ( z ( i ) )
=
(ˆ y ( i ) , u ( i ) ) .
In practice, important design choices are the type of uncertainty head which are generally instantiated with a Gaussian Process (GP) (Liu et al., 2020; van Amersfoort et al., 2021; 2020; Bilo ˇ s et al., 2019; Charpentier et al., 2022b), a density estimator (Charpentier et al., 2020; 2022a;b; Stadler et al., 2021;
Bilo ˇ s et al., 2019; Mukhoti et al., 2021;
Postels et al., 2020a; Winkens et al., 2020; Wu & Goodman, 2020), or an evidential model (Charpentier et al., 2020; 2022a;b; Stadler et al., 2021;
Bilo ˇ s et al., 2019;
Malinin & Gales, 2018), and the choice of the prior used by the uncertainty head (see section 5).
Beyond core architecture and uncertainty head, another important choice is the training procedure which can either couple or decouple the parameters of the core architecture and uncertainty head (see section 3).
In this work we focus on two recent DUMs which cover different types of uncertainty heads:
Natural Posterior Network (NatPN) Charpentier et al.
(2022a) which learns an evidential distribution based on density estimation on the latent space, and Deterministic Uncertainty Estimator (DUE) (van
Amersfoort et al., 2021) which learns a deep Gaussian Process by parametrizing learnable inducing points in the latent space (see appendix A.1 for further method details).
While NatPN is capable to differentiate aleatoric, epistemic, and predictive uncertainty, DUE only outputs the predictive uncertainty.
For all the experiments, we use the same default setup: we first pretrain the encoder with the cross-entropy loss until convergence, then load the pretrained encoder and jointly train the encoder and uncertainty head (see appendices B to D for further compenent details).
We report the predictive performance via accuracy, and the uncertainty performances with Brier Score and OOD detection results after averaging over 5 seeds (see appendix A.3 for further metric details).
We perform our experiments on MNIST (LeCun et al., 1998), CIFAR10 and CIFAR100 (Krizhevsky, 2009), and
Camelyon (Koh et al., 2021).
OOD results reported in tables 1 to 4 averages the uncertainty estimation from five OOD datasets: SVHN, STL10, CelebA, Camelyon and SVHN OODom (Netzer et al., 2011; Coates et al., 2011; Liu et al., 2015).
Our code and additional material is available online 1 .
Related work.
Previous works survey OOD detection methods (Yang et al., 2021), OOD generazilation methods (Shen et al., 2021), or a wide range of uncertainty estimation methods (Gawlikowski et al., 2021; Psaros et al., 2023; Ulmer, 2021; Abdar et al., 2021) by presenting key methods and challenges.
These surveys do not focus on deterministic methods and do not make empirical analysis.
Other works propose great empirical studies to compare uncertainty estimation methods under shifts (Ovadia et al., 2019), or analyze the role of the prior in Bayesian neural networks on weights (Wenzel et al., 2020; Fortuin et al., 2022;
Noci et al., 2021; Kapoor et al., 2022).
These works do not focus on DUMs.
Closer to our work, Postels et al.
(2022) compares methods in the DUMs family and demonstrate calibration limitations.
In contrast, we evaluate the role of components in DUMs and show that carefully specifying training , architecture , or prior can improve uncertainty metrics like calibration and OOD detection but also ID and OOD predictive performances.
T RAINING FOR DUM S
In this section, we study the importance of the training procedure in the performance of DUMs.
To this end, we look at decoupling the learning rates of the core encoder architecture and the uncertainty head, different training schemes , and different pretraining schemes .
Decoupling learning rates.
We decouple the learning rates of the core architecture and the uncertainty head.
We show the validation results for CIFAR100 as ID and SVHN as OOD with the core architecture ResNet18 in fig.
Observation: We observe that, when using different learning rates for the core architecture and the uncertainty head, NatPN improves Brier Score and OOD epistemic results and DUE significantly improves both predictive and uncertainty results.
Hence, this shows that decoupling learning rates can improve results of DUMs, thus suggesting that the core architecture and the uncertainty head have training dynamics which requires different considerations.
[Table]
1e-05 5e-05 1e-04 5e-04 1e-03 5e-03 Core architecture LR 81.2 81.0 80.1 75.9 80.2 57.
Training schemes.
We compare two settings: the joint training in which we jointly train the weights of the core architecture and uncertainty head, and the sequential training in which we only train the uncertainty head by keeping the weights of the pretrained core architecture fixed.
For each of the setting, we apply two additional techniques to stabilize the training: adding a batch normalization to the last layer of the encoder to enforce latent representations to locate in a normalized region (Ioffe & Szegedy, 2015; Charpentier et al., 2022a), and resetting the last layer to retrain its weights to improve robustness to spurious correlation (Kirichenko et al., 2022).
We show the results for CIFAR100 as ID and five difference OOD datasets with the ResNet18 as core architecture in table 1 and additional results in the appendix table 7.
Observation: We observe that, compared to its sequential counterpart, joint training consistently improves DUMs performance for most metrics, thus suggesting that joint training should be preferred in practice for DUMs.
Furthermore, while the GP-based method DUE does not benefit from stabilization techniques, we observe that they can significantly increase performance of the density-based method NatPN.
This behavior is intuitively explained by the practical difficulty to accurately fit densities in high dimensional latent space.
This can be significantly improved by using more powerful density estimator (see table 5 in appendix).
Pretraining schemes.
We compare multiple training schemes which differ in terms of amount and quality of data used for pretraining.
Hence, we do not pretrain the core architecture or pretrain it with 10% of CIFAR100, 100% of CIFAR100 with and without Gaussian noise, or ImageNet.
We show the results for CIFAR100 as ID and five different OOD datasets with ResNet50 as core architecture in table 2 and additional results in the appendix table 8.
Observation: We observe that, while too few data for pretraining does not improve final performance of DUMs, the overall performance significantly increase when the encoder is pretrained with high quantity and high quality of data.
Similarly to Kirichenko et al.
, this suggests that the embedding quality is important to improve uncertainty quantification.
Here, we show additionally that embeddings pretrained with many high quality data are crucial to facilitate the prediction of the uncertainty head.
[Table]
[Figure]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure] [Table] [Figure] [Table]
[Figure] [Table] [Figure] [Table]
[Figure]
[Table]
[Figure] [Table] [Figure] [Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure] [Table] [Figure] [Table]
[Figure]
[Table]
[Figure]  A RCHITECTURE FOR DUM S In this section, we study the impact of the architecture component in DUMs. To this end, we look at different latent dimensions , different architectural types and size , and applying different regularization constraints to avoid feature collapse (van Amersfoort et al., 2021).
Latent dimension.
We vary the dimension of the output space of the core architecture.
We show the results for each pair of ID dataset and its distribution shifted OOD dataset (MNIST/CMNIST, CIFAR/CIFAR-C, CamelyonID/CamelyonOOD) with the core architecture
ResNet18 for MNIST/CIFAR, and WideResNet-28-10 for Camelyon in fig.
2 and additional uncertainty estimation results in the appendix fig.
Observation: We observe that increasing the latent dimensions leads to improvement for DUMs on ID and OOD datasets with particularly significant improvement for NatPN (see fig. 2).
This suggests that higher latent dimensions are more expressive by encoding more information.
However, we observe that a too high latent dimension can degrade OOD detection performance by causing numerical instabilities in the training (see fig. 10), suggesting a trade-off between OOD generalization and OOD detection.
[Figure]
Architecture type and size.
We compare the influence of the type and size of the core architecture on the performance of DUMs.
We consider
residual, convolutional, and transformer architectures like ResNet18, ResNet50, EfficientNetV2, and Swin (He et al., 2016; Tan & Le, 2021; Liu et al., 2021).
We show the results for DUMs trained on CIFAR100 as ID with the different core architectures in table 3 and additional results at appendix table 9.
Observation: We observe that models with more parameters achieve better results.
In particular, ResNet50 achieves significantly better results than ResNet18.
Further, more recent core architectures like EfficientNetV2 and Swin are better calibrated and more expressive leading to a better overall performance.
This can be explained by the fact that they are more expressive and provide more informative embeddings for the uncertainty
[Figure]
Bi-lipschitz can improve OOD detection by mitigating feature collapse (see fig.
8a)
at the expense of degrading OOD generalization
[Figure]
head to operate on.
This aligns with Minderer et al.
(2021) which states that the architecture type is important for the calibration properties
[Table]
Regularization constraints.
Feature collapse is a phenomenon where a model may discard important parts of the input information during its training phase, which may degrade OOD detection performance (van Amersfoort et al., 2021).
Two techniques to avoid feature collapses are bi-Lipschitz constraints via combining residual connections and lipschitz constraints (Liu et al., 2020), and reconstruction constraints via adding an additional reconstruction term in the loss (Postels et al., 2020b).
We show the results for DUMs trained on the datasets MNIST and CIFAR100 with ResNet18 in figs.
3 and 4 and additional results for other datasets (Toy dataset, CIFAR10, Camelyon) at appendix C. Observation: We observe that the reconstruction technique is not capable to avoid feature collapse.
Indeed, we show that, even with reconstruction constraints, some (non-discriminative) features can completely collapse (see figs.
8b and 9b for toy examples).
Hence, while this method can lead to small OOD improvements on simple tasks (see e.g. MNIST in fig. 4), this benefit does not generalize to more complex tasks (see e.g. CIFAR100 in fig. 4).
In contrast, we observe that bilipschitz constraints indeed mitigate the collapse of features (see figs.
8a and 9a for toy examples), leading to similar or higher OOD detection performance (see fig. 3).
The mitigation of feature collapse can be mostly assigned to the residual connection constraints.
However, bilipschitz constraints can improve OOD detection results on simple tasks (e.g. MNIST, CIFAR10), it degrades
OOD generalization performance (see fig. 3) and does not significantly improve OOD detection on more complex tasks (see e.g. Camelyon, CIFAR100 in fig. 6).
Intuitively, maintaining features which are not discriminative to the task might introduce spurious correlations, thus degrading performances.
E.g. enforces the architecture to encode the color feature in the latent space decreases the performance of the OOD CMNIST datasets after training on the ID MNIST dataset.
P RIOR FOR DUM S
In this section, we study the effect that the prior component has in DUMs.
More specifically, we investigate the relationships between aleatoric uncertainty and the prior specified for DUMs.
In particular, this is motivated by Kapoor et al.
(2022) which shows that using priors that forces model to be confident on the training data points can improve its performance by explicitly accounting for aleatoric uncertainty.
To this end, we look at entropy regularization defining a training prior in the loss, prior evidence and kernel function defining a functional prior in the uncertainty head.
Prior.
We compare different prior specifications including entropy regularization defining a training prior in the loss, prior evidence and kernel function defining a functional prior in the uncertainty head.
Entropy regularization is the entropy term H ( Q ) in the Bayesian loss used to train NatPN which encourages a (uniform) prior distributions with high entropy (Charpentier et al., 2022a).
We control the strength of the regularization factor λ .
Further, NatPN also explicitly defines a prior via the parameters χ prior and n prior .
While χ prior defines the default categorical prediction via a uniform categorical distribution, the evidence parameter n prior defines the prior number of pseudoobservations and can be varied.
Finally, we vary the prior of DUE by using different kernel functions in the learned GP including Matern kernel, RQ kernel, and RBF (Rasmussen & Williams, 2006).
Observation: Contrary to other Bayesian neural networks (Kapoor et al., 2022), we observe that predictive and uncertainty performances of DUMs are not very sensitive to the prior specification (see figs.
5 and 14 and table 4), thus suggesting a higher robustness to prior mispecification.
Nonetheless, a too strong entropy regularization toward an uniform prior degrades more performance of
DUMs trained on dataset with low label noise than on high label noise.
This suggests that a too high discrepancy between the model prior and the dataset aleatoric uncertainties can impact performance.
[Figure] [Figure]
[Table]
C ONCLUSION
We investigate important design choice in DUMs.
We show that training of DUMs can be improved by decoupling the the optimization of the core architecture and the uncertainty head.
We show that expressive core architecture can improve DUMs performances.
In contrast, additional constraints to avoid feature collapse do not consistently lead to better performance, potentially degrading the
OOD generalization and detection trade-off.
Finally, we show that the choice of prior for DUMs does not lead to important performance improvements.
A A PPENDIX A. D ETERMINISTIC U NCERTAINTY M ETHODS
NatPN .
The deep Bayesian uncertainty model NatPN (Charpentier et al., 2022a) can be decomposed into these steps: (1) a core architecture predicts one latent representation of the input x ( i ) i.e. z ( i ) = f ϕ ( x ( i ) )
∈ R H , (2)
While a density estimator P ( . | w ) predicts the evidence parameter update n ( i ) = N H P ( z ( i ) | w ) where the N H is a scaling factor named certainty budget, a single linear decoder g ψ outputs the parameter update χ ( i ) =
g ψ ( z ( i ) )
∈ R L , which can be viewed as a softmax output prediction.
(3) We perform an input-dependent Bayesian update which can be expressed in a closed-form as:
[Equation]
where
[Equation]
where χ prior , n prior are fixed prior parameters, and χ post , ( i ) , n post , ( i ) are the input-dependent posterior parameters. For the classification, the variable θ ( i ) represents the normalized categorical vector p ( i ) .
The predictive uncertainty is computed via the entropy of the predictive categorical distribution, and the epistemic uncertainty is computed via the evidence parameter n post, ( i ) .
We train all the components of neural network parameters { ϕ , w , ψ } jointly with the Bayesian loss (Charpentier et al., 2022a):
[Equation]
i ) ] (1) where λ is the regularization factor of the entropy term representing.
We refer to (Charpentier et al., 2022a) for a more detailed description of the method.
DUE.
The deep kernel learning method DUE (van Amersfoort et al., 2021) can be decomposed into these steps: (1) a core architecture predicts one latent representation of the input x ( i ) i.e. z ( i ) =
f θ ( x ( i ) ) ∈
R H , (2) a Gaussian Process defined from a fixed set of K learnable inducing points { ϕ k } Kk =1 and a predefined positive definite kernel κ ( · , · ) predicts the mean µ ( x ( i ) ) and the variance σ ( x ( i ) ) of a Gaussian distribution, and (3) we apply softmax to the mean output µ ( x ( i ) ) for the classification prediction, i.e. p ( i ) =
softmax ( µ ( x ( i ) ) .
We train the neural network parameters θ and the inducing points { ϕ k } Kk =1 jointly with a variational ELBO loss.
For classification, the predictive uncertainty is computed as the entropy of the predictive categorical distribution.
We refer to (van Amersfoort et al., 2021) for a more detailed description of the method.
A. D ATASET DETAILS
We split all the training datasets into train, validation and test sets.
For all the datasets, the test set is fixed while the training/validation sets are split in 80/20% respectively.
The random split of training/validation sets change depending on the seeds to ensure more diversity.
MNIST (LeCun et al., 1998).
Image classification dataset.
Similarly as in Arjovsky et al.
(2019), we create the CMNIST dataset for domain generalization experiments by expanding the input’s size to 3 x 28 x 28 and zeroing one of the three channels.
For OOD detection we use the test set of
MNIST as ID dataset and compare to: KMNIST (Clanuwat et al., 2018), CIFAR10, CMNIST, and
KMNIST OODom, where we scale the input by 255.
The batch size used is 512.
CIFAR (Krizhevsky, 2009).
Image classification dataset.
We apply two data augmentations methods to the training data:the random horizontal flip and random cropping with padding equal to 4.
For domain generalization we use the corrupted version CIFAR-C (Hendrycks & Dietterich, 2019) and report the average metric of 15 corruptions for the level of corruption severity of 1.
For OOD detection we use the test set of CIFAR10 as ID dataset and compare to: SVHN (Netzer et al., 2011),
STL10 (Coates et al., 2011), CelebA (Liu et al., 2015), Camelyon (Test OOD), and SVHN OODom.
Since the Camelyon (Test OOD) dataset is large (85k), we extract only 10k subset of images as the OOD datasets.
The batch size used is 128.
Camelyon (Koh et al., 2021).
Image classification dataset.
We apply two data augmentations emthods to the training data: random horizontal flip and random rotation of 15 degrees.
For domain generalization the dataset already provide the distribution shifted validation and test splits.
For OOD detection we use the ID validation set of Camelyon as ID dataset (the ID test set is not available) and compare to: SVHN, STL10, CelebA, Camelyon (Test OOD), and SVHN OODom.
The batch size used is 32.
Each OOD dataset is rescaled to the same size as the ID dataset and normalized with zero mean and unit variance based on the statistics of ID dataset (for the Camelyon dataset we don’t apply any normalization as in Koh et al.
A. M ETRIC DETAILS
Accuracy.
The standard accuracy 1 N (cid:80) i
[ y ∗ , ( i ) =
y ( i ) ] is used, where y ∗ , ( i ) is the true label and y ( i ) is the predicted label.
Calibration.
The Brier score 1 C (cid:80)
Ni
|| p
( i ) − y ∗ , ( i ) || is used, where p ( i ) is the predicted softmax probability and y ∗ , ( i ) is the one-hot encoded true label.
Lower calibration indicates a better calibrated model.
Note that in constrast with the Expected Calibration Error (ECE), the Brier score is a strictly proper scoring rule which makes it a particularly good evaluation metric for calibration
Gneiting & Raftery (2007).
OOD Generalization.
We apply accuracy and calibration to the distribution shifted OOD dataset and compare the results with the ID dataset to estimate the model’s ability for generalization.
OOD Detection.
We treat this task as a binary classification, where we assign class 1 to ID data and class 0 to OOD data using the aleatoric, epistemic, and predictive uncertainty estimates as scores for OOD data.
This allows to compute the final scores using the area under the receiver operating characteristic curve (AUC-ROC) to measure the model’s ability to detect OOD data.
A. M ODEL DETAILS
Core architecture.
We use the same feature extractor for both the DUMs architecture.
The list of core architectures used across the experiments are: ResNet18 / ResNet50 / EfficientNet /
Swin (He et al., 2016; Tan & Le, 2021; Liu et al., 2021) from the torchvision repository 2 and WideResNet-28-10 (Zagoruyko & Komodakis, 2016) from the original implementation of DUE.
Except for the experiment on architecture type and size where ResNet18 has output channels for the residual blocks with size [64 , 128 , 256 , 512] , ResNet18 has output channels for the residual blocks with size [32 , 64 , 128 , 256] which causes small differences in final accuracy.
Uncertainty head.
For DUE we use the original implementation 3 with by default we use the RBF kernel function.
For NatPN we use the original implementation 4 but change the uncertainty head with a more expressive density estimator.
As seen in Table 5, we found that a more expressive normalizing flow with resampled base (Durkan et al., 2019; Stimper et al., 2022) improves significantly the results over a simpler radial normalizing flow (Rezende & Mohamed, 2015) across all the metrics.
For all the experiments (except toys where we use radial flow) we use NSF-R withlayers.
[Table]
[Table]
B T RAINING FOR DUM S D ETAILS
By default, we first start by only pretraining the core encoder architecture using the cross-entropy loss, before attaching the DUM uncertainty head to the pretrained encoder and continue with the joint training phase. We do not pretrain the encoder for MNIST.
we provide further details in table 6.
Following the original method in Charpentier et al.
(2022a), we train the NatPN uncertainty head before ( warmup ) and after ( finetune ) the joint training.
In the warmup phase, we use the lambda scheduler increasing linearly from zero to LR head value in table 6. In the finetune phase, we use a multistep scheduler that scales the learning rate by 0.2 at 70% and 90% of the training starting from the LR head value in table 6. We warmup for 0/5/0 and finetune for 60/200/5 epochs for the datasets
MNIST/CIFAR/Camelyon respectively. [Table] [Figure] [Table]
[Figure] [Table] [Figure] [Table]
[Figure] [Table] [Figure] [Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
Decoupling learning rate.
In this experiment we use different values for the learning rates of the core architecture and of the uncertainty head.
After the decoupling learning rate experiment, we choose the best combination of learning rates through model selection via the validation results and apply it to other experiments.
E.g., for the joint training schema and pretraining schema experiments, NatPN uses a learning rate of 1e-4/1e-4 for encoder/head respectively, while for DUE it is 1e-5/5e-3.
Training schemes.
In this experiment, we compare the joint training in which we jointly train the weights of the core architecture and uncertainty head, and the sequential training in which we only train the uncertainty head by keeping the weights of the pretrained core architecture fixed.
For each of the setting, we apply two additional techniques to stabilize the training: adding a batch normalization to the last layer of the encoder to enforce latent representations to locate in a normalized region (Ioffe & Szegedy, 2015; Charpentier et al., 2022a), and resetting the last layer to retrain its weights to improve robustness to spurious correlation (Kirichenko et al., 2022).
Pretraining schemes.
In this experiment, we do not pretrain the core encoder architecture or pretrain it with 10% of CIFAR100, 100% of CIFAR100, and ImageNet.
For the schemes None and
C100 (10%) which use no or few pretraining data, we increase the joint training phase to 200 epochs with for the core architecture to ensure proper convergence.
C A RCHITECTURE FOR DUM S D ETAILS
For all the architecture experiments we used the default training settings in appendix B and table 6.
Bi-Lipschitz training details.
In the None configuration, we removed the residual connection from the architecture for both the pretraining of the encoder and the joint training phase.
In the Residual configuration, we did not modify anything since both ResNet and WideResNet already use the residual connection.
While for the bi-Lipschitz configuration, we added the spectral normalization during both the pretraining and also joint training phase.
Following the original method presented in van Amersfoort et al.
(2021), we used the same implementation and applied spectral normalization to the linear, convolution, and batch normalization layers.
During the model selection, using the validation set results, we find that the best Lipschitz constant c for DUE is 4, and for NatPN is 5.
For both the models the power iteration parameter is set to 1.
For the toy dataset we use an encoder with 4 linear layers of 128 dimension each. [Table]
Reconstruction training details.
The decoder reconstructs the input extracted from the last residual block of the encoder, before the pooling layer.
During the pretraining phase, both the encoder and decoder are trained with the cross-entropy loss plus a MSE reconstruction term.
During the joint training phase, we load the pretrained encoder and decoder, and joint train with the DUMs’ respective loss plus the MSE reconstruction term.
For the toy dataset we use an encoder withlinear layers of 128 dimension each
[Table]
[Table]
D P RIOR FOR DUM S D ETAILS
For all the prior experiments we use the default training settings in appendix B and table 6. In the following experiments, we vary the entropy regularization λ and the evidence prior n prior for
NatPN, and the choice of kernel for DUE.
Before starting the training, we inject the artificial aleatoric noise by reassigning the target y with a randomly chosen class. Two datasets with different degree of noise are used, where 10% and 20% of all the labels in the training dataset are reassigned.
[Figure] [Figure]
[Figure] [Figure]
[Figure]
E p
[Figure]
OO D
[Figure]
p
[Figure] [Figure]
[Figure] [Figure] [Table]
[Figure]