  Obtaining reliable and accurate quantification of uncertainty estimates from
deep neural networks is important in safety-critical applications. A
well-calibrated model should be accurate when it is certain about its
prediction and indicate high uncertainty when it is likely to be inaccurate.
Uncertainty calibration is a challenging problem as there is no ground truth
available for uncertainty estimates. We propose an optimization method that
leverages the relationship between accuracy and uncertainty as an anchor for
uncertainty calibration. We introduce a differentiable accuracy versus
uncertainty calibration (AvUC) loss function that allows a model to learn to
provide well-calibrated uncertainties, in addition to improved accuracy. We
also demonstrate the same methodology can be extended to post-hoc uncertainty
calibration on pretrained models. We illustrate our approach with mean-field
stochastic variational inference and compare with state-of-the-art methods.
Extensive experiments demonstrate our approach yields better model calibration
than existing methods on large-scale image classification tasks under
distributional shift.
