Measuring Calibration in Deep Learning
Authors: Jeremy Nixon, Mike Dusenberry, Ghassen Jerfel, Linchuan Zhang, Dustin Tran
Keywords: Deep Learning, Multiclass Classification, Classification, Uncertainty Estimation, Calibration
Abstract: Overconfidence and underconfidence in machine learning classifiers is measured by calibration: the degree to which the probabilities predicted for each class match the accuracy of the classifier on that prediction. We propose two new measures for calibration, the Static Calibration Error (SCE) and Adaptive Calibration Error (ACE). These measures take into account every prediction made by a model, in contrast to the popular Expected Calibration Error.