Quantile Regularization : Towards Implicit Calibration of Regression Models
Authors: Saiteja Utpala, Piyush Rai
Keywords: Calibration, Reliable Uncertainty Quantification, Probabilistic Deep Learning
Abstract: Recent works have shown that most deep learning models are often poorly  calibrated, i.e., they may produce overconfident
predictions that are wrong, implying that their uncertainty estimates are unreliable. While a  number of approaches have been proposed recently to calibrate classification models, relatively little work exists on calibrating regression models. Isotonic Regression has recently been advocated for regression calibration. We provide a detailed formal analysis of the \emph{side-effects} of Isotonic Regression when used for regression  calibration. To address this, we  recast quantile calibration as entropy estimation, and leverage this idea to construct a novel quantile regularizer, which can be used in any optimization based probabilisitc regression models. Unlike most of the existing approaches for calibrating regression models, which are based on \emph{post-hoc} processing of the model's output, and require an additional dataset, our method is trainable in an end-to-end fashion, without requiring an additional dataset. We provide empirical results demonstrating that our approach improves  calibration for regression models trained on diverse architectures that  provide uncertainty estimates, such as Dropout VI, Deep Ensembles