  We propose a new model that estimates uncertainty in a single forward pass
and works on both classification and regression problems. Our approach combines
a bi-Lipschitz feature extractor with an inducing point approximate Gaussian
process, offering robust and principled uncertainty estimation. This can be
seen as a refinement of Deep Kernel Learning (DKL), with our changes allowing
DKL to match softmax neural networks accuracy. Our method overcomes the
limitations of previous work addressing deterministic uncertainty
quantification, such as the dependence of uncertainty on ad hoc
hyper-parameters. Our method matches SotA accuracy, 96.2% on CIFAR-10, while
maintaining the speed of softmax models, and provides uncertainty estimates
that outperform previous single forward pass uncertainty models. Finally, we
demonstrate our method on a recently introduced benchmark for uncertainty in
regression: treatment deferral in causal models for personalized medicine.
