We introduce a new approach for capturing model uncertainty for neural networks (NNs) in regression , which we call Neural Optimization-based Model Uncertainty (NOMU). The main idea of NOMU is to design a network architecture consisting of two connected sub-networks , one for the model prediction and one for the model uncertainty, and to train it using a carefully designed loss function. With this design , NOMU can provide model uncertainty for any given (previously trained) NN by plugging it into the framework as the sub-network used for model prediction. NOMU is designed to yield uncertainty bounds (UBs) that satisfy four important desiderata regarding model uncertainty , which established methods often do not satisfy. Furthermore, our UBs are themselves representable as a single NN, which leads to computational cost advantages in applications such as Bayesian optimization. We evaluate NOMU experimentally in multiple settings. For regression, we show that NOMU performs as well as or better than established benchmarks. For Bayesian optimization, we show that NOMU outperforms all other benchmarks.
We study methods for estimating model uncertainty for neural networks (NNs). To isolate the effect of model uncertainty, we focus on a noiseless setting with scarce training data. We introduce five important desiderata regarding model uncertainty that any method should satisfy. However, we find that established benchmarks often fail to reliably capture some of these desiderata, even those that are required by Bayesian theory. To address this, we introduce a new approach for capturing model uncertainty for NNs , which we call Neural Optimization-based Model Uncertainty (NOMU). The main idea of NOMU is to design a network architecture consisting of two connected sub-NNs , one for model prediction and one for model uncertainty, and to train it using a carefully-designed loss function. Importantly, our design enforces that NOMU satisfies our five desiderata. Due to its modular architecture , NOMU can provide model uncertainty for any given (previously trained) NN if given access to its training data. We first experimentally study noiseless regression with scarce training data to highlight the deficiencies of the established benchmarks. Finally, we study the important task of Bayesian optimization (BO) with costly evaluations, where good model uncertainty estimates are essential. Our results show that NOMU performs as well or better than state-of-the-art benchmarks.
R
introduce
study methods for estimating model uncertainty for neural networks (NNs). To isolate the effect of model uncertainty, we focus on a noiseless setting with scarce training data. We introduce five important desiderata regarding model uncertainty that any method should satisfy. However, we find that established benchmarks often fail to reliably capture some of these desiderata, even those that are required by Bayesian theory. To address this, we introduce
R
neural networks (NNs) in regression
NNs
R
sub-networks
sub-NNs
D
the
D
the
R
carefully designed
carefully-designed
R
With this design
Importantly, our design enforces that NOMU satisfies our five desiderata. Due to its modular architecture
R
by plugging it into the framework as the sub-network used for model prediction. NOMU is designed to yield uncertainty bounds (UBs) that satisfy four important desiderata regarding model uncertainty , which established methods often do not satisfy. Furthermore, our UBs are themselves representable as a single NN, which leads to computational cost advantages in applications such as Bayesian optimization. We evaluate NOMU experimentally in multiple settings. For regression, we
if given access to its training data. We first experimentally study noiseless regression with scarce training data to highlight the deficiencies of the established benchmarks. Finally, we study the important task of Bayesian optimization (BO) with costly evaluations, where good model uncertainty estimates are essential. Our results
D
as
R
established benchmarks. For Bayesian optimization, we show that NOMU outperforms all other
state-of-the-art