  We develop variational Laplace for Bayesian neural networks (BNNs) which
exploits a local approximation of the curvature of the likelihood to estimate
the ELBO without the need for stochastic sampling of the neural-network
weights. Variational Laplace performs better on image classification tasks than
MAP inference and far better than standard variational inference with
stochastic sampling despite using the same mean-field Gaussian approximate
posterior. The Variational Laplace objective is simple to evaluate, as it is
(in essence) the log-likelihood, plus weight-decay, plus a squared-gradient
regularizer. Finally, we emphasise care needed in benchmarking standard VI as
there is a risk of stopping before the variance parameters have converged. We
show that early-stopping can be avoided by increasing the learning rate for the
variance parameters.
