  Isotropic Gaussian priors are the de facto standard for modern Bayesian
neural network inference. However, such simplistic priors are unlikely to
either accurately reflect our true beliefs about the weight distributions, or
to give optimal performance. We study summary statistics of neural network
weights in different networks trained using SGD. We find that fully connected
networks (FCNNs) display heavy-tailed weight distributions, while convolutional
neural network (CNN) weights display strong spatial correlations. Building
these observations into the respective priors leads to improved performance on
a variety of image classification datasets. Moreover, we find that these priors
also mitigate the cold posterior effect in FCNNs, while in CNNs we see strong
improvements at all temperatures, and hence no reduction in the cold posterior
effect.
