  We explore the limitations of and best practices for using black-box
variational inference to estimate posterior summaries of the model parameters.
By taking an importance sampling perspective, we are able to explain and
empirically demonstrate: 1) why the intuitions about the behavior of
approximate families and divergences for low-dimensional posteriors fail for
higher-dimensional posteriors, 2) how we can diagnose the pre-asymptotic
reliability of variational inference in practice by examining the behavior of
the density ratios (i.e., importance weights), 3) why the choice of variational
objective is not as relevant for higher-dimensional posteriors, and 4) why,
although flexible variational families can provide some benefits in higher
dimensions, they also introduce additional optimization challenges. Based on
these findings, for high-dimensional posteriors we recommend using the
exclusive KL divergence that is most stable and easiest to optimize, and then
focusing on improving the variational family or using model parameter
transformations to make the posterior more similar to the approximating family.
Our results also show that in low to moderate dimensions, heavy-tailed
variational families and mass-covering divergences can increase the chances
that the approximation can be improved by importance sampling.
