  Approximate inference in deep Bayesian networks exhibits a dilemma of how to
yield high fidelity posterior approximations while maintaining computational
efficiency and scalability. We tackle this challenge by introducing a new
variational structured approximation inspired by the interpretation of Dropout
training as approximate inference in Bayesian probabilistic models. Concretely,
we focus on restrictions of the factorized structure of Dropout posterior which
is inflexible to capture rich correlations among weight parameters of the true
posterior, and we then propose a novel method called Variational Structured
Dropout (VSD) to overcome this limitation. VSD employs an orthogonal
transformation to learn a structured representation on the variational Dropout
noise and consequently induces statistical dependencies in the approximate
posterior. We further gain expressive Bayesian modeling for VSD via proposing a
hierarchical Dropout procedure that corresponds to the joint inference in a
Bayesian network. Moreover, we can scale up VSD to modern deep convolutional
networks in a direct way with a low computational cost. Finally, we conduct
extensive experiments on standard benchmarks to demonstrate the effectiveness
of VSD over state-of-the-art methods on both predictive accuracy and
uncertainty estimation.
