  How to improve generative modeling by better exploiting spatial regularities
and coherence in images? We introduce a novel neural network for building image
generators (decoders) and apply it to variational autoencoders (VAEs). In our
spatial dependency networks (SDNs), feature maps at each level of a deep neural
net are computed in a spatially coherent way, using a sequential gating-based
mechanism that distributes contextual information across 2-D space. We show
that augmenting the decoder of a hierarchical VAE by spatial dependency layers
considerably improves density estimation over baseline convolutional
architectures and the state-of-the-art among the models within the same class.
Furthermore, we demonstrate that SDN can be applied to large images by
synthesizing samples of high quality and coherence. In a vanilla VAE setting,
we find that a powerful SDN decoder also improves learning disentangled
representations, indicating that neural architectures play an important role in
this task. Our results suggest favoring spatial dependency over convolutional
layers in various VAE settings. The accompanying source code is given at
https://github.com/djordjemila/sdn.
