  We propose a generative model that can infer a distribution for the
underlying spatial signal conditioned on sparse samples e.g. plausible images
given a few observed pixels. In contrast to sequential autoregressive
generative models, our model allows conditioning on arbitrary samples and can
answer distributional queries for any location. We empirically validate our
approach across three image datasets and show that we learn to generate diverse
and meaningful samples, with the distribution variance reducing given more
observed pixels. We also show that our approach is applicable beyond images and
can allow generating other types of spatial outputs e.g. polynomials, 3D
shapes, and videos.
