  Given an incomplete image without additional constraint, image inpainting
natively allows for multiple solutions as long as they appear plausible.
Recently, multiplesolution inpainting methods have been proposed and shown the
potential of generating diverse results. However, these methods have difficulty
in ensuring the quality of each solution, e.g. they produce distorted structure
and/or blurry texture. We propose a two-stage model for diverse inpainting,
where the first stage generates multiple coarse results each of which has a
different structure, and the second stage refines each coarse result separately
by augmenting texture. The proposed model is inspired by the hierarchical
vector quantized variational auto-encoder (VQ-VAE), whose hierarchical
architecture isentangles structural and textural information. In addition, the
vector quantization in VQVAE enables autoregressive modeling of the discrete
distribution over the structural information. Sampling from the distribution
can easily generate diverse and high-quality structures, making up the first
stage of our model. In the second stage, we propose a structural attention
module inside the texture generation network, where the module utilizes the
structural information to capture distant correlations. We further reuse the
VQ-VAE to calculate two feature losses, which help improve structure coherence
and texture realism, respectively. Experimental results on CelebA-HQ, Places2,
and ImageNet datasets show that our method not only enhances the diversity of
the inpainting solutions but also improves the visual quality of the generated
multiple images. Code and models are available at:
https://github.com/USTC-JialunPeng/Diverse-Structure-Inpainting.
