High-quality image inpainting requires filling missing regions in a damaged image with plausible content. Existing works either fill the regions by copying image patches or generating semantically-coherent patches from region context, while neglect the fact that both visual and semantic plausibility are highly-demanded. In this paper, we propose a Pyramid-context ENcoder Network (PEN-Net) for image inpainting by deep generative models. The PEN-Net is built upon a U-Net structure, which can restore an image by encoding contextual semantics from full resolution input, and decoding the learned semantic features back into images. Specifically, we propose a pyramid-context encoder, which progressively learns region affinity by attention from * This work was performed when the first author was visiting Microsoft Research as a research intern. a high-level semantic feature map and transfers the learned attention to the previous low-level feature map. As the missing content can be filled by attention transfer from deep to shallow in a pyramid fashion, both visual and semantic coherence for image inpainting can be ensured. We further propose a multi-scale decoder with deeply-supervised pyramid losses and an adversarial loss. Such a design not only results in fast convergence in training, but more realistic results in testing. Extensive experiments on various datasets show the superior performance of the proposed network.
And to leverage features of both image-level and feature-level, REF propose a pyramid-context encoder network and an attention transfer mechanism which are able to progressively fill the missing regions from highlevel to low-level feature map and ensure the semantic consistencies at the same time.
Learning Pyramid-Context Encoder Network for High-Quality Image Inpainting
2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)
2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)
Computer Science