  This paper studies the task of inpainting man-made scenes. It is very
challenging due to the difficulty in preserving the visual patterns of images,
such as edges, lines, and junctions. Especially, most previous works are failed
to restore the object/building structures for images of man-made scenes. To
this end, this paper proposes learning a Sketch Tensor (ST) space for
inpainting man-made scenes. Such a space is learned to restore the edges,
lines, and junctions in images, and thus makes reliable predictions of the
holistic image structures. To facilitate the structure refinement, we propose a
Multi-scale Sketch Tensor inpainting (MST) network, with a novel
encoder-decoder structure. The encoder extracts lines and edges from the input
images to project them into an ST space. From this space, the decoder is
learned to restore the input images. Extensive experiments validate the
efficacy of our model. Furthermore, our model can also achieve competitive
performance in inpainting general nature images over the competitors.
