  Perceptual image restoration seeks for high-fidelity images that most likely
degrade to given images. For better visual quality, previous work proposed to
search for solutions within the natural image manifold, by exploiting the
latent space of a generative model. However, the quality of generated images
are only guaranteed when latent embedding lies close to the prior distribution.
In this work, we propose to restrict the feasible region within the prior
manifold. This is accomplished with a non-parametric metric for two
distributions: the Maximum Mean Discrepancy (MMD). Moreover, we model the
degradation process directly as a conditional distribution. We show that our
model performs well in measuring the similarity between restored and degraded
images. Instead of optimizing the long criticized pixel-wise distance over
degraded images, we rely on such model to find visual pleasing images with high
probability. Our simultaneous restoration and enhancement framework generalizes
well to real-world complicated degradation types. The experimental results on
perceptual quality and no-reference image quality assessment (NR-IQA)
demonstrate the superior performance of our method.
