  Kernel estimation is generally one of the key problems for blind image
super-resolution (SR). Recently, Double-DIP proposes to model the kernel via a
network architecture prior, while KernelGAN employs the deep linear network and
several regularization losses to constrain the kernel space. However, they fail
to fully exploit the general SR kernel assumption that anisotropic Gaussian
kernels are sufficient for image SR. To address this issue, this paper proposes
a normalizing flow-based kernel prior (FKP) for kernel modeling. By learning an
invertible mapping between the anisotropic Gaussian kernel distribution and a
tractable latent distribution, FKP can be easily used to replace the kernel
modeling modules of Double-DIP and KernelGAN. Specifically, FKP optimizes the
kernel in the latent space rather than the network parameter space, which
allows it to generate reasonable kernel initialization, traverse the learned
kernel manifold and improve the optimization stability. Extensive experiments
on synthetic and real-world images demonstrate that the proposed FKP can
significantly improve the kernel estimation accuracy with less parameters,
runtime and memory usage, leading to state-of-the-art blind SR results.
