  Modern single image super-resolution (SISR) system based on convolutional
neural networks (CNNs) achieves fancy performance while requires huge
computational costs. The problem on feature redundancy is well studied in
visual recognition task, but rarely discussed in SISR. Based on the observation
that many features in SISR models are also similar to each other, we propose to
use shift operation to generate the redundant features (i.e., Ghost features).
Compared with depth-wise convolution which is not friendly to GPUs or NPUs,
shift operation can bring practical inference acceleration for CNNs on common
hardware. We analyze the benefits of shift operation for SISR and make the
shift orientation learnable based on Gumbel-Softmax trick. For a given
pre-trained model, we first cluster all filters in each convolutional layer to
identify the intrinsic ones for generating intrinsic features. Ghost features
will be derived by moving these intrinsic features along a specific
orientation. The complete output features are constructed by concatenating the
intrinsic and ghost features together. Extensive experiments on several
benchmark models and datasets demonstrate that both the non-compact and
lightweight SISR models embedded in our proposed module can achieve comparable
performance to that of their baselines with large reduction of parameters,
FLOPs and GPU latency. For instance, we reduce the parameters by 47%, FLOPs by
46% and GPU latency by 41% of EDSR x2 network without significant performance
degradation.
