Original: L EARNING E FFICIENT I MAGE S UPER -R ESOLUTION N ETWORKS VIA S TRUCTURE -R EGULARIZED P RUNINGAnonymous authors
Revised: L EARNING E FFICIENT I MAGE S UPER -R ESOLUTION N ETWORKS VIA S TRUCTURE -R EGULARIZED P RUNING

Original: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Lightweight image super-resolution (SR) networks have obtained promising results with moderate model size.
Revised: Several image super-resolution (SR) networks have been proposed of late for efﬁcient SR, achieving promising results.

Original: However, they are impractical or neglected to be extended to larger networks.
Revised: However, they are still not lightweight enough and neglect to be extended to larger networks.

Original: At the same time, model compression techniques, like neural architecture search and knowledge distillation, typically consume considerable computation resources.
Revised: At the same time, model compression techniques, like neural architecture search and knowledge distillation, typically consume considerable computation resources.

Original: In contrast, network pruning is a cheap and effective model compression technique.
Revised: In contrast, network pruning is a cheap and effective model compression technique.

Original: However, it is hard to be applied to SR networks directly, because ﬁlter pruning for residual blocks is well-known tricky.
Revised: However, it is hard to be applied to SR networks directly because ﬁlter pruning for residual blocks is well-known tricky.

Original: To address the above issues, we propose structure-regularized pruning (SRP), which imposes regularization on the pruned structure to make sure the locations of pruned ﬁlters are aligned across different layers.
Revised: To address the above issues, we propose structure-regularized pruning (SRP), which imposes regularization on the pruned structure to ensure the locations of pruned ﬁlters are aligned across different layers.

Original: Speciﬁcally, for the layers connected by the same residual, we select the ﬁlters of the same indices as unimportant ﬁlters.
Revised: Speciﬁcally, for the layers connected by the same residual, we select the ﬁlters of the same indices as unimportant ﬁlters.

Original: To transfer the expressive power in the unimportant ﬁlters to the rest of the network, we employ L 2 regularization to drive the weights towards zero so that eventually their absence will cause minimal performance degradation.
Revised: To transfer the expressive power in the unimportant ﬁlters to the rest of the network, we employ L 2 regularization to drive the weights towards zero so that eventually, their absence will cause minimal performance degradation.

Original: We apply SRP to train efﬁcient image SR networks, resulting in a lightweight network SRPN-L and a very deep one SRPN.
Revised: We apply SRP to train efﬁcient image SR networks, resulting in a lightweight network SRPN-Lite and a very deep one SRPN.

Original: We conduct extensive comparisons with both lightweight and largerimage SR networks.
Revised: We conduct extensive comparisons with both lightweight and larger networks.

Original: SRPN-L and SRPN achieve superior performance gains over recent methods quantitatively and visually.
Revised: SRPN-Lite and SRPN perform favorably against other recent efﬁcient SR approaches quantitatively and visually.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Image super-resolution (SR) is a fundamental computer vision task, which aims to recover a highresolution (HR) image from its low-resolution (LR) counterpart.
Revised: Image super-resolution (SR), a classic task in computer vision, aims to recover a high-resolution (HR) image based on its low-resolution (LR) counterpart.

Original: In general, image SR is ill-posed as a many-to-one mapping problem.
Revised: Essentially, image SR is ill-posed as a many-to-one mapping problem.

Original: To alleviate this problem, plenty of deep convolutional neural networks (CNNs) (Dong et al., 2014; 2016; Kim et al., 2016b; Tai et al., 2017b) have been investigated to achieve accurate mapping from LR image to its HR target.
Revised: To tackle this problem, plenty of deep convolutional neural networks (CNNs) (Dong et al., 2014; 2016; Kim et al., 2016b; Zhang et al., 2018c; 2020; 2021) have been investigated to learn the accurate mapping from LR image to its HR target.

Original: Deep CNNis ﬁrstly utilized for image SR in SRCNN (Dong et al., 2014) and  continuously shows promising SR performance.
Revised: Deep CNN for image SR is pioneered by SRCNN (Dong et al., 2014) and has continuously shown promising SR performance.

Original: There are only three convolutional (Conv) layers in SRCNN, hindering its performance.
Revised: There are only three convolutional (Conv) layers in SRCNN, constraining its expressivity.

Original: Kim et al . increased the network depth in VDSR (Kim et al., 2016a) with residual learning and obtained notable improvements over SRCNN.
Revised: Kim et al . increased the network depth in VDSR (Kim et al., 2016a) with residual learning and obtained notable improvements over SRCNN.

Original: Lim et al .
Revised: Lim et al .

Original: (Lim et al., 2017) built a much deeper network EDSR by using simpliﬁed residual blocks.
Revised: (Lim et al., 2017) built a much deeper network EDSR by using simpliﬁed residual blocks.

Original: Zhang et al .
Revised: Zhang et al .

Original: (Zhang et al., 2018b) proposed RCAN, which is one of the deepest networks in SR.
Revised: (Zhang et al., 2018b) proposed RCAN, which is one of the deepest networks in SR at present.

Original: With increased network size,very deep networks, like EDSR (Lim et al., 2017) and RCAN (Zhang et al., 2018b), have achieved remarkable SR performance.
Revised: Empowered by increased network size, deep SR models like EDSR (Lim et al., 2017) and RCAN (Zhang et al., 2018b) have seen remarkable SR performance.

Original: However, they also suffer from heavy model parameters, number of operations, and inference time.
Revised: However, as a cost, the large model size brings about problems such as excessive memory footprint, slow inference speed.

Original: Therefore, it is  infeasible to directly deploy them on resource-limited platformswithout neural processing units or off-chip memory (Lee et al., 2020).
Revised: It is thereby infeasible to directly deploy them on resource-constrained platforms (Lee et al., 2020).

Original: From this point of view, more and more works introduce efﬁcient network architecturesfor lightweight SR networks (Ahn et al., 2018; Luo et al., 2020).
Revised: Aiming for efﬁcient SR, more and more works introduce lightweight network architectures (Ahn et al., 2018; Luo et al., 2020).

Original: Ahn et al .
Revised: Ahn et al .

Original: proposed cascading residual network (CARN)
Revised: proposed cascading residual network (CARN)

Original: (Ahn et al., 2018).
Revised: (Ahn et al., 2018).

Original: Hui et al .
Revised: Hui et al .

Original: proposed information multi-distillation network (IMDN) (Hui et al., 2019).
Revised: proposed information multi-distillation network (IMDN) (Hui et al., 2019).

Original: Lee et al .
Revised: Lee et al .

Original: introduced knowledge distillation (Hinton et al., 2014) for image SR (Lee et al., 2020).
Revised: introduced knowledge distillation (Hinton et al., 2014) for image SR (Lee et al., 2020).

Original: Besides, neural architecture search (NAS) (Zoph & Le, 2017; Elsken et al., 2019) was also utilized for lightweight SR model (Chu et al., 2019a).
Revised: Besides, neural architecture search (NAS) (Zoph & Le, 2017; Elsken et al., 2019) was also utilized for lightweight SR model (Chu et al., 2019a).

Original: However, there are still several downsides to these networks: (1) There is still an obvious performance gap between those lightweight models and the very deep models; (2) The training of these methods can consume heavy computation resources.
Revised: However, there are still several downsides to these networks: (1) There is still an obvious performance gap between those lightweight models and the very deep models; (2) These methods can consume considerable computation resources for training.

Original: For example,  8 Tesla V100 GPUsare needed to train a single network for
Revised: For example, (Chu et al., 2019a) require 8 Tesla V100 GPUs to train a single network, and the

Original: [Figure]
Revised: [Figure]

Original:  about three days in (Chu et al., 2019a); (3) It is hard for most lightweight SR network designs to generalize to very large networks and achieve superior performance at the same time.
Revised: training takes about three days; (3) It is hard for most lightweight SR network designs to generalize to more large-scale networks and achieve superior performance at the same time.

Original: In short, more effective, resource friendly, and generally lightweight SR networks are still in need.
Revised: In short, more effective, resource-friendly, and generic lightweight SR networks are still in need.

Original: On the other hand, neural network pruning is well-known as an effective technique to reduce model complexity (Reed, 1993; Sze et al., 2017).
Revised: On the other hand, neural network pruning is well-known as an effective technique to reduce model complexity (Reed, 1993; Sze et al., 2017).

Original: For acceleration,researchers mainly focus on ﬁlter pruning (a.k.a.
Revised: For acceleration, ﬁlter pruning (a.k.a.

Original: structured pruning)
Revised: structured pruning)

Original: (Li et al., 2017) rather than weight-element pruning (a.k.a.
Revised: (Li et al., 2017) attracts more attention than weight-element pruning (a.k.a.

Original: unstructured pruning)
Revised: unstructured pruning)

Original: (Han et al., 2015; 2016b).
Revised: (Han et al., 2015; 2016b).

Original: Marrying ﬁlter pruning with image SR seems a plausible solution to strike a better performance-complexity trade-off.
Revised: Introducing ﬁlter pruning into image SR is a promising solution to achieve a good trade-off between performance and complexity.

Original: However,  ﬁlter pruning methodsin classiﬁcation can hardly translate to  SR networks directly.
Revised: However, it is not easy to apply ﬁlter pruning methods to image SR networks directly.

Original: The main reason is that, residual connections are well-known hard to prune in structured pruning (Li et al., 2017) while they are extensively used in state-of-the-art  SR networks (e.g., EDSR (Lim et al., 2017) has 32 residual blocks; RCAN (Zhang et al., 2018b) even has nested residual blocks).
Revised: This is mainly because residual connections are well-known difﬁcult to prune in structured pruning (Li et al., 2017). On the other hand, they are extensively used in state-of-the-art (SOTA) image SR methods (e.g., EDSR (Lim et al., 2017) hasresidual blocks; RCAN (Zhang et al., 2018b) even has nested residual blocks).

Original: To tackle the above issue, we propose Structure-Regularized Pruning (SRP), which imposes regularization on the pruned structure to make sure the locations of pruned ﬁlters are aligned across different layers.
Revised: To tackle the above issue, we propose Structure-Regularized Pruning (SRP), which imposes regularization on the pruned structure to ensure the locations of pruned ﬁlters are aligned across different layers.

Original: Speciﬁcally, for the layers connected by the same residual, we select the ﬁlters of the same indices as unimportant ﬁlters (i.e., those we will remove ﬁnally).
Revised: Speciﬁcally, for the layers connected by the same residual, we select the ﬁlters of the same indices as unimportant ﬁlters (i.e., those we will remove ﬁnally).

Original: In order to transfer the expressive power in the unimportant ﬁlters to the remainder of the network, we employ L 2 regularization to drive the weights towards zero gradually so that eventually their absence will incur negligible performance loss.
Revised: To transfer the expressive power in the unimportant ﬁlters to the remainder of the network, we employ L 2 regularization to drive the weights towards zero gradually so that eventually, their absence will incur negligible performance loss.

Original: To the best of our knowledge, our SRP is  the ﬁrst attempt to leverage structured pruning for efﬁcient image SR.
Revised: To the best of our knowledge, our SRP is one of the leading works (Zhang et al., 2021) to leverage structured pruning for efﬁcient image SR.

Original: Our main contributions can be summarized as follows:
Revised: Our main contributions are as follows:

Original: • We propose a network structure-regularized pruning (SRP) method to learn efﬁcient image SR networks.
Revised: • We propose a network structure-regularized pruning (SRP) method to learn efﬁcient image SR networks.

Original: We try to jointly train image SR models with network pruning simultaneously to achieve high reconstruction performance as well as efﬁciency.
Revised: We try to jointly train image SR models with network pruning simultaneously to achieve high reconstruction performance as well as efﬁciency.

Original: • Our SRP provides a general idea to structurally prune networks with extensive residual connections.
Revised: • Our SRP provides a general idea to structurally prune networks, which consists of extensive residual connections.

Original: The introduction of regularization as a pruning tool manages to maintain the expressivity of the original network while peeling off the unnecessary redundancy.
Revised: The introduction of regularization as a pruning tool manages to maintain the expressivity of the original network while peeling off the unnecessary redundancy.

Original: • We employ SRP to train efﬁcient image SR networks, resulting in a lightweight network (named SRPN-L) and a very deep one (named SRPN).
Revised: • We employ SRP to train efﬁcient image SR networks, resulting in a lightweight network (named SRPN-Lite) and a very deep one (named SRPN).

Original: We achieve superior performance gains on both lightweight and large image SR networks.
Revised: We achieve superior performance gains on both lightweight and large image SR networks.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: Deep Image SR Models.
Revised: Deep Image SR Models.

Original: Dong et al .
Revised: Dong et al .

Original: (Dong et al., 2014) ﬁrstly proposed SRCNN for image SR and achieved superior performance with only 3 convolutional (Conv) layers.
Revised: (Dong et al., 2014) ﬁrstly proposed SRCNN for image SR and achieved superior performance with only 3 convolutional (Conv) layers.

Original: Residual learning was introduced in VDSR (Kim et al., 2016a), reaching 20 Conv layers and a signiﬁcant improvement over
Revised: Residual learning was introduced in VDSR (Kim et al., 2016a), reaching 20 Conv layers and a signiﬁcant improvement over

Original: SRCNN.
Revised: SRCNN.

Original: Tai et al .
Revised: Tai et al .

Original: later introduced memory block in MemNet (Tai et al., 2017b) for deeper network structure.
Revised: later introduced memory block in MemNet (Tai et al., 2017b) for deeper network structure.

Original: Lim et al .
Revised: Lim et al .

Original: (Lim et al., 2017) simpliﬁed the residual block (He et al., 2016) and constructed deeper and wider networks with a large number of parameters.
Revised: (Lim et al., 2017) simpliﬁed the residual block (He et al., 2016) and constructed deeper and wider networks with a large number of parameters.

Original: Zhang et al .
Revised: Zhang et al .

Original: (Zhang et al., 2018b) proposed an even deeper network, residual channel attention network (RCAN), where the attention mechanism was ﬁrstly introduced in image SR.
Revised: (Zhang et al., 2018b) proposed an even deeper network, residual channel attention network (RCAN), where the attention mechanism was ﬁrstly introduced in image SR.

Original: Liu et al .
Revised: Liu et al .

Original: proposed
Revised: proposed

Original: FRANet
Revised: FRANet

Original: (Liu et al., 2020) to make the residual features more focused on critical spatial contents.
Revised: (Liu et al., 2020) to make the residual features focus on critical spatial contents.

Original: Later, Zhang et al .
Revised: Later, Zhang et al .

Original: (Zhang et al., 2019) proposed residual non-local attention for image restoration, including image SR.
Revised: (Zhang et al., 2019) proposed residual non-local attention for image restoration.

Original: Mei et al .
Revised: Mei et al .

Original: proposed CSNLN (Mei et al., 2020) by combining local, in-scale/cross-scale non-local feature correlations, and external statistics.
Revised: proposed CSNLN (Mei et al., 2020) by combining local, in-scale/cross-scale non-local feature correlations, and external statistics.

Original: Most of them have achieved state-of-the-art results with deeper and wider networks.
Revised: Most of those methods have achieved SOTA results.

Original: However, they suffer from huge model size (i.e., network parameter number) and/or heavy computation operations (i.e., FLOPs).
Revised: However, they suffer from huge model size (i.e., network parameter number) and/or heavy computation operations (i.e., FLOPs).

Original: [Figure]
Revised: [Figure]

Original:  𝒋− 𝒋+ 𝒋
Revised: ) 𝒋− 𝒋+ 𝒋

Original: F iF
Revised: F iF

Original: i
Revised: i

Original: F i+ F i+
Revised: F i+ F i+

Original: W i
Revised: W i

Original: W i+  1 33 6 

Revised: W i+  1 33 6

Original: [Figure]
Revised: [Figure]

Original: Lightweight Image SR Models.
Revised: Lightweight Image SR Models.

Original: Recent years have been rising interest in investigating lightweight image SR models.
Revised: Recent years have been rising interest in investigating lightweight image SR models.

Original: These approaches try to design lightweight architectures, which mainly take advantage of recursive learning and channel splitting.
Revised: These approaches try to design lightweight architectures, which mainly take advantage of recursive learning and channel splitting.

Original: Kim et al .
Revised: Kim et al .

Original: ﬁrstly introduced recursive learning in DRCNto decrease model size (Kim et al., 2016b).
Revised: ﬁrstly decreased parameter number by utilizing recursive learning in DRCN (Kim et al., 2016b).

Original: Ahn et al .
Revised: Ahn et al .

Original: designed a cascading mechanism upon a residual networkin CARN (Ahn et al., 2018).
Revised: proposed CARN by designing a cascading mechanism upon a residual network (Ahn et al., 2018).

Original: Hui et al .
Revised: Hui et al .

Original: proposed a lightweight information multi-distillation network (IMDN) (Hui et al., 2019).
Revised: proposed a lightweight information multi-distillation network (IMDN) (Hui et al., 2019).

Original: Luo et al . designed the lattice block with butterﬂy structures (Luo et al., 2020).
Revised: Luo et al . designed the lattice block with butterﬂy structures (Luo et al., 2020).

Original: Recently, neural architecture search was introduced for image SR in FALSR (Chu et al., 2019a).
Revised: Recently, neural architecture search was applied for image SR, like FALSR (Chu et al., 2019a).

Original: Besides, model compression techniques, like knowledge distillation, have been investigated for image SR.
Revised: Also, model compression techniques have been explored for image SR.

Original: He et al .
Revised: He et al .

Original: proposed knowledge distillation based feature-afﬁnity for efﬁcient image SR (He et al., 2020).
Revised: proposed knowledge distillation based feature-afﬁnity for efﬁcient image SR (He et al., 2020).

Original: Lee et al .
Revised: Lee et al .

Original: trained a  teacher network todistill its knowledge to a student  (Lee et al., 2020).
Revised: distilled knowledge from a larger teacher network to a student one (Lee et al., 2020).

Original: Although those lightweight networks have achieved great progress,  we still need to investigate deeper for more efﬁcient image SR models.
Revised: Those lightweight image SR models have obtained great progress, but we still need to investigate deeper for more efﬁcient image SR models.

Original: Neural Network Pruning.
Revised: Neural Network Pruning.

Original: Pruning aims to remove parameters in a neural network without compromising its performance seriously (Reed, 1993; Sze et al., 2017).
Revised: Network pruning aims to eliminate redundant parameters in a neural network without compromising its performance seriously (Reed, 1993; Sze et al., 2017).

Original: It mainly falls into two groups: ﬁlter pruning (a.k.a. structured pruning) 1 and weight-element pruning (a.k.a. unstructured pruning).
Revised: The methodology of pruning mainly falls into two groups: ﬁlter pruning (or more generally known as structured pruning) * and weight-element pruning (also referred to as unstructured pruning).

Original: The former aims to remove weights by ﬁlters (i.e., 4-d tensors), while the latter removes weights by single elements (i.e., a scalar).
Revised: The former aims to remove weights by ﬁlters (i.e., 4-d tensors), while the latter removes weights by single elements (i.e., scalars).

Original: Structured pruning results in regular sparsity after pruning.
Revised: Structured pruning results in regular sparsity after pruning.

Original: It does not demand any special hardware features to achieve considerable practical acceleration.
Revised: It does not demand any special hardware features to achieve considerable practical acceleration.

Original: In contrast, unstructured pruning leads to irregular sparsity.
Revised: In contrast, unstructured pruning leads to irregular sparsity.

Original: Leveraging the irregular sparsity for acceleration typically demands special software libraries, while past works have shown the practical speedup is very limited (Wen et al., 2016), unless using customized hardware platform (Han et al., 2016a).
Revised: Leveraging the irregular sparsity for acceleration typically demands special software supports, while past works have shown the practical speedup is very limited (Wen et al., 2016), unless using customized hardware platforms (Han et al., 2016a).

Original: In this paper, we focus on ﬁlter pruning  for easy acceleration.
Revised: In this paper, we tackle ﬁlter pruning instead of weight-element pruning for effortless acceleration.

Original: Most efforts in pruning (mainly in classiﬁcation task) have been spent on ﬁnding a better pruning criterion to select unimportant weights (Reed, 1993; Sze et al., 2017).
Revised: The major efforts in pruning (mainly in image classiﬁcation) have been focusing on proposing a more sound pruning criterion to select unimportant weights (Reed, 1993; Sze et al., 2017).

Original: Magnitude-based (Han et al., 2015; 2016b; Li et al., 2017) is the most prevailing criterion, which we will also employ to develop our method in this paper.As far as we know, no work before has managed to apply ﬁlter pruning to compressing image SR networks. This paper is meant to ﬁll the blank.
Revised: Criteria based on weight magnitude (Han et al., 2015; 2016b; Li et al., 2017) are the most prevailing ones, which we will also employ to develop our method in this paper.

Original: P ROPOSED  M ETHOD
Revised:  M ETHODOLOGY

Original: We ﬁrst give a brief view of the problem setting about deep CNN for image SR.
Revised: We ﬁrst present an overview of the problem setting about deep CNN for image SR.

Original: We also observe that there exists heavy redundancy  in the networks.
Revised: It is also observed that excessive redundancy exists in the SR deep CNNs.

Original: To pursue more efﬁcient image SR networks, we then propose structure-regularized pruning (SRP) method  to compress them.
Revised: Then we move on to proposing our structureregularized pruning (SRP) method attempting to achieve more efﬁcient SR networks.

Original: 3. D EEP CNN FOR I MAGE SR
Revised: 3. D EEP CNN FOR I MAGE SR

Original: Image super-resolution (SR) aims to reconstruct a high-resolution (also known as super-resolved) image I SR  from its low-resolution (LR) image I LR .
Revised: In image super-resolution (SR), a high-resolution (a.k.a. super-resolved) image I SR is reconstructed from its low-resolution (LR) counterpart I LR .

Original: This image SR process can be described as
Revised: This image SR process can be described as

Original: [Equation]
Revised: [Equation]

Original: where F SR ( · ) is thedeep image SR network and Θ denotes the network parameters.
Revised: where F SR ( · ) is the image SR network, parameterized by Θ .

Original: We also model the LR image I LR from its HR counterpart  as a degradation process
Revised: Meanwhile, the LR image I LR from its HR counterpart can be formulated as a degradation process

Original: [Equation]
Revised: [Equation]

Original: where F ↓ s ( · ) downscales the original ground truth I HR with scaling factor s .
Revised: where F ↓ s ( · ) downscales the original ground truth I HR by a scale factor s .

Original: The downscaling process may introduce additional noise, blurring, compression, and/or other unknown artifacts.
Revised: This downscaling process typically introduces additional noise, blurring, compression, and/or other unknown artifacts.

Original: Meanwhile, high-frequency information will be lost more or less.
Revised: Therein, the high-frequency information will be lost more or less.

Original: Image SR models try to recover highfrequency information as much as possible.
Revised: The job of image SR models is to recover the high-frequency information as much as possible.

Original: In this work, we focus on efﬁcient neural networks withrelatively fewer parameters and computation operations, but comparable or even higher performance.
Revised: We aim to present a structural sparsity inducing SR network learning method, which can deliver more efﬁcient SR networks with fewer parameters and FLOPs.

Original: 3. S TRUCTURE -R EGULARIZED P RUNING (SRP)
Revised: 3. S TRUCTURE -R EGULARIZED P RUNING (SRP)

Original: Pruned Index Constraint .
Revised: Pruned Index Constraint .

Original: Filter pruning in residual networks is well-known tricky because the add operators in residual blocks demand the pruned ﬁlter indices  must be aligned.
Revised: Pruning ﬁlters in residual networks is well-known non-trivial as the Add operators in residual blocks require the pruned ﬁlter indices across different residual blocks must be aligned.

Original: Filter pruning within a residual block is shown in Fig. 2(b).
Revised: A ﬁgurative illustration of ﬁlter pruning within a residual block is shown in Fig. 2(b).

Original: A typical residual block (e.g., in EDSR (Lim et al., 2017), RCAN (Zhang et al., 2018b)) is made up of two convolutional layers.
Revised: A typical residual block (e.g., in EDSR (Lim et al., 2017), RCAN (Zhang et al., 2018b)) consists of two convolutional layers.

Original: All the convolutional layers can be categorized into two groups based on their connection relationship among one another.
Revised: According to the mutual connection relationship, the convolutional layers can be categorized into two groups.

Original: One group comprises the layers that can be pruned without any constraint , dubbed free Conv layers in this work; the other comprises  layers in which the ﬁlters must be pruned at the same indices , dubbed constrained Conv layers .
Revised: One group is made up with the layers that can be pruned without any constraint , dubbed free Conv layers in this work; the other comprises Conv layers whose ﬁlters must be pruned at the same indices , dubbed constrained Conv layers .

Original: For a concrete example, in Fig. 2(b), the layer W ( i )  is a free Conv layer and layer W ( i +1) is a constrained Conv layer.
Revised: Concretely, the layer W ( i ) in Fig. 2(b) is a free Conv layer, while the layer W ( i +1) is a constrained one.

Original: Owing to the pruned index constraint problem, many ﬁlter pruning methods in  classiﬁcation simply do not prune the last Conv layer ina residual blocks (Li et al., 2017; Wang et al., 2021), which can still deliver considerable speedup of practical interest.
Revised: Owing to the pruned index constraint problem, many ﬁlter pruning methods in image classiﬁcation simply do not prune the last Conv layer in residual blocks (Li et al., 2017; Wang et al., 2021), which can still deliver considerable speedup of practical interest.

Original: But, this naive solution cannot translate to the image SR networks.
Revised: Nevertheless, this doing-nothing solution can barely translate to the image SR networks if we target a considerable speedup.

Original: The fundamental reason is that image SR networks typically employ much more residual blocks and each block typically has only two Conv layers.
Revised: The root cause is that image SR networks typically employ many more residual blocks, and each block usually has only two Conv layers.

Original: In some top-performed SR networks (e.g., RCAN), there are even nested residual blocks.
Revised: In some top-performing SR networks (e.g., RCAN), there are even nested residual blocks.

Original: To see how serious this problem is, taking EDSR as an example, it has 32 residual blocks and each block has two Conv layers.
Revised: Taking EDSR for a concrete example, it has 32 residual blocks and each block has two Conv layers.

Original: Ifwe do not prune the 2nd Conv layer in a residual block, it means half of the Conv layers are not  pruned.
Revised: If the 2nd Conv layer in a residual block is spared from pruning, half of the Conv layers will not be pruned.

Original: Namely, at best , we can only achieve 2 × theoretical acceleration (measured by FLOPs reduction).
Revised: In other words, at best , we can only achieve 2 × theoretical acceleration measured by FLOPs reduction.

Original: Given the issue above, it is necessary to prune all the Conv layers in residual blocks if we seek acceleration of practical use. Thus, we need a method to align the pruned indices in all constrained Conv layers.
Revised: Given this issue, it is imperative to prune all the Conv layers in residual blocks, thus calling for an approach to align the pruned indices of all constrained Conv layers.

Original: Regularization then arises as a natural solution given its prevailing use to impose priors on the sparsity structure in pruning (Reed, 1993; Wen et al., 2016; Wang et al., 2021).
Revised: Regularization then arises as a promising solution considering it has been widely used before to impose priors on the sparsity structure in classiﬁcation (Reed, 1993; Wen et al., 2016; Wang et al., 2021).

Original: Our SRP method is based on regularization to induce desired  sparsity structure.
Revised: Our SRP method is based on regularization to form desired hardware-friendly sparsity structure.

Original: Primarily, there are three questions to answer in SRP: (1) which pruning criterion is used to choose unimportant ﬁlters; (2) which kind of regularization is chosen to induce sparsity; (3) how to schedule the pruning process to make the algorithm robust and easy to use.
Revised: Primarily, there are three questions to answer in SRP: (1) which pruning criterion is used to choose unimportant ﬁlters; (2) which kind of regularization is chosen to induce sparsity; (3) how to schedule the pruning process to make the algorithm robust and easy to use.

Original: (1) Pruning Criterion .
Revised: (1) Pruning Criterion .

Original: When it comes to formulated pruning criteria, they mainly fall into three groups: magnitude based (Han et al., 2015; Li et al., 2017), 1st-order gradient based (Molchanov et al., 2017; 2019), and 2nd-order gradient based (LeCun et al., 1990; Hassibi & Stork, 1993; Wang et al., 2019a).
Revised: When it comes to formulated pruning criteria, they mainly fall into three groups: magnitude based (Han et al., 2015; Li et al., 2017), 1st-order gradient based (Molchanov et al., 2017; 2019), and 2nd-order gradient based (LeCun et al., 1990; Hassibi & Stork, 1993; Wang et al., 2019a).

Original: The 1st-order and 2nd-order gradient based criteria are based on Taylor approximation of the increased loss when a weight is removed, pioneered by (LeCun et al., 1990).
Revised: The 1st-order and 2nd-order gradient-based criteria are based on Taylor approximation of the increased loss when a weight is removed, pioneered by (LeCun et al., 1990).

Original: They are more accurate than magnitude based one right after pruning.
Revised: They are more accurate than magnitude-based one right after pruning.

Original: However, pruning is typically followed by a ﬁnetuning process (Reed, 1993) to regain performance.
Revised: However, pruning is typically followed by a ﬁnetuning process (Reed, 1993) to regain performance.

Original: After ﬁnetuning, the advantage of the 1st-order and 2nd-order gradient based criteria usually becomes marginal.
Revised: After ﬁnetuning, the advantage of the 1st-order and 2nd-order gradient-based criteria usually becomes marginal.

Original: Meanwhile, they are typically more costly than the magnitude based criteria.
Revised: Meanwhile, they are typically more costly than the magnitude-based criteria.

Original: Taking the cost and ﬂexibility into account, we simply employ magnitude based pruning criteria.
Revised: Taking the cost and ﬂexibility into account, we simply employ magnitude-based pruning criteria.

Original: Speciﬁcally, given a pretrained SR network, for all the free Conv layers, we sort the ﬁlters in ascending order by their L 1 -norms and select the ﬁlers with the least norms as unimportant ﬁlters , by a predeﬁned pruning ratio r ∈ (0 , 1) .
Revised: Speciﬁcally, given a pretrained SR network, for all the free Conv layers, we sort the ﬁlters in ascending order by their L 1 -norms and select the ﬁlers with the least norms as unimportant ﬁlters , by a predeﬁned pruning ratio r ∈ (0 , 1) .

Original: For the constrained Conv layers, the problem becomes a little bit more complex.
Revised: For the constrained Conv layers, the problem becomes a little bit more complex.

Original: Given the pruned index constraint, ideally, we can only prune the ﬁlters in the intersection set (denoted by S ( net ) ) for the whole network,
Revised: Given the pruned index constraint, ideally, we can only prune the ﬁlters in the intersection set (denoted by S ( net ) ) for the whole network,

Original: [Equation]
Revised: [Equation]

Original: where S ( l ) represents the set of unimportant ﬁlters selected by certain pruning criterion in the l -th
Revised: where S ( l ) represents the set of unimportant ﬁlters selected by certain pruning criterion in the l -th

Original: Conv layer; C denotes the set of constrained Conv layers.
Revised: Conv layer; C denotes the set of constrained Conv layers.

Original: For top-performed image SR networks, |C| ( | · | denotes cardinality of a set) is at the order of magnitude of hundreds (e.g., |C| =for RCAN).
Revised: For top-performed image SR networks, |C| ( |·| denotes cardinality of a set) is at the order of magnitude of hundreds (e.g., |C| = 200 for RCAN).

Original: Meanwhile, given a pretrained model, its pruned ﬁlter indices by L 1 -norm are usually random across different layers.
Revised: Meanwhile, given a pretrained model, its pruned ﬁlter indices by L 1 -norm are usually random across different layers.

Original: As a consequence, |S ( net ) | will be rather small (in practice, on 
Revised: As a consequence, |S ( net ) | will be rather small (in practice, on RCAN, we even observe |S ( net ) |

Original: RCAN, we even observe |S ( net ) | = 0 when we try to prune 32 ﬁlters out of 64).
Revised: = 0 when we try to prune 32 ﬁlters out of 64).

Original: This issue brings much trouble to user control when we want to prune a layer to a pre-deﬁned number of ﬁlters.
Revised: This issue brings much trouble to user control when we want to prune a layer to a pre-deﬁned number of ﬁlters.

Original: To resolve this problem, we choose to randomly select a set of ﬁlters as unimportant and ﬁxed during the whole pruning process.
Revised: To resolve this problem, we choose to randomly select a set of ﬁlters as unimportant and ﬁxed during the whole pruning process.

Original: In practice, we ﬁnd this simple scheme works pretty well.
Revised: In practice, we ﬁnd this simple scheme works pretty well.

Original: The reason enabling us to do so is that regularization for pruning is a gradual process.
Revised: The reason enabling us to do so is that regularization for pruning is a gradual process.

Original: It allows abundant time for the network to recover from the introduced penalty term (this will be empirically veriﬁed in our experiments, see Fig. 3).
Revised: It allows ample time for the network to recover from the introduced penalty term (this will be empirically veriﬁed in our experiments, see Fig. 3).

Original: Apart from aligning cross-layer sparsity structure, this is the second reasonthat we choose regularization for pruning in our algorithm.
Revised: Apart from aligning cross-layer sparsity structure, this is the second reason we choose regularization for pruning in our algorithm.

Original: (2) Regularization Form .
Revised: (2) Regularization Form .

Original: Although L 1 regularization is well-known to induce sparsity in machine learning, it is hard to tune the coefﬁcient to strike a good balance between sparsity and performance.
Revised: Although L 1 regularization is well-known to induce sparsity in machine learning, it is hard to tune the coefﬁcient to realize a desired trade-off between sparsity and performance.

Original: Therefore, L 2 regularization is  adopted in our method given its easy control.
Revised: Instead, L 2 regularization is thereby adopted in our method given its more tamable control over the sparsiﬁcation process – note the gradient of L 2 regularization is proportion to the weight magnitude while the gradient of L 1 regularization is not.

Original: Speciﬁcally, given the loss function L of a neural network Θ , the total error function E withthe proposed adaptive Lregularization term can be formulated as
Revised: Speciﬁcally, given the loss function L of a neural network Θ , the total error function E with adaptive L 2 regularization term is formulated as

Original: [Equation]
Revised: [Equation]

Original: where D stands for the training dataset; W [Equation] l ) j refers to the j -th ﬁlter in l -th Conv layer; α ( l ) j stands for the L 2 regularization co-efﬁcient for that ﬁlter.
Revised: where D stands for the training dataset; W ( l ) j refers to the j -th ﬁlter in l -th Conv layer; α ( l ) j stands for the L 2 regularization co-efﬁcient for that ﬁlter.

Original: If a ﬁlter j falls into the set of unimportant ﬁlters
Revised: If a ﬁlter j falls into the set of unimportant ﬁlters

Original: S ( l ) , α ( l ) j > 0 ; otherwise, α ( l ) j = 0 since we do not restrict the learning of important ﬁlters.
Revised: S ( l ) , α ( l ) j > 0 ; otherwise, α ( l ) j = 0 since we do not restrict the learning of important ﬁlters.

Original: (3) Regularization Schedule .
Revised: (3) Regularization Schedule .

Original: As mentioned above, for the constrained Conv layers, we select the unimportant ﬁlters randomly.
Revised: As mentioned above, for the constrained Conv layers, we select the unimportant ﬁlters randomly.

Original: To mitigate the side effect of this sub-optimal choice, the network should be provided with abundant training iterations to adapt so that it can transfer its expressive power to the remaining part of the network.
Revised: To mitigate the side effect of this sub-optimal choice, the network should be provided with abundant training iterations to adapt so that it can transfer its expressive power to the remaining part of the network.

Original: To achieve this goal, we choose to gradually increase the penalty strength, drawing inspirations from (Wang et al., 2019b; 2021),
Revised: To achieve this goal, we choose to gradually increase the penalty strength, drawing inspirations from (Wang et al., 2019b; 2021),

Original: [Equation]
Revised: [Equation]

Original: where ∆ (a pre-deﬁned constant) is a small amount that the penalty strength is increased each time.
Revised: where ∆ (a pre-deﬁned constant) is a small amount that the penalty strength is increased each time.

Original: This design allows us to raise the penalty strength to avery large amount eventually (like 0.5 in our experiments, note the normal L 2 regularization co-efﬁcient is at the scale of 10 − 4 ).
Revised: This design allows us to raise the penalty strength to a large amount eventually (like 0.5 in our experiments; in contrast, note the normal L 2 regularization co-efﬁcient is at the scale of 10 − 4 ).

Original: As a result, the unimportant weights are driven very close to zero and removing them will hardly hurt the performance.
Revised: As a result, the unimportant weights are driven rather close to zero, and removing them will hardly hurt the performance.

Original: Besides, α is updated every T iterations during training for stability.
Revised: Besides, α is updated every T iterations during training for stability.

Original: As a termination condition, we introduce a ceiling limit τ (a pre-deﬁned constant) for the regularization co-efﬁcient α .
Revised: To indicate when to terminate the pruning process, a ceiling limit τ is introduced for the regularization co-efﬁcient α .

Original: When all the α  for unimportant ﬁlters reaches τ , the pruning process is ﬁnished.
Revised: The pruning is over when all the α ’s for unimportant ﬁlters arise to τ .

Original: 3. A RM I MAGE SR M ODELS WITH SRP
Revised: 3. L EARNING E FFICIENT I MAGE SR M ODELS VIA SRP

Original: The proposed SRPmethod can be applied to state-of-the-art SR methods in a plug-and-play manner – simply replace the loss function L in Eq.
Revised: SRP can be applied to top-performing SR algorithms in a plug-and-play manner – simply replace the loss function L in Eq.

Original: (4) with the loss  function of an SR method. All the features in the original SR method can stay as they are.
Revised: (4) with the loss objective function of an SR approach of interest.

Original: The proposed L 2 regularization can be implemented on any deep learning framework very easily.
Revised: The proposed L 2 regularization can be implemented on any deep learning framework very easily.

Original: When the pruning process is ﬁnished, we remove the unimportant ﬁlters (not zeroing out them only, but literally removing them from the model), which results in a small model.
Revised: Upon ﬁnishing pruning, we take away the unimportant ﬁlters (not zeroing out them only, but literally removing them from the model), which will give us a compact SR model.

Original: Then we ﬁnetune the small model  to regain performance following the common practice in pruning (Reed, 1993).
Revised: Finally, the compact model will be ﬁnetuned to regain performance following the common practice in pruning (Reed, 1993).

Original: 3. I MPLEMENTATION D ETAILS
Revised: 3. I MPLEMENTATION D ETAILS

Original: Here we elaborate how to apply SRP to constructing lightweight andvery deep image SR networks.
Revised: We explain how to apply SRP to both lightweight and deep image SR networks to demonstrate the method can work with a wide range of SR networks under different computation budgets.

Original: Lightweight Network.
Revised: Lightweight Networks.

Original: First, we revise EDSR baseline(i.e.16 residual blocks) (Lim et al., 2017)  by removing the ﬁnal Conv layer to save parameters.
Revised: First, we adapt the EDSR baseline (Lim et al., 2017) with 16 residual blocks by removing the ﬁnal convolutional layer to save parameters.

Original: Same as IMDN (Hui et al., 2019), the reconstruction was done within the pixel-shufﬂe layer (Shi et al., 2016).
Revised: The reconstruction upscaling is realized by the pixel-shufﬂe layer (Shi et al., 2016) following common practice.

Original: We set the channel number in  revised EDSR baseline as 256 and then prune it to 45.
Revised: We set the channel number in the revised EDSR baseline as 256 and then prune it to 45.

Original: For × 2, we reduce the number of parameters from 19.5M to 609K and name the compressed model as SRPN-L.
Revised: For × 2 scale, we reduce the number of parameters from 19.5M to 609K and name the compressed model as SRPN-Lite.

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Revised: ); interestingly, the important ones arise spontaneously.

Original: Very Deep Network.
Revised: Deep Networks.

Original: To apply SRP to  very deep networks, like over 400 Conv layers in RCAN, we revise RCAN(Zhang et al., 2018b) by removing all  channel attention modules.
Revised: To apply SRP to the very deep network RCAN (Zhang et al., 2018b), a representative top-performing deep SR network with over 400 Conv layers, we revise RCAN by removing all the channel attention modules (Zhang et al., 2018b).

Original: We set the channel number in the revised RCAN  as 96 and then prune it to 64.
Revised: The channel number in the revised RCAN is chosen as 96 and then pruned to 64.

Original: For × 2, we reduce the number of parameters from 34.5M to 15.3M and name the compressed model as SRPN.
Revised: For × 2 scale, we reduce the number of parameters from 34.5M to 15.3M and dub the compressed model as SRPN.

Original:  E XPERIMENTAL R ESULTS 4. S ETTINGS
Revised:  E XPERIMENTAL R ESULTS 4. S ETTINGS

Original: Data and Evaluation.
Revised: Data and Evaluation.

Original: We use DIV2K dataset (Timofte et al., 2017)  as training data, following most recent works (Timofte et al., 2017; 
Revised: We use DIV2K dataset (Timofte et al., 2017) and Flickr2K Lim et al. (2017) as training data, following most recent works (Timofte et al., 2017; Lim et al., 2017;

Original: Lim et al., 2017; Zhang et al., 2018a; Haris et al., 2018).
Revised: Zhang et al., 2018a; Haris et al., 2018).

Original: For testing, we use ﬁve standard benchmark datasets: Set5 (Bevilacqua et al., 2012), Set14 (Zeyde et al., 2010), B100 (Martin et al., 2001), Urban100 (Huang et al., 2015), and Manga109 (Matsui et al., 2017).
Revised: For testing, we use ﬁve standard benchmark datasets: Set5 (Bevilacqua et al., 2012), Set14 (Zeyde et al., 2010), B100 (Martin et al., 2001), Urban100 (Huang et al., 2015), and Manga109 (Matsui et al., 2017).

Original: The SR results are evaluated with PSNR and SSIM (Wang et al., 2004) on  Y channel of transformed YCbCr space.We also provide model size and FLOPs (a.k.a. Mult-Adds) comparisons.
Revised: The SR results are evaluated with PSNR and SSIM (Wang et al., 2004) on the Y channel in YCbCr space.

Original: Training Settings.
Revised: Training Settings.

Original: Following (Zhang et al., 2018b),we perform data augmentation on the training images, which are randomly rotated by 90 ◦ , 180 ◦ , 270 ◦ and ﬂipped horizontally.
Revised: Following (Zhang et al., 2018b), data augmentation is used in training – training images are randomly rotated by 90 ◦ , 180 ◦ , 270 ◦ and ﬂipped horizontally.

Original: Each training batch consists of 16 LR color patches, whose sizeis 48 × 48.
Revised: Image patches (patch size 48 × 48) are cropped out to form each training batch.

Original: Our SRP model is trained by ADAM optimizer (Kingma & Ba, 2014)  with β 1 =0.9, β 2 =0.999, and (cid:15) =10 − 8 .
Revised: Adam optimizer (Kingma & Ba, 2014) is adopted for training with β 1 =0.9, β 2 =0.999, and (cid:15) = 10 − 8 .

Original: We set the initial learning rate as 10 − 4 and then decrease it to half every 2 × 10 5 iterations of back-propagation.
Revised: Initial learning rate is set to 10 − 4 and then decayed by factor 0 . 5 every 2 × 10 5 iterations.

Original: We use PyTorch (Paszke et al., 2017) to implement our models with a Tesla V100 GPU 2 .
Revised: We use PyTorch (Paszke et al., 2017) to implement our models with a Tesla V100 GPU † .

Original: 4. A BLATION S TUDY
Revised: 4. A BLATION S TUDY

Original: For ablation study, we use EDSR baseline (i.e., 16 residual blocks) (Lim et al., 2017) as backbone, because it isa widely used image SR baseline with public code 3 and results.
Revised: The EDSR baseline with 16 residual blocks (Lim et al., 2017) ‡ is used as the backbone for ablation study given its broad use in the community.

Original: Comparison with Baseline  Methods .
Revised: Comparison with Baseline Pruning Methods .

Original: We ﬁrst conduct ablation study to show the proposed pruning method is effective.
Revised: To start with, we show that our SRP is more effective than the available baseline compact image SR network learning methods. Two baseline methods [Table] [Table]

Original: Two baseline methods are compared: training from scratch and the original L 1norm pruning (Li et al., 2017).
Revised: are compared to: training from scratch and the original L 1 -norm pruning (Li et al., 2017).

Original: The results are shown in Tab. 1.
Revised: The results are presented in Tab. 1.

Original: As seen, our pruned network delivers the best PSNR consistently across different pruning ratios.This shows our method is not [Table] LapSRN
Revised: As we can see, our pruned network delivers the best PSNR consistently across different pruning ratios.

Original: [Table] [Table] only effective (better than scratch training), but also more effective than naively applying pruning method in classiﬁcation to SR (better than L 1 -norm). In addition, under a larger pruning ratio, the advantage of our method over scratch training or L 1 -norm  is more pronounced in general, suggesting that our method is more valuable in the extreme pruning cases.
Revised: Notably, under a greater pruning ratio, the advantage of our method over scratch training or L 1 -norm pruning is more evident in general, suggesting that our method is more valuable in aggressive pruning cases.

Original: Notably, our method  also adopts the
Revised: Of particular note is that, our method SRP also adopts the

Original: L 1 -norm aspruning criterion, the  same as (Li et al., 2017).
Revised: L 1 -norm as the scoring criterion to select unimportant ﬁlters, same as (Li et al., 2017).

Original: However, our results are signiﬁcantly better  than theirs.
Revised: Nevertheless, our method delivers signiﬁcantly better results than theirs.

Original: The  reason is that they do not impose any regularization on the pruned structure, thus the kept feature map channels are misaligned in residual blocks after pruning.
Revised: The primary reason is that they do not impose any regularization on the pruned structure; the remaining feature maps are thus mismatched in residual blocks after pruning.

Original: In contrast, our method does not have this problem, thanks to the  proposed structure regularization.
Revised: In contrast, our method SRP is not bothered by this issue, showing the effectiveness of our proposed structure regularization.

Original: Visualization of Pruning Process .
Revised: Visualization of Pruning Process .

Original: To ﬁguratively understand how SRP works, in Fig. 
Revised: In Fig. 3, we visualize the pruning process by plotting the mean

Original: 3 we plot the mean L 1 -norm of ﬁlters in two layers of EDSR baseline during SRP training.
Revised: L 1 -norm of ﬁlters in two layers of EDSR baseline during SRP training.

Original: The ﬁlters are split into two groups, pruned and kept.
Revised: The ﬁlters are split into two groups, kept ﬁlters and pruned ﬁlters.

Original: As seen, the mean L 1 -norm of the pruned ﬁlters goes down gradually because the penalty grows stronger and stronger, driving them towards zero.
Revised: As is shown in the ﬁgure, the mean L 1 -norm of the pruned ﬁlters goes down gradually because the penalty grows stronger and stronger, driving them towards zero.

Original: Interestingly, note the L 1 -norms of the kept ﬁlters arise themselves (we do not have any regularizationterm to encourage them to grow larger).
Revised: Interestingly, note the L 1 -norms of the kept ﬁlters arise themselves. Recall that there is no explicit regularization to promote them to grow.

Original: It means the network learns to recover  itself , akin to the compensation effect  in human brain (Duffau et al., 2003).We provide more visualization results in the appendix.
Revised: In other words, the network learns to recover by itself , akin to the compensation effect found in human brain (Duffau et al., 2003).

Original: [Table] [Table]
Revised: [Table] [Table]

Original: 4. C OMPARISONS WITH L IGHTWEIGHT N ETWORKS
Revised: 4. C OMPARISONS WITH L IGHTWEIGHT N ETWORKS

Original: We compare our SRPN-L with lightweight SR networks: SRCNN (Dong et al., 2014), FSRCNN (Dong et al., 2016), VDSR (Kim et al., 2016a), DRCN (Kim et al., 2016b), LapSRN (Lai et al., 2017), DRRN (Tai et al., 2017a), MemNet (Tai et al., 2017b), CARN (Ahn et al., 2018) and
Revised: We compare our SRPN-Lite with lightweight SR networks: SRCNN (Dong et al., 2014), FSRCNN (Dong et al., 2016), VDSR (Kim et al., 2016a), DRCN (Kim et al., 2016b), LapSRN (Lai et al., 2017), DRRN (Tai et al., 2017a), MemNet (Tai et al., 2017b), CARN (Ahn et al., 2018) and

Original: IMDN (Hui et al., 2019).
Revised: IMDN (Hui et al., 2019).

Original: We provide more comparisons (e.g., PAN (Zhao et al., 2020)) in appendix.
Revised: The compared results are produced from their released code.

Original: Performance Comparisons.
Revised: Performance Comparisons.

Original: Table 2 shows PSNR/SSIM comparisons for × 2, × 3, and × 4 SR.
Revised: Table 2 shows PSNR/SSIM comparisons for × 2, × 3, and × 4 SR.

Original: When compared with all previous methods, our SRPN-L performs the best on all the datasets with all scaling factors.
Revised: When compared with all previous methods, our SRPN-Lite performs the best on all the datasets under all scaling factors.

Original: Different from careful network designs as most compared methods have done, we start with the existing EDSR baseline (Lim et al., 2017) and prune it to a much smaller network, showing the effectiveness of our proposed SRP.
Revised: Unlike most comparison methods, which achieve efﬁciency through careful network designs, our work starts with the existing EDSR baseline (Lim et al., 2017) and prunes it to a much smaller network, showing the effectiveness of our proposed SRP.

Original: Model Size and Mult-Adds.
Revised: Model Size and Mult-Adds.

Original: Compared with recent  works (e.g., MemNet, CARN, and IMDN), our SRPN-L has the least parameter number.
Revised: Our SRPN-Lite has the fewest parameter number in comparison to recent efﬁcient SR works such as MemNet, CARN, and IMDN.

Original: We also provide operations number with Mult-Adds by setting the output size as 3 × 1280 × 720.
Revised: The comparison in terms of MultAdds (measured when the output size is set to 3 × 1,280 × 720) is also presented.

Original: Our SRPN-L operates less Mult-Adds than most compared methods.
Revised: As seen, our SRPNLite costs fewer Mult-Adds than most comparison methods.

Original: Those comparisons indicate that SRP reduces parameters and operations efﬁciently.
Revised: These results demonstrate the merits of SRP against other counterparts in striking a better network performance-complexity trade-off.

Original: [Table]
Revised: [Table]

Original: Visual Comparisons.
Revised: Visual Comparisons.

Original: We further provide visual comparisons ( × 4) in Fig.4 for challenging cases.
Revised: The visual comparisons at × 4 scale are shown in Fig.

Original: We can observe that most of the compared methods cannot recover structural details with proper directions and suffer from blurring artifacts.
Revised: It is easy to see that most of the comparison algorithms can barely recover the lost texture details in the correct direction; moreover, they suffer from blurring artifacts.

Original: In  contrast, our SRPN-L can better alleviate the blurring artifacts and recover more structures. These visual comparisons are consistent with the quantitative results, demonstrating the  superiority of our SRP method.
Revised: In stark contrast, our SRPN-Lite effectively alleviates the blurring artifacts and recovers sharper structural ﬁne textures in the right way, justifying the performance superiority of our SRP approach over other methods.

Original: 4. C OMPARISONS WITH L ARGER N ETWORKS
Revised: 4. C OMPARISONS WITH L ARGER N ETWORKS

Original: Most previous lightweight SR networks neglect to extend their models to deeper and compare with larger networks.
Revised: Most previous lightweight SR networks neglect to extend their models to deeper and compare with larger networks.

Original: As stated in Sec. 3.4, we extend our SRP method to larger network training and obtain a deeper network SRPN.
Revised: As stated in Sec. 3.4, we extend our SRP method to larger network training and obtain a deeper network SRPN.

Original: In Tab. 3, we compare with large SR networks: EDSR (Lim et al., 2017), DBPN (Haris et al., 2018), RCAN (Zhang et al., 2018b), SRFBN (Li et al., 2019), SAN (Dai et al., 2019), HAN (Niu et al., 2020), IGNN (Zhou et al., 2020), CSNLN (Mei et al., 2020), and
Revised: In Tab. 3, we compare with large SR networks: EDSR (Lim et al., 2017), DBPN (Haris et al., 2018), RCAN (Zhang et al., 2018b), SRFBN (Li et al., 2019), SAN (Dai et al., 2019), HAN (Niu et al., 2020), IGNN (Zhou et al., 2020), CSNLN (Mei et al., 2020), and

Original: RFANet (Liu et al., 2020).
Revised: RFANet (Liu et al., 2020).

Original: We provide more comparisons (e.g., DRN (Guo et al., 2020)) in appendix.

Original: Performance Comparisons.
Revised: Performance Comparisons.

Original: In Tab. 3, we observe that our SRPN achieves the best PSNR/SSIM values across different test sets and scales, except for SSIM value on Set5 ( × 2).
Revised: In Tab. 3, we observe that our SRPN achieves the best PSNR/SSIM values across different test sets and scales, except for SSIM value on Set5 ( × 2).

Original: Our SRPN utilizes slightly fewer parameters (15.3M vs. 15.4M) with  same network depth as RCAN (Zhang et al., 2018b).
Revised: Our SRPN utilizes slightly fewer parameters (15.3M vs. 15.4M) with the same network depth as RCAN (Zhang et al., 2018b).

Original: Notably, our SRPN removes the channel attention module in RCAN, but still achieves better results.
Revised: Notably, our SRPN removes the channel attention module in RCAN, but still achieves better results.

Original: This is mainly because our SRPN is initially trained from a wider and stronger network (with 96 channels).
Revised: This is mainly because our SRPN is initially trained from a wider and stronger network (with 96 channels).

Original: We train the much larger network and then prune it to the target size.
Revised: We train the much larger network and then prune it to the target size.

Original: Visual Comparisons.
Revised: Visual Comparisons.

Original: We further show visual comparisons with larger networks (e.g., CSNLN,
Revised: We further show visual comparisons with larger networks (e.g., CSNLN,

Original: RFANet, and IGNN) in Fig.
Revised: RFANet, and IGNN) in Fig.

Original: 5, we can also observe similar situations as in Fig.
Revised: 5, we can also observe similar situations as in Fig.

Original: For example, in img 011, other methods suffer from heavy blurring artifacts.
Revised: For example, in img 011, other methods suffer from heavy blurring artifacts.

Original: In other cases, they may not recover clear structures in img 044 or even wrong details in img 062.
Revised: In other cases, they may not recover clear structures in img 044 or even wrong details in img 062.

Original: Our SRPN achieves more visually pleasing results with more structural details.
Revised: Our SRPN achieves more visually pleasing results with more structural details.

Original: Those comparisons demonstrate that our SRP method can be applied to train larger networks and obtain comparable or even better performance.
Revised: Those comparisons demonstrate that our SRP method can be applied to train larger networks and obtain comparable or even better performance.

Original: 4. M  ODEL C OMPRESSION C OMPARISONS
Revised: 4. M ORE C OMPARISON WITH M ODEL C OMPRESSION T ECHNIQUES

Original: We further compare our network pruning method with representative model compression techniques for image SR.
Revised: We further compare our SRP to other representative efﬁcient image SR approaches via model compression.

Original: Speciﬁcally, we compare with neural architecture search based methods (Chu et al., 2019b;a) and knowledge distillation (KD) based methods (Lee et al., 2020).
Revised: Concretely, neural architecture search based methods (Chu et al., 2019b;a) and knowledge distillation (KD) based methods (Lee et al., 2020) are compared to.

Original: We provide quantitative results  in Tab. 4. Our SRPN-L obtains the best performance with the least parameter number and Mult-Adds.
Revised: Quantitative results at × 2 scale are presented in Tab. 4, where our SRPN-Lite delivers better PSNR results across different datasets with fewer parameters and Mult-Adds.

Original: With our SRP pruning method, we do not have to search lots of architectures or train a teacher network, which  consumes extra computation resources.These comparisons show that
Revised: With our SRP pruning method, there is no need to search massive network architectures or pretraining a teacher network, which usually consumes considerable computation resources.

Original:  our SRP method has great potential  for efﬁcient image SR.We provide more discussions and comparisons with related works (e.g., DCP (Zhuang et al., 2018), DHP (Li et al., 2020)) in the appendix.
Revised: These comparisons show that our SRP, as a network pruning method, has as much potential (if not more) as other model compression techniques for efﬁcient image SR.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: Lately, researchers have investigated lightweight image super-resolution (SR) networks and achieved promising results with moderate model size and FLOPs.
Revised: Existing efﬁcient image SR methods have achieved promising results with moderate model parameters and Mult-Adds.

Original: However, they neglect to extend their models to larger networks, which usually perform much better with more parameters and/or operations with residual learning.
Revised: However, they neglect to extend their approaches to larger networks, which usually perform much better with more parameters and/or operations with residual learning.

Original: Meanwhile,  model compression techniques, like neural architecture search and knowledge distillation, have also been introduced for efﬁcient SR networks. However, they usually consume expensive computation resources.
Revised: Meanwhile, although model compression approaches, like neural architecture search and knowledge distillation, have also been exploited for efﬁcient SR networks, they usually demand excessive computation resources.

Original: Network pruning, as another promising model compression method, is hard to train lightweight SR networks directly.
Revised: Network pruning, as another promising model compression method, is hard to train lightweight SR networks directly.

Original: To address these issues, we propose structure-regularized pruning (SRP), which imposes L 2 regularization on the pruned structure to align the locations of pruned ﬁlters across different layers.
Revised: To address these issues, we present structure-regularized pruning (SRP), which imposes L 2 regularization on the pruned structure to align the locations of pruned ﬁlters across different layers.

Original: It can transfer the expressive power in the unimportant ﬁlters to the remaining part of the network.
Revised: It can transfer the expressive power in the unimportant ﬁlters to the remaining part of the network.

Original: We employ SRP to train both lightweight and large image SR networks, resulting in a lightweight network SRPNL and a very deep one SRPN.

Original: Our SRPN-L and SRPN achieve superior performance over recent state-of-the-art methods quantitatively and qualitatively.

Original: A M ORE D ISCUSSIONS AND I NVESTIGATIONS A. D IFFERENCES WITH DHP

Original: (1) The main idea of DHP (Li et al., 2020) is to employ meta-learning to automatically decide different layer-wise pruning ratios, while SRP simply employs the same pre-deﬁned ratio for different layers.

Original: (2) DHP (Li et al., 2020) learns unimportant ﬁlters via regularization, while our SRP picks them via a pre-deﬁned importance criterion (i.e., L 1 -norm).

Original: (3) DHP (Li et al., 2020) uses Lregularization to induce sparsity, while SRP uses L 2 regularization.

Original: A. P ERFORMANCE D ROP FROM O RIGINAL M ODEL

Original: The performance degradation between the original model and the compression model should also be discussed.

Original: In Tab. 5, we provide models ﬁnetuned for only 1 epoch.

Original: Compared to the L 1 -norm pruning, the performance on Set5 ( × 2) drop of our pruned models is signiﬁcantly smaller.

Original: [Table]

Original: We compare with other widely used channel pruning methods, like ThiNet (Luo et al., 2017) and DCP (Zhuang et al., 2018). It is hard for us to implement ThiNet in our PyTorch framework from its
Revised: We employ SRP to train both lightweight and large image

Original: Caffe code.

Original: We implement DCP (Zhuang et al., 2018) for image SR and show results on Set5 ( × 2) in Tab. 6, where our SRP performs better.

Original: [Table]

Original: A. C OMPARISONS WITH O THER SOTA M ETHODS

Original: We further compare with other recent lightweight and large image SR networks: PAN (Zhao et al., 2020), LatticeNet (Luo et al., 2020), and DRN-L (Guo et al., 2020)

Original: (the large model version).

Original: In

Original: Tab. 7, lightweight models, LatticeNet obtains better than SRP-L with more parameters. Large models, SRPN performs better than DRN-L. [Table] A. M ODEL C OMPLEXITY C OMPARISONS FOR L ARGE M ODELS
Revised: SR networks, resulting in a lightweight network SRPN-Lite and a very deep one SRPN, delivering superior performance over recent SOTA efﬁcient SR methods.

Original: We provide model size, FLOPs, and inference time comparison for large models in Tab. 8.

Original: We can see our SRPN obtains similar model size, FLOPs with RCAN and SAN, but obtains better performance.

Original: When compared with CSNLN, our SRPN performs more efﬁciently

Original: We further discuss the generalization ability of our SRP method. (1) SRP can prune any kind of block structure. (2) We remove channel attention because we want to remove its effect and observe the effect of SRP better.
Revised: Acknowledgments .

Original: (3) We apply our SRP to IMDN (Hui et al., 2019) and CARN (Ahn et al., Tab. 9. [Table] [Table]
Revised: This research is supported by the U.S. Army Research Ofﬁce Award W911NF-17-1-0367.

Original: A. D ISCUSSIONS WITH O THER P RUNING W ORKS

Original: (1) Existing pruning works are mainly for classiﬁcation , while this paper focuses on image SR .

Original: (2) Pruning methods in classiﬁcation can hardly apply to SR networks directly, thus we propose a pruning method speciﬁcally for SR models.

Original: (3) We show comparison with (Hou & Kung, 2020) in

Original: Tab. 10.

Original: (4) We also discuss the differences with (Kim et al., 2020), which is for image compression .

Original: We do not think the wisdom of image compression can equally apply to image SR .

Original: Our results (Tab. 1) suggest the small-magnitude ﬁlters are unimportant , in line with the case in classiﬁcation.

Original: [Table]

Original: +DI-trim

Original: [Table]

Original: [Table]

Original: Hou & Kung,

Original: [Table]

Original: A. V ALIDATION FOR R ANDOM S ELECTION

Original: (1) Random choosing works well in practice because of the proposed regularization scheme.

Original: Empirically, it is justiﬁed by the strong performance in Tabs.

Original: 2 and 3 and Figs.

Original: 4 and 5.

Original: (2) Results with different random seeds are presented in Tab.

Original: We can see that the statistical variation is small, which further shows the robustness of our method.

Original: B M ORE E XPERIMENTS B. C OMPARISONS WITH L IGHTWEIGHT N ETWORKS

Original: Quantitative Results.

Original: We compare our SRPN-L with representative lightweight SR networks: SRCNN (Dong et al., 2014), FSRCNN (Dong et al., 2016), VDSR (Kim et al., 2016a), DRCN (Kim et al., 2016b), CNF (Ren et al., 2017), LapSRN (Lai et al., 2017), DRRN (Tai et al., 2017a), BTSRN (Fan et al., 2017), MemNet (Tai et al., 2017b), SelNet (Choi & Kim, 2017), CARN (Ahn et al., 2018), CARN-M (Ahn et al., 2018), BSRN (Choi et al., 2018),

Original: FALSR-A (Chu et al., 2019a), and

Original: IMDN (Hui et al., 2019).

Original: We show extensive comparisons in Tab. 12.

Original: Visual Results.

Original: We also provide visual results in Fig.

Original: We can see obviously better results from our

Original: SRPN-L than others.

Original: The code and visual results of LatticeNet (Luo et al., 2020) are not available.

Original: The model size and FLOPs are also much larger than our SRPN

Original: -L.

Original: We aim to propose a smaller one than IMDN and LatticeNet in terms for both parameters and FLOPs.

Original: Consequently, we don’t compare with LatticeNet currently.

Original: However, our code still supports to train a lightweight network with similar model size and FLOPs as LatticeNet but larger than IMDN.

Original: B. C OMPARISONS WITH L ARGER N ETWORKS

Original: Quantitative Results.

Original: Most previous lightweight SR networks neglect to extend their models to deeper and compare with larger networks.

Original: As stated in implementation, we extend our SRP method to larger network training and obtain a deeper network SRPN.

Original: In Tab. 13, we compare with large

Original: SR networks: EDSR (Lim et al., 2017), SRMDNF (Zhang et al., 2018a), DBPN (Haris et al., 2018),

Original: RDN (Zhang et al., 2018c), RCAN (Zhang et al., 2018b), NLRN (Liu et al., 2018), RNAN (Zhang

Original: [Table]

Original: et al., 2019), SRFBN (Li et al., 2019), OISR (He et al., 2019), SAN (Dai et al., 2019), HAN (Niu et al., 2020), NSR (Fan et al., 2020), IGNN (Zhou et al., 2020), CSNLN (Mei et al., 2020), and

Original: RFANet (Liu et al., 2020).

Original: We provide performance comparisons in Tab. 13.

Original: Visual Results.

Original: We show visual comparisons in Fig.

Original: For some very challenging cases, most previous SOTA methods perform similarly and suffer from some blurring artifacts.

Original: However, our large model SRPN recovers better results with sharper structures and more faithful details.

Original: [Table]

Original: In Fig. 8, we present more plots of the pruning process of our SRP method on the EDSR baseline network.

Original: There are 16 residual blocks in EDSR baseline.

Original: We evenly select 5 blocks to plot (layer name is texted as title in each sub-plot).

Original: The free Conv layers are plotted in the left and constrained

Original: Conv layers plotted in the right.

Original: There are two observations worth our attention, as follows.

Original: (1) L 1 -norm at iteration 0 .

Original: In free Conv layers, the unimportant (i.e., pruned) ﬁlters are selected based on L 1 -norm sorting.

Original: As a result, at iteration 0, the L 1 -norm of pruned ﬁlters are always smaller than that of kept ﬁlters.

Original: In comparison, for constrained Conv layers, since the unimportant ﬁlters are picked randomly , thus the L 1 -norm of pruned ﬁlters are close (occasionally higher due to randomness) to that of kept ﬁlters at iteration 0.

Original: This observation serves as a “sanity check” to show our algorithm works normally, being consistent with the robustness shown in Tab. 11.

Original: (2) Compensation effect of free/constrained Conv layers .

Original: The compensation effect of kept ﬁlters (i.e., L 1 -norms of these ﬁlters go up automatically without any extra regularization – see also Line 756-760 in the main paper) is observed in all plots presented here, to different degrees.

Original: For free

Original: Conv layers, this trend is clear (the blue solid line always goes up cross different layers); while for constrained Conv layers, the trend is related to the layer location in the network: for low-stage layers (e.g., “model.body.0.body.2”, “model.body.2.body.2”), the blue line only goes up a little; while for high-stage layers (e.g., “model.body.12.body.2” and “model.body.15.body.2”), the blue line goes up signiﬁcantly.

Original: We conjecture, this is because the high-stage layers are closer to the output, thus more critical.

Original: The low-stage layers do not have to compensate for the pruned ﬁlters since the high-stage layers will do this for them.

Original: This implies that, probably we should prune fewer ﬁlters for the highstage layers (for now, the pruning ratio is the same across different layers).

Original: This issue can potentially bring more performance gains but relatively trivial as compared to the pruning methodology itself, so we do not cover it in the main paper and leave it as a worthy future direction.

Original: B. D EMO C ODE

Original: We have attached a demo testing code and pretrained model for our lightweight method SRPN-L with scaling factor × 2.

Original: The network structure will be printed in the log ﬁle after running the demo code.

Original: The demo code is based on EDSR PyTorch code.

Original: All the training/testing code, pretrained models, and training/ablation study log ﬁles would be available after the paper is published.

Original: [Table]

Original: Urban100: img 005 ( × 4)

Original: HR

Original: Bicubic

Original: SRCNN

Original: FSRCNN

Original: VDSR

Original: LapSRN

Original: MemNet

Original: CARN

Original: IMDN

Original: SRPN-L (ours)

Original: Urban100: img 046 ( × 4)

Original: HR

Original: Bicubic

Original: SRCNN

Original: FSRCNN

Original: VDSR

Original: LapSRN

Original: MemNet

Original: CARN

Original: IMDN

Original: SRPN-L (ours)

Original: Urban100: img 053 ( × 4)

Original: HR

Original: Bicubic

Original: SRCNN

Original: FSRCNN

Original: VDSR

Original: LapSRN

Original: MemNet

Original: CARN

Original: IMDN

Original: SRPN-L (ours)

Original: Urban100: img 061 ( × 4)

Original: HR

Original: Bicubic

Original: SRCNN

Original: FSRCNN

Original: VDSR

Original: LapSRN

Original: MemNet

Original: CARN

Original: IMDN

Original: SRPN-L (ours)

Original: Urban100: img 099 ( × 4)

Original: HR

Original: Bicubic

Original: SRCNN

Original: FSRCNN

Original: VDSR

Original: LapSRN

Original: MemNet

Original: CARN

Original: IMDN

Original: SRPN-L (ours)

Original: [Table]

Original: Urban100: img 008 ( × 4)

Original: HR

Original: Bicubic

Original: EDSR

Original: DBPN

Original: RCAN

Original: SRFBN

Original: IGNN

Original: CSNLN

Original: RFANet

Original: SRPN (ours)

Original: Urban100: img 011 ( × 4)

Original: HR

Original: Bicubic

Original: EDSR

Original: DBPN

Original: RCAN

Original: SRFBN

Original: IGNN

Original: CSNLN

Original: RFANet

Original: SRPN (ours)

Original: Urban100: img 060 ( × 4)

Original: HR

Original: Bicubic

Original: EDSR

Original: DBPN

Original: RCAN

Original: SRFBN

Original: IGNN

Original: CSNLN

Original: RFANet

Original: SRPN (ours)

Original: Urban100: img 092 ( × 4)

Original: HR

Original: Bicubic

Original: EDSR

Original: DBPN

Original: RCAN

Original: SRFBN

Original: IGNN

Original: CSNLN

Original: RFANet

Original: SRPN (ours)

Original: Urban100: img 095 ( × 4)

Original: HR

Original: Bicubic

Original: EDSR

Original: DBPN

Original: RCAN

Original: SRFBN

Original: IGNN

Original: CSNLN

Original: RFANet

Original: SRPN (ours)

Original: [Figure]