Original: L EARNING E FFICIENT I MAGE S UPER -R ESOLUTION N ETWORKS VIA S TRUCTURE -R EGULARIZED P RUNING
Revised: L EARNING E FFICIENT I MAGE S UPER -R ESOLUTION N ETWORKS VIA S TRUCTURE -R EGULARIZED P RUNING

Original: A BSTRACT
Revised: A BSTRACT

Original: Several image super-resolution (SR) networks have been proposedof late for efﬁcient SR, achieving promising results.
Revised: Several image super-resolution (SR) networks have been proposed for efﬁcient SR, achieving promising results.

Original: However, they are still not lightweight enough and neglect to be extended to larger networks.
Revised: However, they are still not lightweight enough and neglect to be extended to larger networks.

Original: At the same time, model compression techniques, like neural architecture search and knowledge distillation, typically consume considerable computation resources.
Revised: At the same time, model compression methods (i.e., knowledge distillation and neural architecture search) usually need heavy extra resources.

Original: In contrast, network pruning is a cheap and effective model compression technique.
Revised: Then, we turn to a cheap and effective model compression way: network pruning.

Original: However, it is hard to be applied toSR networks directly because ﬁlter pruning for residual blocks is well-known tricky.
Revised: However, it is not easy to directly apply network pruning to image SR. It is well-known tricky to conduct ﬁlter pruning for residual blocks commonly used in SR networks.

Original: To address the above issues, we propose structure-regularized pruning (SRP), which imposes regularization on the pruned structure to ensure the locations of pruned ﬁlters are aligned across different layers.
Revised: To solve these problems, we propose structure-regularized pruning (SRP), which imposes regularization on the pruned structure to ensure the locations of pruned ﬁlters are aligned across different layers.

Original: Speciﬁcally, for the layers connected by the same residual, we select the ﬁlters of the same indices as unimportant ﬁlters.
Revised: Speciﬁcally, for the layers connected by the same residual, we select the ﬁlters of the same indices as unimportant ﬁlters.

Original: To transfer the expressive power in the unimportant ﬁlters to the rest of the network, we employ L 2 regularization to drive the weights towards zero so that eventually, their absence will cause minimal performance degradation.
Revised: To transfer the expressive power in the unimportant ﬁlters to the rest of the network, we employ L 2 regularization to drive the weights towards zero so that eventually, their absence will cause minimal performance degradation.

Original: We apply SRP to train efﬁcient image SR networks, resulting in a lightweight network SRPN-Lite and a very deep one SRPN.
Revised: We apply SRP to train efﬁcient image SR networks, resulting in a lightweight network SRPN-Lite and a very deep one SRPN.

Original: We conduct extensive comparisons with both lightweight and larger  networks.
Revised: We provide extensive comparisons with both lightweight and larger image SR networks.

Original:  SRPN-Lite and SRPN perform favorably against other recent efﬁcient SR approaches quantitatively and visually.
Revised: Our SRPN-Lite and SRPN perform favorably against other recent works.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Image super-resolution (SR), a classic task in computer vision, aims to recover a high-resolution (HR) image based on its low-resolution (LR) counterpart.
Revised: Given a low-resolution (LR) image, single image super-resolution (SR) aims to reconstruct a highresolution (HR) output.

Original: Essentially,image SR is ill-posed as a many-to-one mapping problem.
Revised: Essentially, as a many-to-one mapping problemimage SR is ill-posed.

Original: To tackle this problem, plenty of deep convolutional neural networks (CNNs) (Dong et al., 2014; 2016; Kim et al., 2016b; Zhang et al., 2018c; 2020; 2021) have been investigated to learn the accurate mapping from LR image to its HR target.
Revised: To tackle this problem, plenty of deep convolutional neural networks (CNNs) (Dong et al., 2014; 2016; Kim et al., 2016b; Zhang et al., 2018c; 2020; 2021) have been investigated to learn the accurate mapping from LR input to the corresponding HR target.

Original: Deep CNN for image SR is pioneered by SRCNN (Dong et al., 2014) and has continuously shown promising SR performance.
Revised: Deep CNN for image SR is ﬁrst investigated in SRCNN (Dong et al., 2014) and has continuously shown promising SR performance.

Original: There are only three convolutional (Conv) layers in SRCNN, constraining its expressivity.
Revised: SRCNN consists of there convolutional (Conv) layers, constraining its expressivity.

Original: Kim et al . increased the network depth in VDSR (Kim et al., 2016a)with residual learning and obtained notable improvements over SRCNN.
Revised: Kim et al . adopted residual learning to increase network depth in VDSR (Kim et al., 2016a) and obtained notable improvements over SRCNN.

Original: Lim et al .
Revised: Lim et al .

Original: (Lim et al., 2017)  built a much deeper network EDSR by using simpliﬁed residual blocks.
Revised: (Lim et al., 2017) simpliﬁed residual blocks and built a much deeper network EDSR.

Original: Zhang et al .
Revised: Zhang et al .

Original: (Zhang et al., 2018b) proposed RCAN, which is one of the deepest networks in SR at present.
Revised: (Zhang et al., 2018b) proposed a even much deeper one RCAN with the residual in residual structure.

Original: Empowered by increased network size, deep SR models like EDSR (Lim et al., 2017) and RCAN (Zhang et al., 2018b) have seen remarkable SR performance.
Revised: Empowered by increased network size, deep SR models like EDSR (Lim et al., 2017) and RCAN (Zhang et al., 2018b) have seen remarkable SR performance.

Original: However, as a cost, the large model size brings about problems such as excessive memory footprint, slow inference speed.
Revised: However, as a cost, the large model size brings about problems such as excessive memory footprint, slow inference speed.

Original: It is thereby infeasible todirectly deploy them on resource-constrained platforms  (Lee et al., 2020).
Revised: It is thereby impractical to deploy them on resource-constrained platforms directly (Lee et al., 2020).

Original: Aiming for efﬁcient SR, more and more works introduce lightweight network architectures (Ahn et al., 2018;
Revised: Aiming for efﬁcient SR, more and more works introduce lightweight network architectures (Ahn et al., 2018;

Original: Luo et al., 2020).
Revised: Luo et al., 2020).

Original: Ahn et al .
Revised: Ahn et al .

Original: proposed cascading residual network (CARN)
Revised: proposed cascading residual network (CARN)

Original: (Ahn et al., 2018).
Revised: (Ahn et al., 2018).

Original: Hui et al .
Revised: Hui et al .

Original: proposed information multi-distillation network (IMDN) (Hui et al., 2019).
Revised: proposed information multi-distillation network (IMDN) (Hui et al., 2019).

Original: Lee et al .
Revised: Lee et al .

Original: introduced knowledge distillation (Hinton et al., 2014) for image SR (Lee et al., 2020).
Revised: introduced knowledge distillation (Hinton et al., 2014) for image SR (Lee et al., 2020).

Original: Besides, neural architecture search (NAS) (Zoph & Le, 2017; Elsken et al., 2019) was also utilized for lightweight SR model (Chu et al., 2019a).
Revised: Besides, neural architecture search (NAS) (Zoph & Le, 2017; Elsken et al., 2019) was also utilized for lightweight SR model (Chu et al., 2019a).

Original: However, there are still several downsides to these networks: (1) There is still an obvious performance gap between those lightweight models and the very deep models; (2) These methods can consume considerable computation resources for training.
Revised: However, there are still several downsides to these networks: (1) There is still an obvious performance gap between those lightweight models and the very deep models; (2) These methods can consume considerable computation resources for training.

Original: For example, (Chu et al., 2019a)require 8 Tesla V100 GPUs to train a single network, and the 
Revised: For example, (Chu et al., 2019a) train a single network with 8 Tesla V100 GPUs, and the training

Original: [Figure]
Revised: [Figure]

Original: training takes about three days; (3) It is hard for most lightweight SR network designs to generalize to more large-scale networks and achieve superior performance at the same time.
Revised: takes about three days; (3) It is hard for most lightweight SR network designs to generalize to more large-scale networks and achieve superior performance at the same time.

Original: In short, moreeffective, resource-friendly,  and generic lightweight SR networks are still in need.
Revised: So, it is needed to explore more resource-friendly, effective, and generic lightweight SR networks.

Original: On the other hand, neural network pruning is well-known as an effective technique to reduce model complexity (Reed, 1993; Sze et al., 2017).
Revised: On the other hand, neural network pruning is well-known as an effective technique to reduce model complexity (Reed, 1993; Sze et al., 2017).

Original: For acceleration, ﬁlter pruning (a.k.a.
Revised: For acceleration, ﬁlter pruning (a.k.a.

Original: structured pruning)
Revised: structured pruning)

Original: (Li et al., 2017) attracts more attention than weight-element pruning (a.k.a.
Revised: (Li et al., 2017) attracts more attention than weight-element pruning (a.k.a.

Original: unstructured pruning)
Revised: unstructured pruning)

Original: (Han et al., 2015; 2016b).
Revised: (Han et al., 2015; 2016b).

Original: Introducing ﬁlter pruning into image SR is a promising solution to achieve a good trade-off between performance and complexity.
Revised: Introducing ﬁlter pruning into image SR is a promising way to achieve a good trade-off between performance and complexity.

Original: However, it is not easy to apply ﬁlter pruning methods to image SR networks directly.
Revised: However, it is not easy to apply ﬁlter pruning techniques to image SR networks directly.

Original: This is mainly because residual connections are well-known difﬁcult to prune in structured pruning (Li et al., 2017).
Revised: This is mainly because residual connections are well-known difﬁcult to prune in structured pruning (Li et al., 2017).

Original: On the other hand, they are extensively used in state-of-the-art (SOTA) image SR methods (e.g., EDSR (Lim et al., 2017) hasresidual blocks; RCAN (Zhang et al., 2018b) even has nested residual blocks).
Revised: On the other hand, they are extensively used in state-of-the-art (SOTA) image SR methods (e.g., EDSR (Lim et al., 2017) hasresidual blocks; RCAN (Zhang et al., 2018b) even has nested residual blocks).

Original: To tackle the above issue, we propose Structure-Regularized Pruning (SRP), which imposes regularization on the pruned structure to ensure the locations of pruned ﬁlters are aligned across different layers.
Revised: To tackle the above issue, we propose Structure-Regularized Pruning (SRP), which imposes regularization on the pruned structure to ensure the locations of pruned ﬁlters are aligned across different layers.

Original: Speciﬁcally, for the layers connected by the same residual, we select the ﬁlters of the same indices as unimportant ﬁlters (i.e., those we will remove ﬁnally).
Revised: Speciﬁcally, for the layers connected by the same residual, we select the ﬁlters of the same indices as unimportant ﬁlters (i.e., those we will remove ﬁnally).

Original: To transfer the expressive power in the unimportant ﬁlters to the remainder of the network, we employ L 2 regularization to drive the weights towards zero gradually so that eventually, their absence will incur negligible performance loss.
Revised: To transfer the expressive power in the unimportant ﬁlters to the remainder of the network, we employ L 2 regularization to drive the weights towards zero gradually so that eventually, their absence will incur negligible performance loss.

Original: To the best of our knowledge, our SRP is one of the leading works (Zhang et al., 2021) to leverage structured pruning for efﬁcient image SR.
Revised: To the best of our knowledge, our SRP is one of the leading works (Zhang et al., 2021) to leverage structured pruning for efﬁcient image SR.

Original: Our main contributions are as follows:
Revised: Our main contributions are as follows:

Original: • We propose a network structure-regularized pruning (SRP) method to learn efﬁcient image SR networks.
Revised: • We propose a network structure-regularized pruning (SRP) method to learn efﬁcient image SR networks.

Original: We try to jointly train image SR models with network pruning simultaneously to achieve high reconstruction performance as well as efﬁciency.
Revised: We try to jointly train image SR models with network pruning simultaneously to achieve high reconstruction performance as well as efﬁciency.

Original: • Our SRP provides a general idea to structurally prune networks, which consists of extensive residual connections.
Revised: • Our SRP provides a general idea to structurally prune networks, which consists of extensive residual connections.

Original: The introduction of regularization as a pruning tool manages to maintain the expressivity of the original network while peeling off the unnecessary redundancy.
Revised: The introduction of regularization as a pruning tool manages to maintain the expressivity of the original network while peeling off the unnecessary redundancy.

Original: • We employ SRP to train efﬁcient image SR networks, resulting in a lightweight network (named SRPN-Lite) and a very deep one (named SRPN).
Revised: • We employ SRP to train efﬁcient image SR networks, resulting in a lightweight network (named SRPN-Lite) and a very deep one (named SRPN).

Original: We achieve superior performance gains on both lightweight and large image SR networks.
Revised: We achieve superior performance gains on both lightweight and large image SR networks.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: Deep Image SR Models.
Revised: Deep Image SR Models.

Original: Dong et al .
Revised: Dong et al .

Original: (Dong et al., 2014) ﬁrstly proposed SRCNN for image SR and achieved superior performance with only 3 convolutional (Conv) layers.
Revised: (Dong et al., 2014) ﬁrstly proposed SRCNN for image SR and achieved superior performance with only 3 convolutional (Conv) layers.

Original: Residual learning was introduced in VDSR (Kim et al., 2016a), reaching 20 Conv layers and a signiﬁcant improvement over SRCNN.
Revised: Residual learning was introduced in VDSR (Kim et al., 2016a), reaching 20 Conv layers and a signiﬁcant improvement over SRCNN.

Original: Tai et al .
Revised: Tai et al .

Original: later introduced memory block in MemNet (Tai et al., 2017b) for deeper network structure.
Revised: later introduced memory block in MemNet (Tai et al., 2017b) for deeper network structure.

Original: Lim et al .
Revised: Lim et al .

Original: (Lim et al., 2017) simpliﬁed the residual block (He et al., 2016) and constructed deeper and wider networks with a large number of parameters.
Revised: (Lim et al., 2017) simpliﬁed the residual block (He et al., 2016) and constructed deeper and wider networks with a large number of parameters.

Original: Zhang et al .
Revised: Zhang et al .

Original: (Zhang et al., 2018b) proposed an even deeper network, residual channel attention network (RCAN), where the attention mechanism was ﬁrstly introduced in image SR.
Revised: (Zhang et al., 2018b) proposed an even deeper one, residual channel attention network (RCAN), where the attention mechanism was ﬁrstly introduced in image SR.

Original: Liu et al .
Revised: Liu et al .

Original: proposed
Revised: proposed

Original: FRANet
Revised: FRANet

Original: (Liu et al., 2020) to make the residual features focus on critical spatial contents.
Revised: (Liu et al., 2020) to make the residual features focus on critical spatial contents.

Original: Later, Zhang et al .
Revised: Later, Zhang et al .

Original: (Zhang et al., 2019) proposed residual non-local attention for image restoration.
Revised: (Zhang et al., 2019) extended to image restoration with residual non-local attention.

Original: Mei et al .
Revised: Mei et al .

Original: proposed CSNLN (Mei et al., 2020) by combining local, in-scale/cross-scale non-local feature correlations, and external statistics.
Revised: proposed CSNLN (Mei et al., 2020) by combining local and non-local feature correlations.

Original: Most of those methods have achieved SOTA results.
Revised: Most of those methods have achieved SOTA results.

Original: However, they suffer from huge model size (i.e., network parameter number) and/or heavy computation operations (i.e., FLOPs).
Revised: However, they suffer from huge model size (i.e., network parameter number) and/or heavy computation operations (i.e., FLOPs).

Original: [Figure]
Revised: [Figure]

Original: ) 𝒋− 𝒋+ 𝒋
Revised: ) 𝒋− 𝒋+ 𝒋

Original: F iF
Revised: F iF

Original: i
Revised: i

Original: F i+ F i+
Revised: F i+ F i+

Original: W i
Revised: W i

Original: W i+  1 33 6  

Revised: W i+  1 33 6 22

Original: [Figure]
Revised: [Figure]

Original: Lightweight Image SR Models.
Revised: Lightweight Image SR Models.

Original: Recent years have been rising interest in investigating lightweight image SR models.
Revised: Recent years have been rising interest in investigating lightweight image SR models.

Original: These approaches try to design lightweight architectures, which mainly take advantage of recursive learning and channel splitting.
Revised: These approaches try to design lightweight architectures, which mainly take advantage of recursive learning and channel splitting.

Original: Kim et al .
Revised: Kim et al .

Original: ﬁrstly decreased parameter number by utilizing recursive learning in DRCN (Kim et al., 2016b).
Revised: ﬁrstly decreased parameter number by utilizing recursive learning in DRCN (Kim et al., 2016b).

Original: Ahn et al .
Revised: Ahn et al .

Original: proposed CARN by designing a cascading mechanism upon a residual network (Ahn et al., 2018).
Revised: proposed CARN by designing a cascading mechanism upon a residual network (Ahn et al., 2018).

Original: Hui et al .
Revised: Hui et al .

Original: proposed a lightweight information multi-distillation network (IMDN) (Hui et al., 2019).
Revised: proposed IMDN by using distillation and fusion modules (Hui et al., 2019).

Original: Luo et al . designed the lattice block with butterﬂy structures (Luo et al., 2020).
Revised: Luo et al . designed the lattice block with butterﬂy structures (Luo et al., 2020).

Original: Recently, neural architecture search was applied for image SR, like FALSR (Chu et al., 2019a).
Revised: Recently, neural architecture search was applied for image SR, like FALSR (Chu et al., 2019a).

Original: Also, model compression techniques have been explored for image SR.
Revised: Also, model compression techniques have been explored for image SR.

Original: He et al .
Revised: He et al .

Original: proposed knowledge distillation based feature-afﬁnity for efﬁcient image SR (He et al., 2020).
Revised: proposed knowledge distillation based feature-afﬁnity for efﬁcient image SR (He et al., 2020).

Original: Lee et al .
Revised: Lee et al .

Original: distilled knowledge from a larger teacher network to a student one (Lee et al., 2020).
Revised: distilled knowledge from a larger teacher network to a student one (Lee et al., 2020).

Original: Those lightweight image SR models have obtained great progress, but we still need to investigate deeper for more efﬁcient image SR models.
Revised: Those lightweight image SR models have obtained great progress, but we still need to investigate deeper for more efﬁcient image SR models.

Original: Neural Network Pruning.
Revised: Neural Network Pruning.

Original: Network pruning aims to eliminate redundant parameters in a neural network without compromising its performance seriously (Reed, 1993; Sze et al., 2017).
Revised: Network pruning aims to eliminate redundant parameters in a neural network without compromising its performance seriously (Reed, 1993; Sze et al., 2017).

Original: The methodology of pruning mainly falls into two groups: ﬁlter pruning (or more generally known as structured pruning) * and weight-element pruning (also referred to as unstructured pruning).
Revised: The methodology of pruning mainly falls into two groups: ﬁlter pruning (or more generally known as structured pruning) * and weight-element pruning (also referred to as unstructured pruning).

Original: The former aims to remove weights by ﬁlters (i.e., 4-d tensors), while the latter removes weights by single elements (i.e., scalars).
Revised: The former aims to remove weights by ﬁlters (i.e., 4-d tensors), while the latter removes weights by single elements (i.e., scalars).

Original: Structured pruning results in regular sparsity after pruning.
Revised: Structured pruning results in regular sparsity after pruning.

Original: It does not demand any special hardware features to achieve considerable practical acceleration.
Revised: It does not demand any special hardware features to achieve considerable practical acceleration.

Original: In contrast, unstructured pruning leads to irregular sparsity.
Revised: In contrast, unstructured pruning leads to irregular sparsity.

Original: Leveraging the irregular sparsity for acceleration typically demands special software supports, while past works have shown the practical speedup is very limited (Wen et al., 2016), unless using customized hardware platforms (Han et al., 2016a).
Revised: Leveraging the irregular sparsity for acceleration typically demands special software supports, while past works have shown the practical speedup is very limited (Wen et al., 2016), unless using customized hardware platforms (Han et al., 2016a).

Original: In this paper, we tackle ﬁlter pruning instead of weight-element pruning for effortless acceleration.
Revised: In this paper, we tackle ﬁlter pruning instead of weight-element pruning for effortless acceleration.

Original: The major efforts in pruning (mainly in image classiﬁcation) have been focusing on proposing a more sound pruning criterion to select unimportant weights (Reed, 1993;
Revised: The major efforts in pruning (mainly in image classiﬁcation) have been focusing on proposing a more sound pruning criterion to select unimportant weights (Reed, 1993;

Original: Sze et al., 2017).
Revised: Sze et al., 2017).

Original: Criteria based on weight magnitude (Han et al., 2015; 2016b; Li et al., 2017) are the most prevailing ones, which we will also employ to develop our method in this paper.
Revised: Criteria based on weight magnitude (Han et al., 2015; 2016b; Li et al., 2017) are the most prevailing ones, which we will also employ to develop our method in this paper.

Original:  M ETHODOLOGY
Revised:  M ETHODOLOGY

Original: We ﬁrst present an overview of the problem setting about deep CNN for image SR.
Revised: We ﬁrst show the overview of the problem setting about deep CNN for image SR.

Original: It is also observed that excessive redundancy exists in the SR deep CNNs.
Revised: It is also observed that excessive redundancy exists in the SR deep CNNs.

Original: Then we move on to proposing our structureregularized pruning (SRP) method attempting to achieve more efﬁcient SR networks.
Revised: Then we move on to proposing our structureregularized pruning (SRP) method attempting to achieve more efﬁcient SR networks.

Original: 3. D EEP CNN FOR I MAGE SR
Revised: 3. D EEP CNN FOR I MAGE SR

Original: In image super-resolution (SR), a high-resolution (a.k.a.
Revised: In image super-resolution (SR), a high-resolution (a.k.a.

Original: super-resolved) image I SR is reconstructed from its low-resolution (LR) counterpart I LR .
Revised: super-resolved) image I SR is reconstructed from its low-resolution (LR) counterpart I LR .

Original: This image SR process can be described as
Revised: This image SR process can be described as

Original: [Equation]
Revised: [Equation]

Original: where F SR ( · ) is the image SR network, parameterized by Θ .
Revised: where F SR ( · ) is the image SR network, parameterized by Θ .

Original: Meanwhile, the LR image I LR from its HR counterpart can be formulated as a degradation process
Revised: Meanwhile, the LR image I LR from its HR counterpart can be formulated as a degradation process

Original: [Equation]
Revised: [Equation]

Original: where F ↓ s ( · ) downscales the original ground truth I HR by a scale factor s .
Revised: where F ↓ s ( · ) downscales the original ground truth I HR by a scale factor s .

Original: This downscaling process typically introduces additional noise, blurring, compression, and/or other unknown artifacts.
Revised: This downscaling process typically introduces additional noise, blurring, compression, and/or other unknown artifacts.

Original: Therein, the high-frequency information will be lost more or less.
Revised: Therein, the high-frequency information will be lost more or less.

Original: The job of image SR models is to recover the high-frequency information as much as possible.
Revised: The job of image SR models is to recover the high-frequency information as much as possible.

Original: We aim to present a structural sparsity inducing SR network learning method, which can deliver more efﬁcient SR networks with fewer parameters and FLOPs.
Revised: We aim to present a structural sparsity inducing SR network learning method, which can deliver more efﬁcient SR networks with fewer parameters and FLOPs.

Original: 3. S TRUCTURE -R EGULARIZED P RUNING (SRP)
Revised: 3. S TRUCTURE -R EGULARIZED P RUNING (SRP)

Original: Pruned Index Constraint .
Revised: Pruned Index Constraint .

Original: Pruning ﬁlters in residual networks is well-known non-trivial as the Add operators in residual blocks require the pruned ﬁlter indices across different residual blocks must be aligned.
Revised: Pruning ﬁlters in residual networks is well-known non-trivial as the Add operators in residual blocks require the pruned ﬁlter indices across different residual blocks must be aligned.

Original: A ﬁgurative illustration of ﬁlter pruning within a residual block is shown in Fig. 2(b).
Revised: A ﬁgurative illustration of ﬁlter pruning within a residual block is shown in Fig. 2(b).

Original: A typical residual block (e.g., in EDSR (Lim et al., 2017), RCAN (Zhang et al., 2018b)) consists of two convolutional layers.
Revised: A typical residual block (e.g., in EDSR (Lim et al., 2017), RCAN (Zhang et al., 2018b)) consists of two convolutional layers.

Original: According to the mutual connection relationship, the convolutional layers can be categorized into two groups.
Revised: According to the mutual connection relationship, the convolutional layers can be categorized into two groups.

Original: One group is made up with the layers that can be pruned without any constraint , dubbed free Conv layers in this work; the other comprises Conv layers whose ﬁlters must be pruned at the same indices , dubbed constrained Conv layers .
Revised: One group is made up with the layers that can be pruned without any constraint , dubbed free Conv layers in this work; the other comprises Conv layers whose ﬁlters must be pruned at the same indices , dubbed constrained Conv layers .

Original: Concretely, the layer W ( i ) in
Revised: Concretely, the layer W ( i ) in

Original: Fig. 2(b) is a free Conv layer, while the layer W ( i +1) is a constrained one.
Revised: Fig. 2(b) is a free Conv layer, while the layer W ( i +1) is a constrained one.

Original: Owing to the pruned index constraint problem, many ﬁlter pruning methods in image classiﬁcation simply do not prune the last Conv layer in residual blocks (Li et al., 2017; Wang et al., 2021), which can still deliver considerable speedup of practical interest.
Revised: Owing to the pruned index constraint problem, many ﬁlter pruning methods in image classiﬁcation simply do not prune the last Conv layer in residual blocks (Li et al., 2017; Wang et al., 2021), which can still deliver considerable speedup of practical interest.

Original: Nevertheless, this doing-nothing solution can barely translate to the image SR networks if we target a considerable speedup.
Revised: Nevertheless, this doing-nothing solution can barely translate to the image SR networks if we target a considerable speedup.

Original: The root cause is that image SR networks typically employ many more residual blocks, and each block usually has only two Conv layers.
Revised: The root cause is that image SR networks typically employ many more residual blocks, and each block usually has only two Conv layers.

Original: In some top-performing SR networks (e.g., RCAN), there are even nested residual blocks.
Revised: In some top-performing SR networks (e.g., RCAN), there are even nested residual blocks.

Original: Taking EDSR for a concrete example, it has 32 residual blocks and each block has two Conv layers.
Revised: Taking EDSR for a concrete example, it has 32 residual blocks and each block has two Conv layers.

Original: If the 2nd Conv layer in a residual block is spared from pruning, half of the Conv layers will not be pruned.
Revised: If the 2nd Conv layer in a residual block is spared from pruning, half of the Conv layers will not be pruned.

Original: In other words, at best , we can only achieve 2 × theoretical acceleration measured by FLOPs reduction.
Revised: In other words, at best , we can only achieve 2 × theoretical acceleration measured by FLOPs reduction.

Original: Given this issue, it is imperative to prune all the Conv layers in residual blocks, thus calling for an approach to align the pruned indices of all constrained Conv layers.
Revised: Given this issue, it is imperative to prune all the Conv layers in residual blocks, thus calling for an approach to align the pruned indices of all constrained Conv layers.

Original: Regularization then arises as a promising solution considering it has been widely used before to impose priors on the sparsity structure in classiﬁcation (Reed, 1993; Wen et al., 2016; Wang et al., 2021).
Revised: Regularization then arises as a promising solution considering it has been widely used before to impose priors on the sparsity structure in classiﬁcation (Reed, 1993; Wen et al., 2016; Wang et al., 2021).

Original: Our SRP method is based on regularization to form desired hardware-friendly sparsity structure.
Revised: Our SRP method is based on regularization to form desired hardware-friendly sparsity structure.

Original: Primarily, there are three questions to answer in SRP: (1) which pruning criterion is used to choose unimportant ﬁlters; (2) which kind of regularization is chosen to induce sparsity; (3) how to schedule the pruning process to make the algorithm robust and easy to use.
Revised: Primarily, there are three questions to answer in SRP: (1) which pruning criterion is used to choose unimportant ﬁlters; (2) which kind of regularization is chosen to induce sparsity; (3) how to schedule the pruning process to make the algorithm robust and easy to use.

Original: (1) Pruning Criterion .
Revised: (1) Pruning Criterion .

Original: When it comes to formulated pruning criteria, they mainly fall into three groups: magnitude based (Han et al., 2015; Li et al., 2017), 1st-order gradient based (Molchanov et al., 2017; 2019), and 2nd-order gradient based (LeCun et al., 1990; Hassibi & Stork, 1993; Wang et al., 2019a).
Revised: When it comes to formulated pruning criteria, they mainly fall into three groups: magnitude based (Han et al., 2015; Li et al., 2017), 1st-order gradient based (Molchanov et al., 2017; 2019), and 2nd-order gradient based (LeCun et al., 1990; Hassibi & Stork, 1993; Wang et al., 2019a).

Original: The 1st-order and 2nd-order gradient-based criteria are based on Taylor approximation of the increased loss when a weight is removed, pioneered by (LeCun et al., 1990).
Revised: The 1st-order and 2nd-order gradient-based criteria are based on Taylor approximation of the increased loss when a weight is removed, pioneered by (LeCun et al., 1990).

Original: They are more accurate than magnitude-based one right after pruning.
Revised: They are more accurate than magnitude-based one right after pruning.

Original: However, pruning is typically followed by a ﬁnetuning process (Reed, 1993) to regain performance.
Revised: However, pruning is typically followed by a ﬁnetuning process (Reed, 1993) to regain performance.

Original: After ﬁnetuning, the advantage of the 1st-order and 2nd-order gradient-based criteria usually becomes marginal.
Revised: After ﬁnetuning, the advantage of the 1st-order and 2nd-order gradient-based criteria usually becomes marginal.

Original: Meanwhile, they are typically more costly than the magnitude-based criteria.
Revised: Meanwhile, they are typically more costly than the magnitude-based criteria.

Original: Taking the cost and ﬂexibility into account, we simply employ magnitude-based pruning criteria.
Revised: Taking the cost and ﬂexibility into account, we simply employ magnitude-based pruning criteria.

Original: Speciﬁcally, given a pretrained SR network, for all the free Conv layers, we sort the ﬁlters in ascending order by their L 1 -norms and select the ﬁlers with the least norms as unimportant ﬁlters , by a predeﬁned pruning ratio r ∈ (0 , 1) .
Revised: Speciﬁcally, given a pretrained SR network, for all the free Conv layers, we sort the ﬁlters in ascending order by their L 1 -norms and select the ﬁlers with the least norms as unimportant ﬁlters , by a predeﬁned pruning ratio r ∈ (0 , 1) .

Original: For the constrained Conv layers, the problem becomes a little bit more complex.
Revised: For the constrained Conv layers, the problem becomes a little bit more complex.

Original: Given the pruned index constraint, ideally, we can only prune the ﬁlters in the intersection set (denoted by S ( net ) ) for the whole network,
Revised: Given the pruned index constraint, ideally, we can only prune the ﬁlters in the intersection set (denoted by S ( net ) ) for the whole network,

Original: [Equation]
Revised: [Equation]

Original: where S ( l ) represents the set of unimportant ﬁlters selected by certain pruning criterion in the l -th
Revised: where S ( l ) represents the set of unimportant ﬁlters selected by certain pruning criterion in the l -th

Original: Conv layer; C denotes the set of constrained Conv layers.
Revised: Conv layer; C denotes the set of constrained Conv layers.

Original: For top-performed image SR networks, |C| ( |·| denotes cardinality of a set) is at the order of magnitude of hundreds (e.g., |C| = 200 for RCAN).
Revised: For top-performed image SR networks, |C| ( |·| denotes cardinality of a set) is at the order of magnitude of hundreds (e.g., |C| = 200 for RCAN).

Original: Meanwhile, given a pretrained model, its pruned ﬁlter indices by L 1 -norm are usually random across different layers.
Revised: Meanwhile, given a pretrained model, its pruned ﬁlter indices by L 1 -norm are usually random across different layers.

Original: As a consequence, |S ( net ) | will be rather small (in practice, on RCAN, we even observe |S ( net ) |
Revised: As a consequence, |S ( net ) | will be rather small (in practice, on RCAN, we even observe |S ( net ) |

Original: = 0 when we try to prune 32 ﬁlters out of 64).
Revised: = 0 when we try to prune 32 ﬁlters out of 64).

Original: This issue brings much trouble to user control when we want to prune a layer to a pre-deﬁned number of ﬁlters.
Revised: This issue brings much trouble to user control when we want to prune a layer to a pre-deﬁned number of ﬁlters.

Original: To resolve this problem, we choose to randomly select a set of ﬁlters as unimportant and ﬁxed during the whole pruning process.
Revised: To resolve this problem, we choose to randomly select a set of ﬁlters as unimportant and ﬁxed during the whole pruning process.

Original: In practice, we ﬁnd this simple scheme works pretty well.
Revised: In practice, we ﬁnd this simple scheme works pretty well.

Original: The reason enabling us to do so is that regularization for pruning is a gradual process.
Revised: The reason enabling us to do so is that regularization for pruning is a gradual process.

Original: It allows ample time for the network to recover from the introduced penalty term (this will be empirically veriﬁed in our experiments, see Fig. 3).
Revised: It allows ample time for the network to recover from the introduced penalty term (this will be empirically veriﬁed in our experiments, see Fig. 3).

Original: Apart from aligning cross-layer sparsity structure, this is the second reason we choose regularization for pruning in our algorithm.
Revised: Apart from aligning cross-layer sparsity structure, this is the second reason we choose regularization for pruning in our algorithm.

Original: (2) Regularization Form .
Revised: (2) Regularization Form .

Original: Although L 1 regularization is well-known to induce sparsity in machine learning, it is hard to tune the coefﬁcient to realize a desired trade-off between sparsity and performance.
Revised: Although L 1 regularization is well-known to induce sparsity in machine learning, it is hard to tune the coefﬁcient to realize a desired trade-off between sparsity and performance.

Original: Instead, L 2 regularization is thereby adopted in our method given its more tamable control over the sparsiﬁcation process – note the gradient of L 2 regularization is proportion to the weight magnitude while the gradient of L 1 regularization is not.
Revised: Instead, L 2 regularization is thereby adopted in our method given its more tamable control over the sparsiﬁcation process – note the gradient of L 2 regularization is proportion to the weight magnitude while the gradient of L 1 regularization is not.

Original: Speciﬁcally, given the loss function L of a neural network Θ , the total error function E with adaptive L 2 regularization term is formulated as
Revised: Speciﬁcally, given the loss function L of a neural network Θ , the total error function E with adaptive L 2 regularization term is formulated as

Original: [Equation]
Revised: [Equation]

Original: where D stands for the training dataset; W ( l ) j refers to the j -th ﬁlter in l -th Conv layer; α ( l ) j stands for the L 2 regularization co-efﬁcient for that ﬁlter.
Revised: where D stands for the training dataset; W ( l ) j refers to the j -th ﬁlter in l -th Conv layer; α ( l ) j stands for the L 2 regularization co-efﬁcient for that ﬁlter.

Original: If a ﬁlter j falls into the set of unimportant ﬁlters
Revised: If a ﬁlter j falls into the set of unimportant ﬁlters

Original: S ( l ) , α ( l ) j > 0 ; otherwise, α ( l ) j = 0 since we do not restrict the learning of important ﬁlters.
Revised: S ( l ) , α ( l ) j > 0 ; otherwise, α ( l ) j = 0 since we do not restrict the learning of important ﬁlters.

Original: (3) Regularization Schedule .
Revised: (3) Regularization Schedule .

Original: As mentioned above, for the constrained Conv layers, we select the unimportant ﬁlters randomly.
Revised: As mentioned above, for the constrained Conv layers, we select the unimportant ﬁlters randomly.

Original: To mitigate the side effect of this sub-optimal choice, the network should be provided with abundant training iterations to adapt so that it can transfer its expressive power to the remaining part of the network.
Revised: To mitigate the side effect of this sub-optimal choice, the network should be provided with abundant training iterations to adapt so that it can transfer its expressive power to the remaining part of the network.

Original: To achieve this goal, we choose to gradually increase the penalty strength, drawing inspirations from (Wang et al., 2019b; 2021),
Revised: To achieve this goal, we choose to gradually increase the penalty strength, drawing inspirations from (Wang et al., 2019b; 2021),

Original: [Equation]
Revised: [Equation]

Original: where ∆ (a pre-deﬁned constant) is a small amount that the penalty strength is increased each time.
Revised: where ∆ (a pre-deﬁned constant) is a small amount that the penalty strength is increased each time.

Original: This design allows us to raise the penalty strength to a large amount eventually (like 0.5 in our experiments; in contrast, note the normal L 2 regularization co-efﬁcient is at the scale of 10 − 4 ).
Revised: This design allows us to raise the penalty strength to a large amount eventually (like 0.5 in our experiments; in contrast, note the normal L 2 regularization co-efﬁcient is at the scale of 10 − 4 ).

Original: As a result, the unimportant weights are driven rather close to zero, and removing them will hardly hurt the performance.
Revised: As a result, the unimportant weights are driven rather close to zero, and removing them will hardly hurt the performance.

Original: Besides, α is updated every T iterations during training for stability.
Revised: Besides, α is updated every T iterations during training for stability.

Original: To indicate when to terminate the pruning process, a ceiling limit τ is introduced for the regularization co-efﬁcient α .
Revised: To indicate when to terminate the pruning process, a ceiling limit τ is introduced for the regularization co-efﬁcient α .

Original: The pruning is over when all the α ’s for unimportant ﬁlters arise to τ .
Revised: The pruning is over when all the α ’s for unimportant ﬁlters arise to τ .

Original: 3. L EARNING E FFICIENT I MAGE SR M ODELS VIA SRP
Revised: 3. L EARNING E FFICIENT I MAGE SR M ODELS VIA SRP

Original: SRP can be applied to top-performing SR algorithms in a plug-and-play manner – simply replace the loss function L in Eq.
Revised: SRP can be applied to top-performing SR algorithms in a plug-and-play manner – simply replace the loss function L in Eq.

Original: (4) with the loss objective function of an SR approach of interest.
Revised: (4) with the loss objective function of an SR approach of interest.

Original: The proposed L 2 regularization can be implemented on any deep learning framework very easily.
Revised: The proposed L 2 regularization can be implemented on any deep learning framework very easily.

Original: Upon ﬁnishing pruning, we take away the unimportant ﬁlters (not zeroing out them only, but literally removing them from the model), which will give us a compact SR model.
Revised: Upon ﬁnishing pruning, we take away the unimportant ﬁlters (not zeroing out them only, but literally removing them from the model), which will give us a compact SR model.

Original: Finally, the compact model will be ﬁnetuned to regain performance following the common practice in pruning (Reed, 1993).
Revised: Finally, the compact model will be ﬁnetuned to regain performance following the common practice in pruning (Reed, 1993).

Original: 3. I MPLEMENTATION D ETAILS
Revised: 3. I MPLEMENTATION D ETAILS

Original: We explain how to apply SRP to both lightweight and deep image SR networks to demonstrate the method can work with a wide range of SR networks under different computation budgets.
Revised: We explain how to apply SRP to both lightweight and deep image SR networks to demonstrate the method can work with a wide range of SR networks under different computation budgets.

Original: Lightweight Networks.
Revised: Lightweight Networks.

Original: First, we adapt the EDSR baseline (Lim et al., 2017) with 16 residual blocks by removing the ﬁnal convolutional layer to save parameters.
Revised: First, we adapt the EDSR baseline (Lim et al., 2017) with 16 residual blocks by removing the ﬁnal convolutional layer to save parameters.

Original: The reconstruction upscaling is realized by the pixel-shufﬂe layer (Shi et al., 2016) following common practice.
Revised: The reconstruction upscaling is realized by the pixel-shufﬂe layer (Shi et al., 2016) following common practice.

Original: We set the channel number in the revised EDSR baseline as 256 and then prune it to 45.
Revised: We set the channel number in the revised EDSR baseline as 256 and then prune it to 45.

Original: For × 2 scale, we reduce the number of parameters from 19.5M to 609K and name the compressed model as SRPN-Lite.
Revised: For × 2 scale, we reduce the number of parameters from 19.5M to 609K and name the compressed model as SRPN-Lite.

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: ); interestingly, the important ones arise spontaneously.
Revised: ); interestingly, the important ones arise spontaneously.

Original: Deep Networks.
Revised: Deep Networks.

Original: To apply SRP to the very deep network RCAN (Zhang et al., 2018b), a representative top-performing deep SR network with over 400 Conv layers, we revise RCAN by removing all the channel attention modules (Zhang et al., 2018b).
Revised: To apply SRP to the very deep network RCAN (Zhang et al., 2018b), a representative top-performing deep SR network with over 400 Conv layers, we revise RCAN by removing all the channel attention modules (Zhang et al., 2018b).

Original: The channel number in the revised RCAN is chosen as 96 and then pruned to 64.
Revised: The channel number in the revised RCAN is chosen as 96 and then pruned to 64.

Original: For × 2 scale, we reduce the number of parameters from 34.5M to 15.3M and dub the compressed model as SRPN.
Revised: For × 2 scale, we reduce the number of parameters from 34.5M to 15.3M and dub the compressed model as SRPN.

Original:  E XPERIMENTAL R ESULTS 4. S ETTINGS
Revised:  E XPERIMENTAL R ESULTS 4. S ETTINGS

Original: Data and Evaluation.
Revised: Data and Evaluation.

Original: We use DIV2K dataset (Timofte et al., 2017) and Flickr2K Lim et al.
Revised: We use DIV2K dataset (Timofte et al., 2017) and Flickr2K Lim et al.

Original: (2017) as training data, following most recent works (Timofte et al., 2017; Lim et al., 2017;
Revised: (2017) as training data, following most recent works (Timofte et al., 2017; Lim et al., 2017;

Original: Zhang et al., 2018a; Haris et al., 2018).
Revised: Zhang et al., 2018a; Haris et al., 2018).

Original: For testing, we use ﬁve standard benchmark datasets: Set5 (Bevilacqua et al., 2012), Set14 (Zeyde et al., 2010), B100 (Martin et al., 2001), Urban100 (Huang et al., 2015), and Manga109 (Matsui et al., 2017).
Revised: For testing, we use ﬁve standard benchmark datasets: Set5 (Bevilacqua et al., 2012), Set14 (Zeyde et al., 2010), B100 (Martin et al., 2001), Urban100 (Huang et al., 2015), and Manga109 (Matsui et al., 2017).

Original: The SR results are evaluated with PSNR and SSIM (Wang et al., 2004) on the Y channel in YCbCr space.
Revised: The SR results are evaluated with PSNR and SSIM (Wang et al., 2004) on the Y channel in YCbCr space.

Original: Training Settings.
Revised: Training Settings.

Original: Following (Zhang et al., 2018b), data augmentation is used in training – training images are randomly rotated by 90 ◦ , 180 ◦ , 270 ◦ and ﬂipped horizontally.
Revised: Following (Zhang et al., 2018b), data augmentation is used in training – training images are randomly rotated by 90 ◦ , 180 ◦ , 270 ◦ and ﬂipped horizontally.

Original: Image patches (patch size 48 × 48) are cropped out to form each training batch.
Revised: Image patches (patch size 48 × 48) are cropped out to form each training batch.

Original: Adam optimizer (Kingma & Ba, 2014) is adopted for training with β 1 =0.9, β 2 =0.999, and (cid:15) =
Revised: Adam optimizer (Kingma & Ba, 2014) is adopted for training with β 1 =0.9, β 2 =0.999, and (cid:15) =

Original: 10 − 8 .
Revised: 10 − 8 .

Original: Initial learning rate is set to 10 − 4 and then decayed by factor 0 .
Revised: Initial learning rate is set to 10 − 4 and then decayed by factor 0 .

Original: 5 every 2 × 10 5 iterations.
Revised: 5 every 2 × 10 5 iterations.

Original: We use PyTorch (Paszke et al., 2017) to implement our models with a Tesla V100 GPU † .
Revised: We use PyTorch (Paszke et al., 2017) to implement our models with a Tesla V100 GPU † .

Original: 4. A BLATION S TUDY
Revised: 4. A BLATION S TUDY

Original: The EDSR baseline with 16 residual blocks (Lim et al., 2017) ‡ is used as the backbone for ablation study given its broad use in the community.
Revised: The EDSR baseline with 16 residual blocks (Lim et al., 2017) ‡ is used as the backbone for ablation study given its broad use in the community.

Original: Comparison with Baseline Pruning Methods .
Revised: Comparison with Baseline Pruning Methods .

Original: To start with, we show that our SRP is more effective than the available baseline compact image SR network learning methods.
Revised: To start with, we show that our SRP is more effective than the available baseline compact image SR network learning methods.

Original: Two baseline methods
Revised: Two baseline methods

Original: [Table] [Table]
Revised: [Table] [Table]

Original: are compared to: training from scratch and the original L 1 -norm pruning (Li et al., 2017).
Revised: are compared to: training from scratch and the original L 1 -norm pruning (Li et al., 2017).

Original: The results are presented in Tab. 1.
Revised: The results are presented in Tab. 1.

Original: As we can see, our pruned network delivers the best PSNR consistently across different pruning ratios.
Revised: As we can see, our pruned network delivers the best PSNR consistently across different pruning ratios.

Original: Notably, under a greater pruning ratio, the advantage of our method over scratch training or L 1 -norm pruning is more evident in general, suggesting that our method is more valuable in aggressive pruning cases.
Revised: Notably, under a greater pruning ratio, the advantage of our method over scratch training or L 1 -norm pruning is more evident in general, suggesting that our method is more valuable in aggressive pruning cases.

Original: Of particular note is that, our method SRP also adopts the
Revised: Of particular note is that, our method SRP also adopts the

Original: L 1 -norm as the scoring criterion to select unimportant ﬁlters, same as (Li et al., 2017).
Revised: L 1 -norm as the scoring criterion to select unimportant ﬁlters, same as (Li et al., 2017).

Original: Nevertheless, our method delivers signiﬁcantly better results than theirs.
Revised: Nevertheless, our method delivers signiﬁcantly better results than theirs.

Original: The primary reason is that they do not impose any regularization on the pruned structure; the remaining feature maps are thus mismatched in residual blocks after pruning.
Revised: The primary reason is that they do not impose any regularization on the pruned structure; the remaining feature maps are thus mismatched in residual blocks after pruning.

Original: In contrast, our method SRP is not bothered by this issue, showing the effectiveness of our proposed structure regularization.
Revised: In contrast, our method SRP is not bothered by this issue, showing the effectiveness of our proposed structure regularization.

Original: Visualization of Pruning Process .
Revised: Visualization of Pruning Process .

Original: In Fig.
Revised: In Fig.

Original: 3, we visualize the pruning process by plotting the mean
Revised: 3, we visualize the pruning process by plotting the mean

Original: L 1 -norm of ﬁlters in two layers of EDSR baseline during SRP training.
Revised: L 1 -norm of ﬁlters in two layers of EDSR baseline during SRP training.

Original: The ﬁlters are split into two groups, kept ﬁlters and pruned ﬁlters.
Revised: The ﬁlters are split into two groups, kept ﬁlters and pruned ﬁlters.

Original: As is shown in the ﬁgure, the mean L 1 -norm of the pruned ﬁlters goes down gradually because the penalty grows stronger and stronger, driving them towards zero.
Revised: As is shown in the ﬁgure, the mean L 1 -norm of the pruned ﬁlters goes down gradually because the penalty grows stronger and stronger, driving them towards zero.

Original: Interestingly, note the L 1 -norms of the kept ﬁlters arise themselves.
Revised: Interestingly, note the L 1 -norms of the kept ﬁlters arise themselves.

Original: Recall that there is no explicit regularization to promote them to grow.
Revised: Recall that there is no explicit regularization to promote them to grow.

Original: In other words, the network learns to recover by itself , akin to the compensation effect found in human brain (Duffau et al., 2003).
Revised: In other words, the network learns to recover by itself , akin to the compensation effect found in human brain (Duffau et al., 2003).

Original: [Table] [Table]
Revised: [Table] [Table]

Original: 4. C OMPARISONS WITH L IGHTWEIGHT N ETWORKS
Revised: 4. C OMPARISONS WITH L IGHTWEIGHT N ETWORKS

Original: We compare our SRPN-Lite with lightweight SR networks: SRCNN (Dong et al., 2014), FSRCNN (Dong et al., 2016), VDSR (Kim et al., 2016a), DRCN (Kim et al., 2016b), LapSRN (Lai et al., 2017), DRRN (Tai et al., 2017a), MemNet (Tai et al., 2017b), CARN (Ahn et al., 2018) and
Revised: We compare our SRPN-Lite with lightweight SR networks: SRCNN (Dong et al., 2014), FSRCNN (Dong et al., 2016), VDSR (Kim et al., 2016a), DRCN (Kim et al., 2016b), LapSRN (Lai et al., 2017), DRRN (Tai et al., 2017a), MemNet (Tai et al., 2017b), CARN (Ahn et al., 2018) and

Original: IMDN (Hui et al., 2019).
Revised: IMDN (Hui et al., 2019).

Original: The compared results are produced from their released code.
Revised: The compared results are produced from their released code.

Original: Performance Comparisons.
Revised: Performance Comparisons.

Original: Table 2 shows PSNR/SSIM comparisons for × 2, × 3, and × 4 SR.
Revised: Table 2 shows PSNR/SSIM comparisons for × 2, × 3, and × 4 SR.

Original: When compared with all previous methods, our SRPN-Lite performs the best on all the datasets under all scaling factors.
Revised: When compared with all previous methods, our SRPN-Lite performs the best on all the datasets under all scaling factors.

Original: Unlike most comparison methods, which achieve efﬁciency through careful network designs, our work starts with the existing EDSR baseline (Lim et al., 2017) and prunes it to a much smaller network, showing the effectiveness of our proposed SRP.
Revised: Unlike most comparison methods, which achieve efﬁciency through careful network designs, our work starts with the existing EDSR baseline (Lim et al., 2017) and prunes it to a much smaller network, showing the effectiveness of our proposed SRP.

Original: Model Size and Mult-Adds.
Revised: Model Size and Mult-Adds.

Original: Our SRPN-Lite has the fewest parameter number in comparison to recent efﬁcient SR works such as MemNet, CARN, and IMDN.
Revised: Our SRPN-Lite has the fewest parameter number in comparison to recent efﬁcient SR works such as MemNet, CARN, and IMDN.

Original: The comparison in terms of MultAdds (measured when the output size is set to 3 × 1,280 × 720) is also presented.
Revised: The comparison in terms of MultAdds (measured when the output size is set to 3 × 1,280 × 720) is also presented.

Original: As seen, our SRPNLite costs fewer Mult-Adds than most comparison methods.
Revised: As seen, our SRPNLite costs fewer Mult-Adds than most comparison methods.

Original: These results demonstrate the merits of
Revised: These results demonstrate the merits of

Original: SRP against other counterparts in striking a better network performance-complexity trade-off.
Revised: SRP against other counterparts in striking a better network performance-complexity trade-off.

Original: [Table]
Revised: [Table]

Original: Visual Comparisons.
Revised: Visual Comparisons.

Original: The visual comparisons at × 4 scale are shown in Fig.
Revised: The visual comparisons at × 4 scale are shown in Fig.

Original: It is easy to see that most of the comparison algorithms can barely recover the lost texture details in the correct direction; moreover, they suffer from blurring artifacts.
Revised: It is easy to see that most of the comparison algorithms can barely recover the lost texture details in the correct direction; moreover, they suffer from blurring artifacts.

Original: In stark contrast, our SRPN-Lite effectively alleviates the blurring artifacts and recovers sharper structural ﬁne textures in the right way, justifying the performance superiority of our SRP approach over other methods.
Revised: In stark contrast, our SRPN-Lite effectively alleviates the blurring artifacts and recovers sharper structural ﬁne textures in the right way, justifying the performance superiority of our SRP approach over other methods.

Original: 4. C OMPARISONS WITH L ARGER N ETWORKS
Revised: 4. C OMPARISONS WITH L ARGER N ETWORKS

Original: Most previous lightweight SR networks neglect to extend their models to deeper and compare with larger networks.
Revised: Most previous lightweight SR networks neglect to extend their models to deeper and compare with larger networks.

Original: As stated in Sec. 3.4, we extend our SRP method to larger network training and obtain a deeper network SRPN.
Revised: As stated in Sec. 3.4, we extend our SRP method to larger network training and obtain a deeper network SRPN.

Original: In Tab. 3, we compare with large SR networks: EDSR (Lim et al., 2017), DBPN (Haris et al., 2018), RCAN (Zhang et al., 2018b), SRFBN (Li et al., 2019), SAN (Dai et al., 2019), HAN (Niu et al., 2020), IGNN (Zhou et al., 2020), CSNLN (Mei et al., 2020), and
Revised: In Tab. 3, we compare with large SR networks: EDSR (Lim et al., 2017), DBPN (Haris et al., 2018), RCAN (Zhang et al., 2018b), SRFBN (Li et al., 2019), SAN (Dai et al., 2019), HAN (Niu et al., 2020), IGNN (Zhou et al., 2020), CSNLN (Mei et al., 2020) [Table] and

Original: RFANet (Liu et al., 2020).
Revised: RFANet (Liu et al., 2020).

Revised: The compared results are produced from their released code.

Original: Performance Comparisons.
Revised: Performance Comparisons.

Original: In Tab. 3, we observe that our SRPN achieves the best PSNR/SSIM values across different test sets and scales, except for SSIM value on Set5 ( × 2).
Revised: In Tab. 3, we observe that our SRPN achieves the best PSNR/SSIM values across different test sets and scales, except for SSIM value on Set5 ( × 2).

Original: Our SRPN utilizes slightly fewer parameters (15.3M vs. 15.4M) with the same network depth as RCAN (Zhang et al., 2018b).
Revised: Our SRPN utilizes slightly fewer parameters (15.3M vs. 15.4M) with the same network depth as RCAN (Zhang et al., 2018b).

Original: Notably, our SRPN removes the channel attention module in RCAN, but still achieves better results.
Revised: Notably, our SRPN removes the channel attention module in RCAN, but still achieves better results.

Original: This is mainly because our SRPN is initially trained from a wider and stronger network (with 96 channels).
Revised: This is mainly because our SRPN is initially trained from a wider and stronger network (with 96 channels).

Original: We train the much larger network and then prune it to the target size.
Revised: We train the much larger network and then prune it to the target size.

Original: Visual Comparisons.
Revised: Visual Comparisons.

Original: We further show visual comparisons with larger networks (e.g., CSNLN,
Revised: We further show visual comparisons with larger networks (e.g., CSNLN,

Original: RFANet, and IGNN) in Fig.
Revised: RFANet, and IGNN) in Fig.

Original: 5, we can also observe similar situations as in Fig.
Revised: 5, we can also observe similar situations as in Fig.

Original: For example, in img 011, other methods suffer from heavy blurring artifacts.
Revised: For example, in img 011, other methods suffer from heavy blurring artifacts.

Original: In other cases, they may not recover clear structures in img 044 or even wrong details in img 062.
Revised: In other cases, they may not recover clear structures in img 044 or even wrong details in img 062.

Original: Our SRPN achieves more visually pleasing results with more structural details.
Revised: Our SRPN achieves more visually pleasing results with more structural details.

Original: Those comparisons demonstrate that our SRP method can be applied to train larger networks and obtain comparable or even better performance.
Revised: Those comparisons demonstrate that our SRP method can be applied to train larger networks and obtain comparable or even better performance.

Original: 4. M ORE C OMPARISON WITH M ODEL C OMPRESSION T ECHNIQUES
Revised: 4. M ORE C OMPARISON WITH M ODEL C OMPRESSION T ECHNIQUES

Original: We further compare our SRP to other representative efﬁcient image SR approaches via model compression.
Revised: We further compare our SRP to other representative efﬁcient image SR approaches via model compression.

Original: Concretely, neural architecture search based methods (Chu et al., 2019b;a) and knowledge distillation (KD)
Revised: Concretely, neural architecture search based methods (Chu et al., 2019b;a) and knowledge distillation (KD)

Original: based methods (Lee et al., 2020) are compared to.
Revised: based methods (Lee et al., 2020) are compared to.

Original: Quantitative results at × 2 scale are presented in Tab. 4, where our SRPN-Lite delivers better PSNR results across different datasets with fewer parameters and Mult-Adds.
Revised: Quantitative results at × 2 scale are presented in Tab. 4, where our SRPN-Lite delivers better PSNR results across different datasets with fewer parameters and Mult-Adds.

Original: With our SRP pruning method, there is no need to search massive network architectures or pretraining a teacher network, which usually consumes considerable computation resources.
Revised: With our SRP pruning method, there is no need to search massive network architectures or pretraining a teacher network, which usually consumes considerable computation resources.

Original: These comparisons show that our SRP, as a network pruning method, has as much potential (if not more) as other model compression techniques for efﬁcient image SR.
Revised: These comparisons show that our SRP, as a network pruning method, has as much potential (if not more) as other model compression techniques for efﬁcient image SR.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: Existing efﬁcient image SR methods have achieved promising results with moderate model parameters and Mult-Adds.
Revised: Existing efﬁcient image SR methods have achieved promising results with moderate model parameters and Mult-Adds.

Original: However, they neglect to extend their approaches to larger networks, which usually perform much better with more parameters and/or operations with residual learning.
Revised: However, they neglect to extend their approaches to larger networks, which usually perform much better with more parameters and/or operations with residual learning.

Original: Meanwhile, although model compression approaches, like neural architecture search and knowledge distillation, have also been exploited for efﬁcient SR networks, they usually demand excessive computation resources.
Revised: Meanwhile, although model compression approaches, like neural architecture search and knowledge distillation, have also been exploited for efﬁcient SR networks, they usually demand excessive computation resources.

Original: Network pruning, as another promising model compression method, is hard to train lightweight SR networks directly.
Revised: Network pruning, as another promising model compression method, is hard to train lightweight SR networks directly.

Original: To address these issues, we present structure-regularized pruning (SRP), which imposes L 2 regularization on the pruned structure to align the locations of pruned ﬁlters across different layers.
Revised: To address these issues, we present structure-regularized pruning (SRP), which imposes L 2 regularization on the pruned structure to align the locations of pruned ﬁlters across different layers.

Original: It can transfer the expressive power in the unimportant ﬁlters to the remaining part of the network.
Revised: It can transfer the expressive power in the unimportant ﬁlters to the remaining part of the network.

Original: We employ SRP to train both lightweight and large image
Revised: We employ SRP to train both lightweight and large image

Original: SR networks, resulting in a lightweight network SRPN-Lite and a very deep one SRPN, delivering superior performance over recent SOTA efﬁcient SR methods.
Revised: SR networks, resulting in a lightweight network SRPN-Lite and a very deep one SRPN, delivering superior performance over recent SOTA efﬁcient SR methods.

Original: Acknowledgments .
Revised: Acknowledgments .

Original: This research is supported by the U.S. Army Research Ofﬁce Award W911NF-17-1-0367.
Revised: This research is supported by the U.S. Army Research Ofﬁce Award W911NF-17-1-0367.