On the Sparsity of Image Super-resolution Network
Abstract
The over-parameterization of neural networks has been widely considered for a long time.
This allows us to find sub-networks that can improve the parameter efficiency of neural networks from over-parameterized networks.
In our study, we used EDSR (25) as the backbone network to explore the parameter efficiency in super-resolution(SR) networks in the form of sparsity.
Specifically, we search for sparse sub-networks at the two granularity of weight and kernel through various methods and analyze the relationship between the structure and performance of the sub-networks.
In summary, our work: (1) On weight granularity, we observe the “Lottery Ticket Hypothesis” (10) from a new perspective in the regression task of SR; (2) On convolution kernel granularity, we apply several methods to explore the influence of different sparse sub-networks on network performance and found that based on certain rules, the performance of different sub-networks rarely depends on their structures; (3) We propose a very convenient width-sparsity method on convolution kernel granularity, which can improve the parameter utilization efficiency of most SR networks.
Introduction
Super-resolution (SR) networks aim at predicting high-resolution (HR) images from low-resolution (LR) observations.
SR networks often contain a large number of parameters. Obtaining better performance with fewer parameters is an important research topic. Network sparsification and network pruning are considered to be promising methods. The success of the “Lottery ticket hypothesis” (LTH) (10) seems to point in a surprising direction.
LTH uses a simple iterative pruning method to find a sparse sub-network with better performance than the densely connected network. In this work, we try to obtain a more efficient SR network substructure by building a sparse SR network. However, finding “lottery tickets” for SR networks is difficult.
In other words, there is a positive correlation between the parameters of the SR networks and the performance. We first conduct our research at the weight level sparsity – we try to find a sparse subnetwork that is comparable to the original network when trained in isolation.
This is also called the “winning lottery ticket” (10) for SR networks.
Our experimental results show that a sparse connection will lead to performance degradation when an SR network is well-trained.
While the “winning ticket phenomenon” only exists in the networks that are not well-trained, and can not generate comparable performance to the original network.
In other words, we can not find a sparse sub-network from a dense-large SR network without experiencing a performance drop as Lottery Ticket Hypothesis described. [Figure]
lines”, “dash-dotted lines”, and “dashed lines” denotes the experiment with the iteration number of 1,000K, 300K, and 100K, respectively.
[Figure] Secondly, we use four methods to sparse the network at the convolution kernel granularity, which can be abstracted as pruning off a part of the connections between channels.
Our experiments show that all these methods of obtaining sub-networks, whether they are reasonable or not, have obtained similar performance under the same width and sparsity (the difference in PSNR does not exceed 0.03dB) unless transmission of information is blocked in some channels.
Though we can’t find a sparse sub-network comparable to the fully-connected network, we implement a very convenient width-sparsity trade-off method that can optimize the parameter efficiency of SR networks.
According to our experimental results at the kernel level, we can conveniently define a large-sparse network structure in the initialization phase.
In this way, we can improve the network’s performance (PSNR improved 0.04dB-0.08dB on Div2k_val) without changing the FLOPS or reducing parameters without changing the network’s performance.
Such a method only needs to decorate the convolution layer without changing the network infrastructure or training strategy.
Moreover, the network structure can be defined at the initialisation stage, saving computational expenses in the pruning process.
In summary, our work: (1) Apply several methods to the EDSR network, and we could not find a sparse sub-network that can perform comparably to the original network like the lottery ticket hypothesis; (2) Draws an important observation that the performance of sparse SR networks with the same width and sparsity are surprisingly similar at the kernel level unless the transmission of information between channels is blocked; (3) We propose a very convenient method that can be deployed on most existing super-resolution networks to improve parameter efficiency.
Weight Level Sparsity
In this part of the experiment, we use three methods based on the lottery hypothesis to obtain subnetworks, and have a new understanding of the lottery ticket hypothesis from the perspective of the regression task like super-resolution.
2.1 Obtain sub-network
Weight is the smallest unit in a neural network; therefore, sparsity at the weight level can minimize the performance drop of networks.
The LTH (10) is one of the most influential methods in recent years.
The LTH proposes that in classification tasks, we are able to find sparse sub-networks from a randomly-initialized dense network that this sub-network can be trained from scratch in isolation and reach test accuracy comparable to the original network.
[-Complete connected   5 6 
 [Figure]-]Specifically, in LTH, we first initialize a classification network randomly with parameters θ 0 .
And then, we train the network for T iterations and arrive at θ T .
The third, we use a mask m ∈ { 0 , 1 } | θ | to prune p % of the existing parameters in θ T .
At last, we reset the remaining parameters to their initialized value θ 0 , apply m to θ 0 and convert parameters to m ⊙ θ 0 .
Then, the algorithm will repeat the above steps in the way of iterative pruning for several rounds.
Through this pruning method, a sparse sub-network comparable to the fully-connected network can be found in the "winning ticket" classification task.
Besides, LMC-LTH (11) proposes to rewind parameters to a certain time after a small amount of training θ t , instead of the initialization stage θ 0 .
Liu et al. (27) found that replacing then parameters rewind operation in LTH with a fine-tune process for another T -epoch can obtain a better sub-network compared with LTH.
We chose these three representative methods to obtain sub-networks and specific experimental settings will be described in detail in Appendix.
2.2 Results
Our experimental results are shown in Figure 1.
Figures (a), (b), and (c) show the results of LTH,
LMC-LTH, and LTH-PWC respectively, where the solid lines represent the experimental results under different learning rates with 1000K iteration.
Although these results differ slightly in numerical value, these curves have similar trends.
We can see that in the three groups of experiments with 1,000K iterations, except for the experiment where the learning rate is 1 × 10 − 3 , the network performance decreases with the decrease of sparsity.
The first experimental phenomenon we draw is that, different from the classification task, even if we set the learning rate ( 2 × 10 − 5 ) to a value much smaller than the normal case ( 2 × 10 − 4 ), when the network is sufficiently trained, we can not find a “winning ticket” in SR network (“winning ticket” usually appears in the classification task when the learning rate is relatively small).
On the contrary, when the learning rate is set too large ( 1 × 10 − 3 ), a sub-network with better performance than the original network appears.
But we also noticed that there was a significant performance gap between the network at 1 × 10 − 3 learning rate and the appropriate learning rate when fully connected.
In figures (a) and (b), dash-dotted lines and dotted lines show the experimental results with the iteration of 300K and 100k, respectively.
The results of these two methods still have a similar phenomenon.
It can be noticed that when the number of iterations is 300K, the “winning ticket” appears in the network with a learning rate of 2 × 10 − 5 .
And when the number of iterations is further reduced and set to 100k, there are “winning tickets” under all learning rates.
We note that though the reasons for winning tickets’ appearance are different from those before ( 1 × 10 − 3 learning rate), the same phenomenon is that the network has not reached its own capacity of expressiveness.
Specifically, there will be a gap of more than 0.2dB in PSNR.
We want to emphasize that super-resolution is essentially a complex regression task, therefore, there will be no obvious overfitting phenomenon like the classification task in the training process.
This makes the model always in the convergence stage during the SR network training process.
In other words, the SR network will continue to improve the performance in the later stage of training even with a small learning rate.
While in the classification task with a low learning rate, the network may converge to the local optimal solution.
To verify our idea, we compare the gradient distribution of the network with a small learning rate under different sparsity.
It can be seen from the Figure 2a that the sparser the network, the smaller the gradient.
In conclusion, we tend to think that the sparse network changes the solution space of the network by constraining the weights and gradients, the change of gradient makes the network jump out of the local optimal (usually appears under the small learning rate).
According to the experimental results above, we can draw two novel conclusions about Lottery Ticket
Hypothesis in the super-resolution task: (1) Different from classification tasks, in a sufficiently trained
SR network, even if the learning rate is set much smaller than the normal case, the sparsity of the network will lead to performance degradation, and the higher the sparsity degree, the greater the [Figure] [Figure] [Figure]
[Figure]
[Figure]
[Figure]
Clustering
[Figure]
A v e r a g e
P a t h
L e n g t h
[Figure]
o f n e u r o n s 12.5% Sparsity Randomly
[Figure]
Relational Graph      90 100 Number of connections
{+N u m b e r o f n e u r o n s 50% Sparsity Randomly Connection Relational Graph 0.1 0.2 0.3 0.4 0.
+}
0.6 0.7 Clustering Coefficient 1.
A v e r a g e
P a t h
L e n g t h
[Figure]
[Figure]
performance degradation.
(2) When the SR network does not reach its expression ability through training, a fake "winning lottery" will appear in the process of network pruning.
Kernel level sparsity
In this section, we conducted experiments at the kernel level sparsity.
Although we have obtained similar experimental results at weight level (sparsity will lead to the decline of network performance), we are also surprised to find that in an image super-resolution network, the performance of the network almost only depends on the width and sparsity of the network rather than the structures.
3.1 Obtain sub-network
We describe the obtaining of sparse sub-networks under the following unified framework.
Consider an SR network f
( I LR ; Ω , Θ) parameterized by convolution weights for L layers Ω = { ω 1 , . . .
, ω L } and all the other parameters Θ with an input LR image I LR .
Then, a sub-network is represented as
[Equation]
where m 1 , . .
, m L represent L masks for every layers, ⊙ indicates the masking operation with broadcasting.
Suppose a weight ω
l is a tensor of size C in × C out × K × K , where C in and C out denote the number of input and output channels, and K is the kernel size.
Then m l is a binary mask of size C in × C out , where each element indicates whether the corresponding kernel is masked out.
In this way, the masks obtained through different methods indicate different sparse sub-networks.
The sparsity of the obtained sub-network can be approximated by (cid:80) C in
[Equation]
For the global
(a)            (b)            (c)
[Figure]
[Table]
2” denotes relational graphs with high Average Path Length.
“ERSS-L” denotes layer-wise pruning, and ERSS-G denotes global pruning. unified operations, we omit the subscript l without loss of generality.
Next, we will introduce the mask acquisition methods involved in this part.
Grouped and randomly-connected convolutions.
These two are the most straightforward and simple methods.
We show examples for these two kinds of sub-networks in Figure 2b.
For a grouped convolution layer with G groups, we assume that the input has N channels.
Then this operation is to first divide N channels into G parts.
Each group corresponds to NG channels, which are independently connected within each group.
After each group convolution is calculated, the output is concatenated as the output of this layer.
For a grouped convolution layer with G groups, only 1 G of all elements in the mask are set to 1 , thus the sparsity is equal to 1 G .
If grouped convolution is the most regular way to obtain the mask, then randomly-connected convolution is the other extreme.
Given a pre-defined sparsity q , then each element in the mask m i,j follows a Bernoulli distribution P ( m i,j = 1)
=
q .
This is the simplest way to achieve a certain sparsity in a convolution layer.
Network pruning.
Generally, this kind of method starts with training a large, over-parameterized network, and then proposes to remove redundant parameters without significantly compromising accuracy.
Mao et al. (30) have taken the sum of the absolute values of all weights in a 2-D convolution kernel as a unit, and then prune part of kernels with the lowest-magnitude.
In our study, we will use this simplest way to do iterative pruning for 6 rounds (corresponding to six sparsity degrees).
For this method, we will apply global pruning (pruning the the lowest-magnitude kernels collectively across all convolutional layers) and layer-wise pruning (pruning each layer separately at the same rate).
Networks guided by relational graphs.
You et al. (41) have described a novel approach to represent the complex and diverse possible sparse connections between layers in a neural network.
The relational graph depicts the information interaction between the input channels and the output channels, and a graph G =
( V , C ) is used to represent this interaction, where V is the channel set and C contains connection between channels.
The graph measures of G can characterize important properties for the corresponding architecture.
You et al. (41) have presented that average path length and clustering coefficient are two important measures.
Through the WS-flex graph generator (39), we can sample graphs with different average path lengths and clustering coefficients as much as possible.
Figure 3 shows the distribution of generated graphs and some network connection examples.
It is worth noting that the architectures obtained through random connection only account for a small part of all possible architectures, and we mark them by the gray area in Figure 3.
With the above correspondence between the relational graphs and the network architectures, we can discover various other possible architectures that cannot be simply obtained by the other methods.
3.2 Results
Table 1 show the experimental results of different methods.
To our surprise, the substructure generated by the relational graph can achieve a result comparable to the pruning method.
And these two methods perform better than group convolution and random connection, and the more sparse the network, the greater the impact of its connection structure on the performance of the network.
In order to further study the relationship between the SR network’s structures and performance, we selected six sparsity degrees (12.5%, 18%, 25%, 35%, 50%, 70%), and uniformly sampled from Figure 3 according to these six sparsity degrees.
Figure 4a shows our experimental results.
In this figure, the PSNR is indicated by the color depth.
It can be found that even if our sampling under the same sparsity covers a large range, the performance between the substructures generated by the relational graph is almost equivalent.
This shows to a certain extent that for an SR network, the connection structure of the network at the kernel level has little impact on the model performance.
The difference between the performance of randomly (group) connection and relational graph connection is that the substructure sampled from the graph is a "connected structure".
Figureshows three kinds of graph structure mapping.
It can be seen that although the three structures have the same number of connections between the four channels, only the mapping of structure (a) is a connected graph.
Such a structure ensures that each node in the network can directly or indirectly complete information transmission (Receive or send) with any other node in the network.
In other words, the connected graph not only ensures smooth information transmission but also ensures global information transmission.
Figure 5 shows the node connection distribution of random connection and relational graph connection under two sparsity degrees.
It can be seen that the relationship graph suppresses the number of nodes with fewer connections at low sparsity through the principle of "connected graph".
When the sparsity increases, this repression is weakened, and the performance gap between the two connection methods is also reduced.
In summary, in this part of the study, we make a very interesting discovery: When the relational graph structure of an SR network can be mapped to a "connected graph", its performance depends almost entirely on its channel number and sparsity, and rarely on its structure.
At the same time, such a substructure has an expression capacity comparable to that of the pruning method.
Width-sparsity trade off
Zhu et al. (49) have proposed that on weight level, pruned models (large-sparse) can outperform their smaller, but dense (small-dense) counterparts with identical FLOPS.
This is a simple method to improve the performance of most existing models in the classification task, but a pruning process is still needed to obtain a superior sub-network.
Moreover, although we all know that sparsity on the weight level can better maintain network performance than on the kernel level, our experimental results show that fine-grained sparsity on large SR networks will lead the model difficult to converge.
At the same time, the model is prone to collapse on networks with an attention mechanism (more details in appendix).
Based on these phenomena above and our experimental results on the kernel level, we hope to obtain a large sparse network more conveniently on the kernel level.
According to our conclusion obtained on kernel-level: when the relational graph of the network structure can be mapped to a connected graph, we can define an approximate optimal sub-network in the initialization phase, which means we can save the computational cost of the pruning process.
Meanwhile, sparsity on the kernel level means that compared with the weight level, we can use a coarser granularity to sparse the network, which is helpful for the convergence of the network
Combined with the experimental results on two levels, we propose the following methods to improve the parameter utilization efficiency of the network: (1) Expand the channels of the existing SR network (1.5-2.5 times the original); (2) Sparse the network according to the principle of “a connected graph”; (3) Keep the training settings of the original model unchanged and conduct training.
We apply our method to EDSR (25), PAN (47), IMDN (16) and Lattice-net (29), Experimental results are shown in Appendix.
We are pleasantly surprised to find that our method can work on a small network with only 270k parameters such as PAN.
We divide the sparse network into performance and efficient modes.
The efficient mode can reduce flops by 20% - 40% while maintaining the original performance of the network, performance mode can improve the PSNR on the Div2k_val set by 0.04dB-0.08dB while maintaining the FLOPS of the original network.
A Appendix A.1 Related Work A.1.1 Image Super-Resolution
Since Dong (7) introduce the first SR network, plenty of deep learning based methods have been proposed for SR, deeper (8; 18) and wider networks (25), different network strategies (19; 35; 12; 13), residual or dense connections (21; 44; 38), attention mechanism (42; 6; 47; 4), non-local operations (43; 31), transformers (24; 5).
In order to make the SR network more efficient and meet the needs of algorithm miniaturization, designing a lightweight SR model is also the focus of both the research community and industry.
Early works directly achieved the effect of adjusting the computational complexity by adjusting the scale (depth and width) of the network (8).
Other works include introducing less complex operations to improve the efficiency of the network (33; 2; 47), adding complex operations selectively at important locations (4), employing network architecture search (9) technology to find efficient architectures (34).
Zhang et al. (45; 46) lighter the specific network by pruning on filter-level.
More related to this work, CARN, (2) BSRN (23) introduces group convolution into SR networks to reduce the computational cost.
Wang (36) adopts dynamic unstructured pruning to find efficient sparse sub-networks of specific networks.
To the best of our knowledge, no other work analyses the utilization efficiency of parameters from the perspective of sparse structure and found a promotion method that can be generalized on most networks.
A.1.2 Sparse Sub-network
Obtaining sparse sub-networks for efficient inference follows a long line of works.
Network compression/sparseness can be generally categorized into unstructured and structured methods.
Unstructured methods (15; 14) directly pop unimportant weights, but the obtained sparse network can only show computational efficiency when using specialized libraries or hardware.
On the contrary, the structured methods trims the entire channel or filter of little importance, so the actual acceleration can be easily achieved without the need for a special accelerator.
Kernel-level pruning is a pruning method between structured pruning and unstructured pruning.
This kind of pruning method prunes the convolution kernel of 2D dimension as a unit and has attracted more attention in recent years (26; 22; 48).
For unstructured weight pruning, since the lottery ticket hypothesis (10) was proposed, a lot of follow-up work (11; 41; 3) for classification networks has been devoted to finding a sub-network that performs as well as the fully-connected network.
We naturally raise a question: can SR networks play lottery tickets?
In other words, can we find a sparse sub-network from a dense large SR network without experiencing any performance drop?
In addition, in order to obtain more sparse sub-network on different levels, we also conducted sparse experiments on the kernel-level.
We note that there are works that try to apply network pruning or network compression technologies to SR networks.
However, these methods usually require modification of the network (40; 17) and lose generality.
A.2 Implementations of the main body A.2.1 Weight Level
We use EDSR(25) as the backbone large network.
EDSR only contains residual blocks, no special modules or layers are used, which makes our conclusions without loss of generality.
For training, we train all the networks using DIV2K training dataset (1).
Our training program is implemented using the PyTorch framework (32) and is based on the wonderful BasicSR (37) toolbox.
We use the
Adam (20) optimization method with β 1 = 0 .
9 , β 2 = 0 .
For learning rate, We select four fixed learning rates ( 1 × 10 − 3 , 1 × 10 − 4 , 5 × 10 − 5 , 2 × 10 − 5 ) from a large range and an initial learning rate 2 × 10 − 4 with cosine decay function and warm restarts strategy (28).
For pruning, we follow
LTH’s settings for deep neural networks, so we do not prune the first and last layers of the model and conduct a global pruning in all three methods.
For LTH-PWC (27), we choose to fine-tune for another 1,000K iterations after 1,000K iterations’ training.
More detailed experimental settings are shown in Table 2. [Table]
A.2.2 Kernel level
In this part, we also use EDSR (25) as the backbone large network.
For grouped convolution, we use the official built-in implementation of PyTorch.
For relational graph based convolutions, we follow the official implementation and integrate it into the SR networks.
The randomly-connected convolutions are implemented with similar way as it can be viewed as a special case of relational graph based convolutions.
For EDSR, we use the same settings as weight-level, iterations are set to 1000K.
The only difference is that we only use the Cosine learning rate in weight-level part.
For pruning part, we do not prune the first layer and the last layer of the network and we will apply 500K iterations for fine-tune with the same training strategy.
A.3 Width-sparsity on weight level
Figure 6a and Figure 6b show the results of finding trade-off on EDSR and pan with extended channels through “Lottery ticket hypothesis" (LTH).
Because we want to find a convenient way to improve the performance of the model, we have not changed any training strategy during the pruning process.
Specifically, all the experiments in the figure are conducted at the iteration number of 1000K, an initial learning rate 2 × 10 − 4 with cosine decay function and warm restarts strategy.
We use the
Adam optimization method with β 1 = 0 .
9 , β 2 = 0 .
First, in Figure 6a, the performance of the network decreases with the increase of sparsity.
At the same time, under the low sparsity, when the number of training iterations is 1,000K, the performance is not as good as the result of our sparsity on the kernel granularity.
In Figure 6b, the same phenomenon as EDSR appears.
At the same time, when the network sparsity of PAN (47) is low, it is difficult for us to train the model to the end.
The problem of model collapse will occur between 300K and 500K iterations.
Therefore, in these sub-networks, we selected the
PSNR maximum value that appeared in the training process.
As the degree of sparsity increases, we can complete the training process.
But the model effect is still not as good as the sparse model at the kernel level under the same number of channels and sparsity.
[Figure] [Figure] In the experiments of LTH (10) , the higher the degree of network sparsity, the higher the number of iterations required to train the network.
Therefore, thinning out on too fine-grained granularity will slow the convergence speed of the model.
Because of this, sparsity at the kernel level will be more advantageous under the same number of iterations.
A.4 Toy models
We use 64 channel two-layer convolution and four-layer convolution to replace the 15 residual blocks in EDSR (25) .
We selected four fixed learning rates (0.001, 0.0001, 0.00005, 0.00002),
In order to sufficiently train the network, we still set the number of training iterations to 1000K.
Meanwhile, we followed the experimental setup in LTH, so we set the p of each pruning to 10% and apply a layer-wise pruning (prune each layer separately at the same rate).
Although we have not thinned out the two toy models to less than 10% sparsity, we can still observe the experimental phenomenon from Figure 7a and Figure 7b.
That is, different from LTH in classification task, we could not find a sparse network that can achieve the expression ability of its fully connected network ( In the LTH experiment, compared with deep neural networks, the “winning ticket" of a toy model is easier to find ).
A.5 Quantitative results
Figure 8 and Table 3 shows the experimental results of our width-sparsity trade-off method.
We are pleasantly surprised to find that our method can work not only on the midsize model Lattice with a parameters of 777k, but also on small network with only 270k parameters such as PAN.
We divide the sparse network into performance and efficient modes (denoted "P" and "E" in Table 3).
Efficient mode can reduce flops by 20% - 40% while maintaining the original performance of the network, performance mode can improve the PSNR on the Div2k_val set by 0.04dB-0.8dB while maintaining the FLOPS of the original network.
A.6 Qualitative results
Figure 9 shows the visualization effect of our proposed method.
It can be seen that the proposed method improves the processing of dense texture in the network.
[Figure]
[Figure]
channel sparsity
Params(K)
BSD
Urban
DIV2K_val psnr ssim psnr ssim psnr ssim
[Table]
[Table] [Table] [Table] [Table]
[Table] [Table]
[Table] [Table] [Table]
[Table]
PAN
[Table] [Table]
[Table] [Table]
O
[Table]
[Table] [Table] [Table]
[Table] [Table]
[Figure]
HR
Bicubic
EDSR
IMDN
LatticeNet
EDSR-P
IMDN-P
LatticeNet-P
HR