GAN S C AN P LAY L OTTERY T ICKETS T OO
Anonymous authors
Paper under double-blind review
A BSTRACT
Deep generative adversarial networks (GANs) have gained growing popularity in numerous scenarios, while usually suffer from high parameter complexities for resource-constrained real-world applications.
However, the compression of GANs has less been explored.
A few works show that heuristically applying compression techniques normally leads to unsatisfactory results, due to the notorious training instability of GANs.
In parallel, the lottery ticket hypothesis shows prevailing success on discriminative models, in locating sparse matching subnetworks capable of training in isolation to full model performance.
In this work, we for the ﬁrst time study the existence of such trainable matching subnetworks in deep GANs.
For a range of GANs, we certainly ﬁnd matching subnetworks at 67% - 74% sparsity.
We observe that with or without pruning discriminator has a minor effect on the existence and quality of matching subnetworks, while the initialization weights used in the discriminator plays a signiﬁcant role.
We then show the powerful transferability of these subnetworks to unseen tasks.
Furthermore, extensive experimental results demonstrate that our found subnetworks substantially outperform previous state-of-the-art GAN compression approaches in both image generation (e.g.
SNGAN) and image-to-image translation GANs (e.g. CycleGAN).
I NTRODUCTION
Generative adversarial networks (GANs) have been successfully applied to many ﬁelds like image translation (Jing et al., 2019; Isola et al., 2017; Liu & Tuzel, 2016; Shrivastava et al., 2017; Zhu et al., 2017) and image generation (Miyato et al., 2018;
Radford et al., 2016; Gulrajani et al., 2017;
Arjovsky et al., 2017).
However, they are often heavily parameterized and often require intensive calculation at the training and inference phase.
Network compressing techniques (LeCun et al., 1990; Wang et al., 2019; 2020b; Li et al., 2020) can be of help at inference by reducing the number of parameters or usage of memory; nonetheless, they can not save computational burden at no cost.
Although they strive to maintain the performance after compressing the model, a non-negligible drop in generative capacity is usually observed.
A question is raised:
Is there any way to compress a GAN model while preserving or even improving its performance?
The lottery ticket hypothesis (LTH) (Frankle & Carbin, 2019) provides positive answers with matching subnetworks (Chen et al., 2020).
It states that there exist matching subnetworks in dense models that can be trained to reach a comparable test accuracy to the full model within similar training iterations.
The hypothesis has successfully shown its success in various ﬁelds (Yu et al., 2020; Renda et al., 2020; Chen et al., 2020), and its property has been studied widely (Malach et al., 2020; Pensia et al., 2020; Elesedy et al., 2020).
However, it is never introduced to GANs and therefore the presence of matching subnetworks in generative adversarial networks still remains mysterious.
To address this gap in the literature, we investigate the lottery ticket hypothesis in GANs.
One most critical challenge of extending LTH in GANs emerges: how to deal with the discriminator during compressing the generator, including (i) whether prunes the discriminator simultaneously and (ii) what initialization should be adopted by discriminators during the re-training?
Previous
GAN compression methods (Shu et al., 2019; Wang et al., 2019; Li et al., 2020; Wang et al., 2020b) prune the generator model only since they aim at reducing parameters in the inference stage.
The effect of pruning the discriminator has never been studied by these works, which is unnecessary for them but possibly essential in ﬁnding matching subnetworks.
It is because that ﬁnding matching subnetworks involves re-training the whole GAN network, in which an imbalance in generative and discriminative power could result in degraded training results.
For the same reason, the disequilibrium between initialization used in generators and discriminators incurs severe training instability and unsatisfactory results.
Another attractive property of LTH is the powerful transferability of located matching subnetworks.
Although it has been well studied in discriminative models (Mehta, 2019; Morcos et al., 2019; Chen et al., 2020), an in-depth understanding of transfer learning in GAN tickets is still missing.
In this work, we not only show whether the sparse matching subnetworks in GANs can transfer across multiple datasets but also study what initialization beneﬁts more to the transferability.
To convert parameter efﬁciency of LTH into the advantage of computational saving, we also utilize channel pruning (He et al., 2017) to ﬁnd the structural matching subnetworks of GANs which enjoys the bonus of accelerated training and inference.
Our contributions can be summarized in the following four aspects:
• Using unstructured magnitude pruning, we identify matching subnetworks at 74% sparsity in SNGAN (Miyato et al., 2018) and 67% in CycleGAN (Zhu et al., 2017).
The matching subnetworks in GANs exist no matter whether pruning discriminators, while the initialization weights used in the discriminator are crucial.
• We show that the matching subnetworks found by iterative magnitude pruning outperform subnetworks extracted by randomly pruning and random initialization, in terms of the extreme sparsity and performance.
To fully exploit the trained discriminator, we using the dense discriminator as a distillation source and further improve the quality of winning tickets.
• We demonstrate that the found subnetworks in GANs transfer well across diverse generative tasks.
• The matching subnetworks found by channel pruning surpass previous state-of-the-art GAN compression methods (i.e. GAN Slimming (Wang et al., 2020b)) in both efﬁciency and performance.
R ELATED W ORK
GAN Compression Generative adversarial networks (GANs) have succeeded in computer vision ﬁelds, for example, image generation and translation.
One signiﬁcant drawback of the generative models is the high computational cost of the models’ complex structure.
A wide range of neural network compression techniques has been applied to generative models to address this problem.
There are several categories of compression techniques, including pruning (removing some parameters), quantization (reducing the bit width), and distillation.
Shu et al.
(2019) proposed a channel pruning method for CycleGAN by using a co-evolution algorithm.
Wang et al.
(2019) proposed a quantization method for GANs based on the EM algorithm.
Li et al.
(2020) used a distillation method to transfer knowledge of the dense to the compressed model.
Recently Wang et al.
(2020b) proposed a GAN compression framework, GAN slimming, that integrated the above three mainstream compression techniques into a uniﬁed form.
Previous works on GAN pruning usually aim at ﬁnding a sparse structure of the trained generator model for faster inference speed, while we are focusing on ﬁnding trainable structures of GANs following the lottery ticket hypothesis.
Moreover, in existing
GAN compression methods, only the generator is pruned, which could undermine the performance of re-training since the left-out discriminator may have a stronger computational ability than the pruned generator and therefore cause a degraded result due to the imparity of these two models.
The Lottery Ticket Hypothesis The lottery ticket hypothesis (LTH) (Frankle & Carbin, 2019) claims the existence of sparse, separate trainable sub-networks in a dense network.
These subnetworks are capable of reaching comparable or even better performance than full dense model, which has been evidenced in various ﬁelds, such as image classiﬁcation (Frankle & Carbin, 2019;
Liu et al., 2019; Wang et al., 2020a; Evci et al., 2019; Frankle et al., 2020;
Savarese et al., 2020;
Yin et al., 2020; You et al., 2020), natural language processing (Gale et al., 2019), reinforcement learning (Yu et al., 2020), and adversarial robustness (Cosentino et al., 2019).
Most works of LTH use unstructured weight magnitude pruning (Han et al., 2016; Frankle & Carbin, 2019) to ﬁnd the matching subnetworks, and the channel pruning also adopted in a recent work (You et al., 2020).
In order to scale up LTH to larger networks and datasets, the “late rewinding” technique is proposed by Frankle et al.
; Renda et al.
Mehta (2019); Morcos et al.
; Desai et al.
(2019) are the pioneers to study the transferability of found subnetworks.
However, all previous works focus on discriminative models.
In this paper, we extend LTH to GANs and reveal unique ﬁndings of GAN tickets.
P RELIMINARIES
In this section, we describe our pruning algorithms and list related experimental settings.
Backbone Networks We use two GANs in our experiments in Section 4: SNGAN (Miyato et al., 2018) and CycleGAN (Zhu et al., 2017)).
SNGAN with ResNet (He et al., 2016) is one of the most popular noise-to-image GAN network and has strong performance on several datasets like CIFAR10.
CycleGAN is a popular and well-studied image-to-image GAN network that also perform well on several benchmarks.
For SNGAN, let g ( z ; θ g ) be the output of the generator network G with parameters θ g and a latent variable z ∈ R d z and d ( x ; θ d ) be the output of the discriminator network D with parameters θ d and input example x .
For CycleGAN which is composed of two generatordiscriminator pairs, we use g ( x ; θ g ) and θ g again to represent the output and the weights of the two generators where x = ( x 1 , x 2 ) , where ( x 1 , x 2 ) indicates a pair of input examples.
The same modiﬁcation can be done for the two discriminators in CycleGAN.
Datasets For image-to-image experiments, we use a widely-used benchmark horse2zebra (Zhu et al., 2017) for model training.
As for noise-to-image experiments, we use CIFAR-10 (Krizhevsky et al., 2009) as the benchmark.
For the transfer study, the experiments are conducted on CIFARand STL-10 (Coates et al., 2011).
For better transferring, we resize the image in STL-10 to 32 × 32 .
Subnetworks For a network f ( · ; θ ) parameterized by θ , a subnetwork is deﬁned as f ( · ; m (cid:12) θ ) , where m ∈ { 0 , 1 } d θ is a pruning mask for θ ∈ R d θ and (cid:12) is the element-wise product.
For GANs, two separate masks, m d and m g , are needed for both the generator and the discriminator.
Consequently, a subnetwork of GANs is consistent of: a sparse generator g ( · ; m g (cid:12) θ g ) and a sparse discriminator d ( · ; m d (cid:12) θ d ) .
Let θ 0 be the initialization weights of model f
and θ t be the weights at training step t .
Following Frankle et al.
(2019), we deﬁne a matching network as a subnetwork f ( · ; m (cid:12) θ ) , where θ is initialized with θ t , that can reach the comparable performance to the full network within a similar training iterations when trained in isolation; a winning ticket is deﬁned as a matching subnetwork where t = 0 , i.e. θ initialized with θ 0 .
Finding subnetworks Finding GAN subnetworks is to ﬁnd two masks m g and m d for the generator and the discriminator.
We use both an unstructured magnitude method, i.e. the iterative magnitude pruning (IMP), and a structured pruning method, i.e. the channel pruning (He et al., 2017), to generate the masks.
For unstructured pruning, we follow the following steps.
After we ﬁnish training the full GAN model for N iterations, we prune the weights with the lowest magnitude globally (Han et al., 2016) to obtain masks m =
( m g , m d ) , where the position of a remaining weight in m is marked as one, and the position of a pruned weight is marked as zero.
The weights of the sparse generator and the sparse discriminator are then reset to the initial weights of the full network.
Previous works have shown that the iterative magnitude pruning (IMP) method is better than the one-shot pruning method.
So rather than pruning the network only once to reach the desired sparsity, we prune a certain amount of non-zero parameters and re-train the network several times to meet the requirement.
Details of this algorithm are in Appendix A.1, Algorithm 1.
As for channel pruning, the ﬁrst step is to train the full model as well.
Besides using a normal loss function L GAN , we follow Liu et al.
(2017) to apply a (cid:96) 1 -norm on the trainable scale parameters γ in the normalization layers to encourage channel-level sparsity: L cp =
||
γ || 1 .
To prevent the compressed network behave severely differently with the original large network, we introduce a distillation loss as Wang et al.
(2020b) did: L dist =
E z [dist( g ( z ; θ g ) , g ( z ; m g (cid:12) θ g ))] .
We train the
GAN network with these two additional losses for N 1 epochs and get the sparse networks g ( · ; m g (cid:12) θ g ) and d ( · ; m d (cid:12) θ g ) .
Details of this algorithm are in Appendix A.1, Algorithm 2.
Evaluation of subnetworks After obtaining the subnetworks g ( · ; θ g (cid:12) m g ) and d ( · ; θ d (cid:12) m d ) , we test whether the subnetworks are matching or not.
We reset the weights to a speciﬁc step i , and train the subnetworks for N iterations and evaluate them using two speciﬁc metrics, Inception
Score (Salimans et al., 2016) and Fr´echet Inception Distance (Heusel et al., 2017).
Other Pruning Methods We compare the size and the performance of subnetworks found by IMP with subnetworks found by other techniques that aim at compressing the network after training to reduce computational costs at inference.
We use a benchmark pruning approach named Standard
Pruning (Chen et al., 2020; Han et al., 2016), which iteratively prune the 20% of lowest magnitude weights, and train the network for another N iterations without any rewinding, and repeat until we have reached the target sparsity.
In order to verify that the statement of iterative magnitude pruning is better than one-shot pruning, we compare IMP G and IMP GD with their one-shot counterparts.
Additionally, we compare IMP with some randomly pruning techniques to prove the effectiveness of IMP.
They are: 1) Randomly
Pruning : Randomly generate a sparsity mask m (cid:48) .
Random Tickets :
Rewinding the weights to another initialization θ (cid:48) 0 .
T HE E XISTENCE OF W INNING T ICKETS IN GAN
In this section, we will validate the existence of winning tickets in GANs with initialization θ 0 :
= ( θ g 0 , θ d 0 ) .
Speciﬁcally, we will empirically prove several important properties of the tickets by authenticating the following four claims:
Claim 1 : Iterative Magnitude Pruning (IMP) ﬁnds winning tickets in GANs, g ( · ; m g (cid:12) θ g 0 ) and d ( · ; m d (cid:12) θ d 0 ) .
Channel pruning is also able to ﬁnd wining tickets as well.
Claim 2 : Whether pruning the discriminator D does not change the existence of winning tickets.
It is the initialization used in D that matters.
Moreover, pruning the discriminator has a slight boost of matching networks in terms of extreme sparsity and performance.
Claim 3 : IMP ﬁnds winning tickets at sparsity where some other pruning methods (randomly pruning, one-shot magnitude pruning, and random tickets) are not matching.
Claim 4 : The late rewinding technique (Frankle et al., 2019) helps.
Matching subnetworks that are initialized to θ i , i.e. , i steps from θ 0 , can outperform those initialized to θ 0 .
Moreover, matching subnetworks that are late rewound can be trained to match the performance of standard pruning.
Claim 1: Are there winning tickets in GANs?
To answer this question, we ﬁrst conduct experiments on SNGAN by pruning the generator only in the following steps: 1) Run IMP to get sequential sparsity masks ( m d i , m g i ) of sparsity s i % remaining weights; 2) Apply the masks to the GAN and reset the weights of the subnetworks to the same random initialization θ 0 ; 3) Train models to evaluate whether they are winning tickets.
1 We set s i %
=
(1 − 0 .
8 i ) × 100% , which we use for all the experiments that involve iteratively pruning hereinafter.
The number of training epochs for subnetworks is identical to that of training the full models.
[Figure]
Figure 1 veriﬁes the existence of winning tickets in SNGAN and CycleGAN.
We are able to ﬁnd winning tickets by iterative pruning the generators at highest sparsity, around 74% in SNGAN, and around 67% in CycleGAN, where the FID scores of these subnetworks successfully match the FID scores of the full network respectively.
The conﬁdence interval also suggests that the winning tickets at some sparsities are statistically signiﬁcantly better than the full model.
To show that channel pruning can ﬁnd winning tickets as well, we extract several subnetworks from the trained full SNGAN and CycleGAN by varying ρ in Algorithm 2.
We deﬁne the channel-pruned model’s sparsity as the ratio of MFLOPs between the sparse model and the full model.
We can conﬁrm that winning tickets can also be found by channel pruning (CP).
CP is able to ﬁnd winning tickets in SNGAN at sparsity around 34%.
We will analyze it more carefully in Section 7.
[Figure]
G e n
[Figure]
a t i o n I n t e r p o l a t i o n
[Figure]
Claim 2: Does the treatment of the discriminator affect the existence of winning tickets?
Previous works of GAN pruning did not analyze the effect of pruning the discriminator.
To study the effect, we compare two different iterative pruning settings: 1) Prune the generator only ( IMP G ) and 2)
Prune both the generator and the discriminator iteratively ( IMP GD ).
Both the generator and the discriminator are reset to the same random initialization θ 0 after the masks are obtained.
The FID scores of the two experiments are shown in Figure 4.
The graph suggests that the two settings share similar patterns: the minimal FID of IMP G is 14.19, and the minimal FID of IMP GD is 14.59.
The difference between these two best FID is only 0.4, showing a slight difference in generative power.
The FID curve of IMP G lies below that of IMP GD at low sparsity but lies above at high sparsity, indicating that pruning the discriminator produces slightly better performance when the percent of remaining weights is small.
The extreme sparsity where IMP GD can match the performance of the full model is 73.8%.
In contrast, IMP G can only match no sparser than 67.2%, demonstrating that pruning the discriminator can also push the frontier of extreme sparsity where the pruned models are still able to match.
In addition, we study the effects of different initialization for the sparse discriminator.
We compare different weights loading methods when applying the iterative magnitude pruning process: 1) Reset the weights of generator to θ g 0 and reset the weights of discriminator to θ d 0 , which is identical to IMP G ; 2) Reset the weights of generator to θ g 0 and ﬁne-tune the discriminator, which we will call IMP FG .
Figure 4 shows that resetting both the weights to θ 0 produces a much better result than only resetting the generator.
The discriminator D without resetting its weights is too strong for the generator G with initial weights θ g 0 that will lead to degraded performance.
In summary, different initialization of the discriminator will signiﬁcantly inﬂuence the existence and quality of winning tickets in GAN models. [Figure]
A follow-up question arises from the previous observations: is there any way to use the weights of the dense discriminator [Figure] as the direct usage of the dense weights yields inferior results?
One possible way is to use it as a “teacher” and transfer the knowledge to pruned discriminator using a consistency loss.
Formally speaking, an additional regularization term is used when training the whole network:
[Equation]
where KL Div denotes the KL-Divergence.
We name the iterative pruning method with this additional regularization IMP KDGD .
[Figure]
[Figure]
Figure 5 shows the result of pruning method
IMP KDGD compared to the previous two pruning methods we proposed, IMP G and IMP GD .
IMP KDGD is capable of ﬁnding winning tickets at sparsity around 70%, outperforming IMP G , and showing comparable results to IMP GD regarding the extreme sparsity.
The FID curve of setting IMP KDGD is further mostly located below the curve of IMP GD , demonstrating a stronger generative ability than IMP GD .
It suggests transferring knowledge from the full discriminator beneﬁts to ﬁnd the winning tickets.
Claim 3: Can IMP ﬁnd matching subnetworks sparser than other pruning methods?
Previous works claim that both a speciﬁc sparsity mask and a speciﬁc initialization are necessary for ﬁnding winning tickets (Frankle & Carbin, 2019), and iterative magnitude pruning is better than one-shot pruning.
To extend such a statement in the context of GANs, we compare IMP with several other benchmarks, randomly pruning (RP), one-shot magnitude pruning (OMP), and random tickets (RT), to see if IMP can ﬁnd matching networks at higher sparsity. [Figure] [Figure]
[Table] Figure 6 and Table 1 show that iterative magnitude pruning outperforms one-shot magnitude pruning no matter pruning the discriminator or not.
IMP ﬁnds winning tickets at higher sparsity (67.23% and 73.79%, respectively) than one-shot pruning (59.00% and 48.80%, respectively).
The minimal FID score of subnetworks found by IMP is smaller than that of subnetworks found by OMP as well.
This observation defends the statement that pruning iteratively is superior compared to one-shot pruning.
We also list the minimal FID scores and extreme sparsity of matching networks for other pruning methods in Figure 7 and Table 2.
It can be seen that IMP ﬁnds winning tickets at sparsity where some other pruning methods, randomly pruning and random initialization, cannot match.
Since
IMP G shows the best overall result, we authenticate the previous statement that both the speciﬁc sparsity mask and the speciﬁc initialization are essential for ﬁnding winning tickets.
[Figure] [Figure]
[Table]
Claim 4: Does rewinding improve performance?
In previous paragraphs, we show that we are able to ﬁnd winning tickets in both SNGAN and CycleGAN.
However, these subnetworks cannot match the performance of the original network at extremely high sparsity, while the subnetworks found by standard pruning can (Table 3).
To ﬁnd matching subnetworks at such high sparsity, we adopt the rewinding paradigm: after the masks are obtained, the weights of the model are rewound to θ i , the weights after i steps of training, rather than reset to the same random initialization θ 0 .
It was pointed out by Renda et al.
(2020) that subnetworks found by IMP and rewound early in training can be trained to achieve the same accuracy at the same sparsity as subnetworks found by the standard pruning, providing a possibility that rewinding can also help GAN subnetworks.
[Table]
We choose different rewinding settings: 5%, 10%, and 20% of the whole training epochs.
The results are shown in Table 3.
We observe that rewinding can signiﬁcantly increase the extreme sparsity of matching networks.
Rewinding to even only 5% of the training process can raise the extreme sparsity from 67.23% to 86.26%, and rewinding to 20% can match the performance of standard pruning.
We also compare the FID score of subnetworks found at 89% sparsity.
Rewind to 20% of the training process can match the performance of standard pruning at 89% sparsity, and other late rewinding settings can match the performance of the full model.
This suggests that late rewinding techniques can greatly contribute to matching subnetworks with higher sparsity.
Summary Extensive experiments are conducted to examine the existence of matching subnetworks in generative adversarial models.
We conﬁrmed that there were matching subnetworks at high sparsities, and both the sparsity mask and the initialization matter for ﬁnding winning tickets.
We also studied the effect of pruning the discriminator and demonstrate that pruning the discriminator can slightly boost the performance regarding the extreme sparsity and the minimal FID.
We proposed a method to utilize the weights of the dense discriminator model to boost the performance further.
We also compare IMP with different pruning methods, showing that IMP is superior to random tickets and random pruning.
In addition, late rewinding can match the performance of standard pruning, which again shows consistency with previous works.
T HE TRANSFER LEARNING OF GAN MATCHING NETWORKS
In the previous section, we conﬁrm the presence of winning tickets in GANs.
In this section, we will study the transferability of winning tickets.
Existing works (Mehta, 2019) show that the matching networks in discriminative models can transfer across tasks.
Here we evaluate this claim in GANs.
To investigate the transferability [Table] we propose three transfer experiments from CIFAR-10 to STLon SNGAN.
We ﬁrst identify matching subnetworks g ( · , m g (cid:12) θ ) and d ( · , m d (cid:12) θ ) on CIFAR10, and then train and evaluate the subnetworks on STL-10.
To assess whether the same random initialization θ 0 is needed for transferring, we test three different weights loading method: 1) reset the weights to θ 0 ; 2) reset the weights to another initialization θ (cid:48) 0 ; 3) rewind the weights to θ N .
We train the network on STL-10 using the same hyper-parameters as on CIFAR-10.
It is noteworthy that the hyper-parameters setting might not be optimal for the target task, yet it is fair to compare different transferring settings.
[Table]
The FID score of different settings is shown in Table 4.
Subnetworks initialized by θ 0 and using masks generated by IMP G can be trained to achieve comparable results to the baseline model.
Surprisingly, random re-initialization θ (cid:48) 0 shows better transferability than using the same initialization θ 0 in our transfer settings and outperforms the full model trained on STL-10, indicating that the combination of θ 0 and the mask generated by IMP GD is more focused on the source dataset and consequently has lower transferability.
Summary
In this section, we tested the transferability of IMP subnetworks.
Transferring from θand θ r both produce matching results on the target dataset, STL-10.
θ 0 works better with masks generated by IMP G while the masks generated by IMP GD prefer a different initialization θ r .
Given that IMP GD performs better on CIFAR-10, it is reasonable that the same initialization θ 0 has lower transferability when using masks from IMP GD .
E XPERIMENTS ON OTHER GAN MODELS AND OTHER DATASETS
We conducted experiments on DCGAN (Radford et al., 2016), WGAN-GP (Gulrajani et al., 2017), ACGAN (Odena et al., 2017), GGAN (Lim & Ye, 2017), DiffAugGAN (Zhao et al., 2020a), as well as a NAS-based GAN, AutoGAN (Gong et al., 2019). We use CIFAR-10 and Tiny ImageNet (Wu et al., 2017) as our benchmark datasets.
[Table]
[Figure]
[Table]
Table 5 and Table 6 verify that the existence of winning tickets in several GAN networks in spite of the different extreme sparsities, showing that the lottery ticket hypothesis can be generalized to various GAN models.
E FFICIENCY OF GAN W INNING T ICKETS
[Figure]
Unlike the unstructured magnitude pruning method, channel pruning can reduce the number of parameters in GANs.
Therefore, winning tickets found by channel pruning are more efﬁcient than the original model regarding computational cost.
To fully exploit the advantage of subnetworks founded by structural pruning, we further compare our prune-and-train pipeline with a state-of-the-art GAN compression framework (Wang et al., 2020b).
The pipeline is described as follows: after extracting the sparse structure generated by channel pruning, we reset the model weights to the same random initialization θ 0 and then train for the same number of epochs as the dense model used.
We can see from Figure 8 that subnetworks are founded by channel pruning method at about 67% sparsity, which provides a new path for winning tickets other than magnitude pruning.
The matching networks at 67.7% outperform GS-32 regarding Inception Score by 0.25; the subnetworks at about 29% sparsity can outperform GS-32 by 0.20, setting up a new benchmark for GAN compressions.
C ONCLUSION
In this paper, the lottery ticket hypothesis has been extended to GANs.
We successfully identify winning tickets in GANs, which are separately trainable to match the full dense GAN performance.
Pruning the discriminator, which is rarely studied before, had only slight effects on the ticket ﬁnding process, while the initialization used in the discriminator is essential.
We also demonstrate that the winning tickets found can transfer across diverse tasks.
Moreover, we provide a new way of ﬁnding winning tickets that alter the structure of models.
Channel pruning is able to extract matching subnetworks from a dense model that can outperform the current state-of-the-art GAN compression after resetting the weights and re-training.
A M ORE T ECHNICAL D ETAILS A. A LGORITHMS
In this section, we describe the details of algorithm we used in ﬁnding lottery tickets.
Two distinct pruning methods are used in Algorithm 1 and Algorithm 2.
[Table]
B M ORE E XPERIMENTS R ESULTS AND A NALYSIS
We will provide extra experiments results and analysis in this section
. B. M ORE VISUALIZATION OF IMP W INNING T ICKETS
We also conducted experiments to ﬁnd winning tickets in CycleGAN on dataset winter2summer (Zhu et al., 2017).
We observed similar patterns and found matching networks at 79.02% sparsity.
We randomly sample four images from the dataset and show the translated images in Figure 10, Figure and Figure 12.
The winning tickets of CycleGAN can generate comparable visual quality to the full model under all cases.
[Figure]
B. E XTRA M ETRICS FOR E VALUATING SNGAN
We also evaluate the quality of images generated by SNGAN using precision and recall Sajjadi et al.
The results are shown in Table 7 and Figure 9.
The results show that the best winning tickets have higher F 8 and F 1 / 8 compared to the full model, and the extreme winning tickets have on-par performance.
[Figure]
[Figure] B. C HANNEL P RUNING FOR SNGAN
We study the relationship between the Inception Score and the remaining model size , i e. the ratio between the size of a channel-pruned model and its original model.
The results are plotted in Figure 13.
A similar conclusion can be drawn from the graph that matching networks exist and at the same sparsity, the matching networks can be trained to outperform the current state-of-the-art GAN compression framework
B. C HANNEL P RUNING FOR C YCLE GAN
We also conducted experiments on CycleGAN using channel pruning.
The task we choose is horseto-zebra, i.e. , we prune each of the two generators separately, which is aligned with SNGAN which has only one generator.
We prove that channel pruning is also capable of ﬁnding winning tickets in
CycleGAN in Figure 14.
Moreover, at extreme sparsity the sparse subnetwork that we obtain can be trained to reach slightly better results than the current state-of-the-art GAN compression framework without quantization.
Input: The desired sparsity s
Output: An sparse GAN g ( · ; m g (cid:12) θ g ) and d ( · ; m d (cid:12) θ d ) 1 Set
m g =
{ 1 } | θ g 0 | and m d = { 1 } | θ
d 0 | .
2 Set θ g 0 :
= initial weights of the generator model, θ d 0 := initial weights of the discriminator model.
3 Iteration i =4 while the sparsity of m g < s do
Train the generator g ( · ; m g (cid:12) θ g 0 ) and the discriminator d ( · ; m d (cid:12) θ d 0 ) for N epochs to get parameters θ i g N and θ id N .
if pruning the discriminator then
Prune 20% of the parameters in θ ig N and θ id N , creating two mask m (cid:48) g and m (cid:48) d . else
Prune 20% of the parameters in θ ig N , creating a mask m (cid:48) g .
m (cid:48)
d remains { 1 } | θ d 0 | .
end
Update m g = m (cid:48) g and m d = m (cid:48)
d i =
i +13 end
Input: A threshold ρ of importance score, number of steps for training N
Output: An sparse GAN g ( · ; m g (cid:12) θ g 0 ) and d ( · ; m d (cid:12) θ d 0 ) 1 Randomly initialize γ g for every normalization layers in generator G and γ d for discriminator D .
γ =
( γ g , γ d ) 2 Set θ g 0 :
= initial weights of the generator model, θ d 0 := initial weights of the discriminator model.
3 i =4 while i < N do
Compute masks m d from γ d and m g from γ g .
Compute L cp , L GAN and L dist .
Update θ g and θ d by training the generator g ( · ; m g (cid:12) θ g ) and the discriminator d ( · ; m d (cid:12) θ d ) for one step.
Update γ : γ ← prox ρη ( γ − η ∇ γ L cp ) , where prox λ ( x ) = sgn( x ) (cid:12) max( | x |− λ · 1 , 0)
i ← i +10 end
Calculate the ﬁnal masks m g from γ g and m d from γ d