S ELF -S UPERVISED GAN C OMPRESSION
Anonymous authors
Paper under double-blind review
A BSTRACT
Deep learning’s success has led to larger and larger models to handle more and more complex tasks; trained models can contain millions of parameters.
These large models are compute- and memory-intensive, which makes it a challenge to deploy them with minimized latency, throughput, and storage requirements.
Some model compression methods have been successfully applied on image classiﬁcation and detection or language models, but there has been very little work compressing generative adversarial networks (GANs) performing complex tasks.
In this paper, we show that a standard model compression technique, weight pruning, cannot be applied to GANs using existing methods.
We then develop a selfsupervised compression technique which uses the trained discriminator to supervise the training of a compressed generator.
We show that this framework has a compelling performance to high degrees of sparsity, can be easily applied to new tasks and models, and enables meaningful comparisons between different pruning granularities.
I NTRODUCTION
Deep Neural Networks (DNNs) have proved successful in various tasks like computer vision, natural language processing, recommendation systems, and autonomous driving.
Modern networks are comprised of millions of parameters, requiring signiﬁcant storage and computational effort.
Though accelerators such as GPUs make realtime performance more accessible, compressing networks for faster inference and simpler deployment is an active area of research.
Compression techniques have been applied to many networks, reducing memory requirements and improving their performance.
Though these approaches do not always harm accuracy, aggressive compression can adversely affect the behavior of the network.
Distillation (Schmidhuber, 1991; Hinton et al., 2015) can improve the accuracy of a compressed network by using information from the original, uncompressed network.
Generative Adversarial Networks (GANs) (Schmidhuber, 1990; Goodfellow et al., 2014) are a class of DNN that consist of two sub-networks: a generative model and a discriminative model.
Their training process aims to achieve a Nash Equilibrium between these two sub-models.
GANs have been used in semi-supervised and unsupervised learning areas, such as fake dataset synthesis (Radford et al., 2016;
Brock et al., 2019), style transfer (Zhu et al., 2017b; Azadi et al., 2018), and image-to-image translation (Zhu et al., 2017a; Choi et al., 2018).
As with networks used in other tasks, GANs have millions of parameters and nontrivial computational requirements.
In this work, we explore compressing the generative model of GANs for more efﬁcient deployment.
We show that applying standard pruning techniques, with and without distillation, can cause the generator’s behavior to no longer achieve the network’s goal.
Similarly, past work targeted at compressing GANs for simple image synthesis fall short when they are applied to large tasks.
In some cases, this result is masked by loss curves that look identical to the original training.
By modifying the loss function with a novel combination of the pre-trained discriminator and the original and compressed generators, we can overcome this behavioral degradation and achieve compelling compression rates with little change in the quality of the compressed generator’s ouput.
We apply our technique to several networks and tasks to show generality.
Finally, we study the behavior of compressed generators when pruned with different amounts and types of sparsity, ﬁnding that ﬁlter pruning, a technique commonly used for accelerating image classiﬁcation networks, is not trivially applicable to GANs.
Our main contributions are:
• We illustrate that and explain why pruning the generator of a GAN with existing methods is unsatisfactory for complex tasks.
(Section 3) •
We propose self-supervised compression for the generator in a GAN (Section 4) • We show that our technique can apply to several networks and tasks (Section 5) • We show and analyze qualitative differences in pruning ratio and granularities.
(Section 6)
R ELATED R ESEARCH
A common method of DNN compression is network pruning (Han et al., 2015): setting the small weights of a trained network to zero and ﬁne-tuning the remaining weights to recover accuracy.
Zhu & Gupta (2018) proposed a gradual pruning technique (AGP) to compress the model during the initial training process.
Wen et al.
(2016) proposed a structured sparsity learning method that uses group regularization to force weights towards zero, leading to pruning groups of weights together.
Li et al.
(2017) pruned entire ﬁlters and their connecting feature maps from models, allowing the network to run with standard dense software libraries.
Though it was initially applied to image classiﬁcation networks, network pruning has been extended to natural language processing tasks (See et al., 2016; Narang et al., 2017) and to recurrent neural networks (RNNs) of all types - vanilla
RNNs, GRUs (Cho et al., 2014), and LSTMs (Hochreiter & Schmidhuber, 1997).
As with classiﬁcation networks, structured sparsity within recurrent units has been exploited (Wen et al., 2018).
A complementary method of network compression is quantization.
Sharing weight values among a collection of similar weights by hashing (Chen et al., 2015) or clustering (Han et al., 2016) can save storage and bandwidth at runtime.
Changing fundamental data types adds the ability to accelerate the arithmetic operations, both in training (Micikevicius et al., 2018) and inference regimes (Jain et al., 2019).
Several techniques have been devised to combat lost accuracy due to compression, since there is always the chance that the behavior of the network may change in undesirable ways when the network is compressed.
Using GANs to generate unique training data (Liu et al., 2018b) and extracting knowledge from an uncompressed network, known as distillation (Hinton et al., 2015), can help keep accuracy high.
Since the pruning process involves many hyperparameters, Lin et al.
(2019) use a
GAN to guide pruning, and Wang et al. (2019a) structure compression as a reinforcement learning problem; both remove some of the burden from the user.
E XISTING T ECHNIQUES F AIL TO P RUNE A C OMPLEX T ASK
Though there are two networks in a single GAN, the main workload at deployment is usually from the generative model, or generator.
For example, in image synthesis and style transfer tasks, the ﬁnal output images are created solely by the generator.
The discriminative model (discriminator) is vital in training, but it is abandoned afterward for many tasks.
So, when we try to apply state-of-the-art compression methods to GANs, we focus on the generator for efﬁcient deployment.
As we will see, the generative performance of the compressed generators is quite poor for the selected image-toimage translation task.
We look at two broad categories of baseline approaches: standard pruning techniques that have been applied to other network architectures, and techniques that were devised to compress the generator of a GAN performing image synthesis.
We compare to the dense baseline [a], our technique [b], as well as a small, dense network with the same number of parameters [c].
(Labels correspond to entries in Table 1, the overview of all techniques, and Figure 1, results of each technique).
Standard Pruning Techniques .
To motivate GAN-speciﬁc compression methods, we try variations of two state-of-the-art pruning methods: manually pruning and ﬁne tuning (Han et al., 2015) a trained dense model
[d], and AGP (Zhu & Gupta, 2018) from scratch [e] and during ﬁne-tuning [f].
We also include distillation (Hinton et al., 2015) to improve the performance of the pruned network with manual pruning [g] and AGP ﬁne-tuning [h].
Distillation is typically optional for other network types, since it is possible to get decent accuracy with moderate pruning in isolation.
For very aggressive compression or challenging tasks, distillation aims to extract knowledge for the compressed (student) network from original (teacher) network’s behavior.
We also ﬁx the discriminator of [g] to see if the discriminator was being weakened by the compressed generator [i].
Targeted GAN Compression .
There has been some work in compressing GANs with methods other than pruning, and only one technique applied to an image-to-image translation task.
We ﬁrst examine two approaches similar to ours.
Adversarial training (Wang et al., 2018)
[j] posits that during distillation of a classiﬁcation network, the student network can be thought of as a generative model attempting to produce features similar to that of the teacher model.
So, a discriminator was trained alongside the student network, trying to distinguish between the student and the teacher.
One could apply this technique to compress the generator of a GAN, but we ﬁnd that its key shortcoming is that it trains a discriminator from scratch.
Similarly, distillation has been used to compress GANs in Aguinaldo et al.
[k], but again, the “teacher” discriminator was not used when teaching the “student” generator.
Learned Intermediate Representation Training (LIT) (Koratana et al., 2019)
[l] compresses StarGAN by a factor of 1 . 8 × by training a shallower network.
Crucially, LIT does not use the pre-trained discriminator in any loss function.
Quantized GANs (QGAN) (Wang et al., 2019b)
[m] use a training process based on Expectation-Maximization to achieve impressive compression results on small generative tasks with output images of 32x32 or 64x64 pixels.
Liu et al.
(2018a)
ﬁnd that maintaining a balance between discriminator and generator is key: their approach is to selectively binarize parts of both networks in the training process on the Celeb-A generative task, up to 64x64 pixels.
So, we try pruning both networks during the training process [n].
Experiments .
For these experiments, we use StarGAN (Choi et al., 2018) trained with the Distiller (Zmora et al., 2018) library for the pruning.
StarGAN 1 extends the image-to-image translation capability from two domains to multiple domains within a single uniﬁed model.
It uses the CelebFaces Attributes (CelebA)
(Liu et al., 2015) as the dataset.
CelebA contains 202,599 images of celebrities’ faces, each annotated with 40 binary attributes.
As in the original work, we crop the initial images from size 178 × 218 to 178 × 178 , then resize them to 128 × 128 and randomly select 2,000 images as the test dataset and use remaining images for training.
The aim of StarGAN is facial attribute translation: given some image of a face, it generates new images with ﬁve domain attributes changed: 3 different hair colors (black, blond, brown), different gender (male/female), and different age (young/old).
Our target sparsity is 50% for each approach.
We stress that we attempted to ﬁnd good hyperparameters when using the existing techniques, but standard approaches like reducing the learning rate for ﬁne-tuning (Han et al., 2015), etc., were not helpful.
Further, the target sparsity, 50%, is not overly aggressive, and we do not impose any structure; other tasks readily achieve 80%-90% ﬁne-grained sparsity with minimal accuracy impact.
The results of these trials are shown in Figure 1.
Subjectively, it is easy to see that the existing approaches (1c through 1n) produce inferior results to the original, dense generator.
Translated facial images from pruning & na¨ıve ﬁne-tuning (1d and 1e) do give unique results for each latent variable, but the images are hardly recognizable as faces.
These ﬁne-tuning procedures, along with AGP from scratch (1f) and distillation from intermediate representations (1l), simply did not converge.
One-shot pruning and traditional distillation (1g), adversarial learning (1j), knowledge distillation (1k), training a “smaller, dense” half-sized network from scratch (1c) and pruning both generator and discriminator (1n) keep facial features intact, but the image-to-image translation effects are lost to mode collapse (see below).
There are obvious mosaic textures and color distortion on the translated images from ﬁne-tuning & distillation (1h), without ﬁne-tuning the original loss (1i), and from the pruned model based on the Expectation-Maximization (E-M) algorithm (1m).
However, the translated facial images from a generator compressed with our proposed self-supervised GAN compression method (1b) are more natural, nearly indistinguishable from the dense baseline (1a), matching the quantitative Frechet Inception Distance (FID) scores (Heusel et al., 2017) in Table 1.
While past approaches have worked to prune some networks on other tasks (DCGAN generating
MNIST digits, see the supplementary material), we show that they do not succeed on larger imageto-image translation tasks, while our approach works on both.
Similarly, though LIT (Koratana et al., 2019)
[l] was able to achieve a compression rate of 1 . 8 × on this task by training a shallower network, it does not see the same success at network pruning. Discussion .
It is tempting to think that the loss curves of the experiment for each technique can tell us if the result is good or not.
We found that for many of these experiments, the loss curves correctly predicted that the ﬁnal result would be poor.
However, the curves for [h] and [m] look very
[Table]
[Figure] Dense [Figure] Pretrained
Yes - - - -
Nofaces 194. [Figure] Dense
[Figure] Sparse [Figure] Pretrained
No
Yes Yes
YesColorartifacts 159. [Figure]
Dense,Sparse FromDense Sparse [Figure] Pretrained
No
Yes Yes
YesModecollapse 46.
a b c d e f g h
i
j k l m n good - the compressed generator and discriminator losses converge at 0, just as they did for baseline training.
It is clear from the results of querying the generative models (Figures 1h and 1m), though, that this promising convergence is a false positive.
In contrast, the curves for our technique predict good performance, and, as we prune more aggressively in Section 6, higher loss values correlate well with worsening FID scores.
(Loss curves are provided in the Appendix .)
As pruning and distillation are very effective when compressing models for image classiﬁcation tasks, why do they fail to compress this generative model?
We share three potential reasons:
Standard pruning techniques need explicit evaluation metrics; softmax easily reﬂects the probability distribution and classiﬁcation accuracy.
GANs are typically evaluated subjectively, though some imperfect quantitative metrics have been devised.
GAN training is relatively unstable (Arjovsky et al., 2017;
Liu et al., 2018a) and sensitive to hyperparameters.
The generator and discriminator must be well-matched, and pruning can disrupt this ﬁne balance.
The energy of the input and output of a GAN is roughly constant, but other tasks, such as classiﬁcation, produce an output (1-hot label vector) with much less entropy than the input (three-channel color image of thousands of pixels).
Elaborating on this last point, there is more tolerance in the reduced-information space for the compressed classiﬁcation model to give the proper output.
That is, even if the probability distribution inferred by the original and compressed classiﬁcation models are not exactly the same, the classiﬁed labels can be the same.
On the other hand, tasks like style-transfer and dataset synthesis have no obvious energy reduction.
We need to keep entropy as high as possible (Kumar et al., 2019) during the compression process to avoid mode collapse – generating the same output for different inputs or tasks.
Attempting to train a new discriminator to make the compressed generator behave more like the original generator (Wang et al., 2018) suffers from this issue – the new discriminator quickly falls into a low-entropy solution and cannot escape.
Not only does this preclude its use on generative tasks, but it means that the compressed network for any task must also be trained from scratch during the distillation process, or the discriminator will never be able to learn.
S ELF -S UPERVISED G ENERATOR C OMPRESSION
We seek to solve each of the problems highlighted above.
Let us restate the general formulation of GAN training: the purpose of the generative model is to generate new samples which are very similar to the real samples, but the purpose of the discriminative model is to distinguish between real samples and those synthesized by the generator.
A fully-trained discriminator is good at spotting differences, but a well-trained generator will cause it to believe that the a generated sample is both real and generated with a probability of 0.5.
Our main insight follows:
By using this powerful discriminator that is already well-trained on the target data set, we can allow it to stand in as a quantitative subjective judge (point 1, above) – if the discriminator can’t tell the difference between real data samples and those produced by the compressed generator, then the compressed generator is of the same quality as the uncompressed generator.
A human no longer needs to inspect the results to judge the quality of the compressed generator.
This also addresses our second point: by starting with a trained discriminator, we know it is well-matched to the generator and will not be overpowered.
Since it is so capable (there is no need to prune it to), it also helps to avoid mode collapse.
As distillation progresses, it can adapt to and induce ﬁne changes in the compressed generator, which is initialized from the uncompressed generator.
Since the original discriminator is used as a proxy for a human’s subjective evaluation, we refer to this as “self-supervised” compression.
We illustrate the workﬂow in Figure 2, using a GAN charged with generating a map image from a satellite image in a domain translation task.
In the right part of Figure 2, the R eal S atellite I mage ( RSI ) goes through the original generative model ( G O ) to produce a F ake M ap I mage ( FMI- G O ).
The corresponding generative loss value is l- G O .
Accordingly, in the left part of Figure 2, RSI goes through the compressed generative model ( G C ) to produce a F ake M ap I mage ( FMI- G C ).
The corresponding generative loss value is l- G C .
This is the inference process of the original and compressed generators, expressed as follows:
[Equation]
The overall generative difference is measured between the two corresponding generative losses 2 .
We use a generative consistent loss function ( L GC ) in the bottom of Figure 2 to represent this process.
[Equation]
[Figure]
Since the GAN training process aims to reduce the differences between real and generated samples, we stick to this principle in the compression process.
In the upper right of Figure 2, R eal M ap I mage ( RMI ) and F ake M ap I mage ( FMI- G O ) go through the original discriminative model D O .
D O tries to ensure that the distribution of FMI- G O is indistinguishable from RMI using an adversarial loss.
The corresponding discriminative loss value is l- D O .
In the upper left of Figure 2, RMI and FMI- G C also go through the original discriminative model D O .
In this way, we use the original discriminative model as a ”self-supervisor.”
The corresponding discriminative loss value is l- D C .
[Equation]
So the discriminative difference is measured between two corresponding discriminative losses.
We use the discriminative consistent loss function L DC in the top of Figure 2 to represent this process.
[Equation]
The generative and discriminative consistent loss functions ( L GC and L DC ) use the weighted normalized Euclidean distance.
Taking the StarGAN task as the example (other tasks may use different losses):
[Equation]
where l- Gen is the generation loss term, l-
Cla is the classiﬁcation loss term, and l-
Rec is the reconstruction loss term.
α and β are the weight ratios among three loss types.
(We use the same values of α and β used in the original StarGAN baseline.)
[Equation]
where l- Dis is the discriminative loss item, l-
GP is the gradient penalty loss item, and δ is a weighting factor (again, we use the same value as the baseline).
The overall loss function of GAN compression consists of generative and discriminative differences:
[Equation]
where λ is the parameter to adjust the percentages between generative and discriminative losses.
We showed promising results with this method above in the context of prior methods.
In the following experiments, we investigate how well the method applies to other networks and tasks (Section 5) and how well the method works on different sparsity ratios and pruning granularities (Section 6).
[Table]
A PPLICATION TO N EW T ASKS AND N ETWORKS
For the experiments in this section, we choose to prune individual weights in the generator.
The ﬁnal sparsity rate is 50% for all convolution and deconvolution layers in the generator (more aggressive sparsities are discussed in Section 6).
Following AGP (Zhu & Gupta, 2018) [Table] we gradually increase the sparsity from 5% at the beginning to our target of 50% halfway through the self-supervised training process
[Table]
and we set the loss adjustment parameter λ to 0.5 in all experiments.
We use PyTorch (Paszke et al., 2017), implement the pruning and training schedules with Distiller (Zmora et al., 2018), and train and generate results with a V100 GPU (NVIDIA, 2017) using FP32 to match public baselines.
In all experiments, the data sets, data preparation, and baseline training all follow from the public repositories - details are summarized in Table 2.
We start by assuming an extra 10% of the original number of epochs will be required; in some cases, we reduced the overhead to only 1% while maintaining subjective quality.
We include representative results for each task, but a more comprehensive collection of outputs for each experiment is included in the Appendix .
Image Synthesis .
We apply the proposed compression method to DCGAN (Radford et al., 2016) 3 , a network that learns to synthesize novel images belonging to a given distribution.
We task DCGAN with generating images that could belong to the MNIST data set, with results shown in Figure 3.
Domain Translation .
We apply the proposed compression method to pix2pix (Isola et al., 2017) 4 , an approach to learn the mapping between paired training examples by applying conditional adversarial networks.
In our experiment, the task is synthesizing fake satellite images from label maps and vice-versa.
Representative results of this bidirectional task are shown in Figure 4.
Style Transfer .
We apply the proposed compression method to CycleGAN (Zhu et al., 2017a), used to exchange the style of images from a source domain to a target domain in the absence of paired training examples.
In our experiment, the task is to transfer the style of real photos with that
[Figure]
https com/
-and
Input
Dense 50% Sparse
Difference (10x)
Mapto
Satellite
FID: 35.
SatellitetoMap
FID: 17.
Input
Dense 50% Sparse
Difference (10x)
Monet to
Photo
FID: 59.
Photo to
Monet
FID: 35. of the Monet’s paintings.
Representative results of this bidirectional task are shown in Figure 5: photographs are given the style of Monet’s paintings and vice-versa.
Image-to-image Translation .
In addition to the StarGAN results above (Section 3, Figure 1), we apply the proposed compression method to CycleGAN (Zhu et al., 2017a) performing bidirectional translation between zebra and horse images.
Results are shown in Figure 6.
Input
Dense 50% Sparse
Difference (10x)
ZebratoHorse
FID: 47.
Horse to
Zebra
FID: 52.
[Table]
Super Resolution .
We apply self-supervised compression to SRGAN (Ledig et al., 2017) 5 , which uses a discriminator network trained to differentiate between upscaled and the original highresolution images.
We trained SRGAN on the DIV2K data set Agustsson & Timofte (2017), and use the DIV2K validation images, as well as Set5 Bevilacqua et al.
(2012) and Set14 Zeyde et al.
(2010) to report deployment quality.
In this task, quality is often evaluated by two metrics: Peak Signal-toNoise Ratio (PSNR) (Huynh-Thu & Ghanbari, 2008) and Structural Similarity (SSIM)
(Wang et al., 2004).
We also show FID scores (Heusel et al., 2017) for our results in the results summarized in
Table 3, and a representative output is shown in Figure 7.
These results also include ﬁlter-pruned generators (see Section 6).
E FFECT OF P RUNING R ATIO AND G RANULARITY
After showing that self-supervised compression applies to many tasks and networks with a moderate, ﬁne-grained sparsity of 50%, we expand the scope of the investigation to include different pruning granularities and rates.
From coarse to ﬁne, we can compress and remove the entire ﬁlters (3D-level), kernels (2D-level), vectors (1D-level) or individual elements (0D-level).
In general, ﬁner-grained pruning results in higher accuracy for a given sparsity rate, but coarser granularities are easier to exploit for performance gains due to their regular structure.
Similarly, different sparsity rates, leaving many nonzero weights or few, can result in varying levels of quality in the ﬁnal network.
We pruned all tasks by removing both single elements (0D) and entire ﬁlters (3D).
Further, for each granularity, we pruned to ﬁnal sparsities of 25%, 50%, 75%, and 90%.
Representative results for
CycleGAN (Monet → Photo) are shown in Figure 8, but in general, 0D pruning is less invasive, even at higher sparsities.
Up to 90% ﬁne-grained sparsity
, some ﬁne details faded away in pix2pix, but ﬁlter pruning results in drastic color shifts and loss of details at even 25% sparsity.
C ONCLUSION AND F UTURE W ORK
In this paper, we propose using a pre-trained discriminator to self-supervise the compression of a generative adversarial network.
We show that it is effective and applies to many tasks commonly solved with GANs, unlike traditional compression approaches.
Comparing the compressed generators with the baseline models on different tasks, we can conclude that the compression method
Sparsity 0% 25% 50% 75% 90%
Fine-grained
FID: 32.
Filter-pruned
FID: 32. performs well both in subjective and quantitative evaluations.
Advantages of the proposed method include:
• The results from the compressed generators are greatly improved over past work.
• The self-supervised compression is much shorter than the original GAN training process.
It only takes 1%-10% training effort to get an optimal compressed generative model.
• It is an end-to-end compression schedule that does not require objective evaluation metrics.
• We introduce a single optional hyperparameter (ﬁxed to 0.5 for all our experiments).
We use self-supervised GAN compression to show that pruning whole ﬁlters, which can work well for image classiﬁcation models (Li et al., 2017), may perform poorly for GAN applications.
Even pruned at a moderate sparsity (e.g. 25% in Figure 8), the generated image has an obvious color shift and does not transfer the photorealistic style.
In contrast, the ﬁne-grained compression stategy works well for all tasks we explored.
SRGAN seems to be an exception to ﬁlter-pruning’s poor results; we have to look closely to see differences, and it’s not clear which is subjectively better.
We have not tried to achieve extremely aggressive compression rates with complicated pruning strategies.
Different models may be able to tolerate different amounts of pruning when applied to a task, which we leave to future work.
Similarly, we have used network pruning to show the importance and utility of the proposed method, but self-supervised compression is general to other techniques, such as quantization, weight sharing, etc.
There are other tasks for which GANs can provide compelling results, and newer networks for tasks we have already explored; future work will extend our self-supervised compression method to these new areas.
Finally, self-supervised compression may apply to other network types and tasks if a discriminator is trained alongside the teacher and student networks.
A A PPENDIX
Since GANs are typically judged with subjective, qualitative observations, we present several results for each of the experiments in the main paper so readers can see the motivation for the conclusions drawn therein.
We organize this document in the following way:
• Section A.1: Na¨ıve Compression with StarGAN • Section A.2:
Image Synthesis with DCGAN • Section A.3: Domain Translation with Pix2Pix • Section A.4: Style Transfer with CycleGAN • Section A.5: Image-Image Translation with CycleGAN • Section A.6: Image-Image Translation with StarGAN • Section A.7: Super Resolution with SRGAN • Section A.8: Effect of Sparsity Granularity and Ratio
[Figure]
The loss curves for the comparative experiment in Figure 1 are shown in Figure 9.
[Figure]
Figures 10-12 show outputs of StarGAN compressed with various existing techniques ( c-n ), and the proposed
-supervised method ( b ).
baseline output
Each one input face translated to have black hair, blond hair, brown hair, the gender a different age different
Figure 9) a b c d e f g h
i
j k l m n a b c d e f g h
i
j k l m n a b c d e f g h
i
j k l m n
A. I MAGE S YNTHESIS : DCGAN (50% AND 75% F INE -G RAINED S PARSITY )
FID: 37. FID: 37.
[Figure] [Figure]
[Figure] [Figure]
A. S UPER R ESOLUTION : SRGAN (50% F INE -G RAINED AND F ILTER -P RUNED S PARSITY )
[Figure] [Figure]
E FFECT OF S PARSITY G RANULARITY AND R ATIO
[Figure]
[Figure] [Figure]
The loss curves for the comparative experiment in Figure 38 and 40 are shown in Figure 43.
[Figure]
The loss curves for the comparative experiment in Figure 39 and 41 are shown in Figure 44.
[Figure]