FreGAN: Exploiting Frequency Components for Training GANs under Limited Data
Abstract
Training GANs under limited data often leads to discriminator overﬁtting and memorization issues, causing divergent training.
Existing approaches mitigate the overﬁtting by employing data augmentations, model regularization, or attention mechanisms.
However, they ignore the frequency bias of GANs and take poor consideration towards frequency information, especially high-frequency signals that contain rich details.
To fully utilize the frequency information of limited data, this paper proposes FreGAN, which raises the model’s frequency awareness and draws more attention to producing high-frequency signals, facilitating high-quality generation.
In addition to exploiting both real and generated images’ frequency information, we also involve the frequency signals of real images as a self-supervised constraint, which alleviates the GAN disequilibrium and encourages the generator to synthesize adequate rather than arbitrary frequency signals.
Extensive results demonstrate the superiority and effectiveness of our FreGAN in ameliorating generation quality in the low-data regime (especially when training data is less than 100).
Besides, FreGAN can be seamlessly applied to existing regularization and attention mechanism models to further boost the performance.
Introduction
Generative adversarial networks (GANs) [10] have shown impressive achievements in synthesising plausible and photorealistic visual objects, such as image [18] [17] and video [39] generation, image inpainting [25], image translation [35] and so on.
However, a prerequisite of such success is sufﬁcient training data, which impedes applications of GANs in areas where only dozens of data are available or where it is challenging to collect massive data due to geographical, spatial, temporal, or privacy reasons.
Thus developing data-efﬁcient GANs that can generate plausible images under limited data, without compromising the quality, is necessary and meaningful
Training GANs under limited data often leads to overﬁtting and instability issues [16] [14].
Speciﬁcally, when the discriminator (D) overﬁts to the limited training data, it simply remembers the input real images and classiﬁes others as fake images, thus providing meaningless feedback to the generator (G), leading to divergent training and poor-quality generation.
Ameliorating the synthesize quality under limited data is still an unexplored problem.
Recent approaches for this problem include enlarging the training set with different data augmentations [52] [36] [51] [16] [14], regularizing the output of D with an additional constraint [37], and devising new network architectures [24].
However, existing methods are mainly developed from the perspective of data scale and model capacity, and they ignore a critical property of the data itself, i.e. , frequency signals.
GANs have been demonstrated to have a spectral bias in ﬁtting frequency signals [29] [34].
They preferentially ﬁt low-frequency signals and tend to ignore high-frequency signals [45], which encode ﬁne details like vertical and horizontal edges [47] [9].
Missing them may lead to unrealistic image synthesize with unsatisfactory artifacts (see Fig. 1).
This paper proposes a frequency-aware model, termed as FreGAN, to raise the frequency awareness of G and D. By encouraging G to generate more reasonable and adequate high-frequency signals, our FreGAN ameliorates the synthesize quality under limited data, as shown in Fig.
To fully exploit the frequency information of limited training data, we ﬁrst decompose images into different frequency components via Haar wavelet transformation [6].
Unlike traditional wavelet transformation that is employed at the image level, we perform it on the intermediate features of both D and
G. We then employ a high-frequency discriminator (HFD) and frequency skip connection (FSC) to raise the frequency awareness of G and D, respectively.
However, G still has no explicit clue about what high-frequency signals it should synthesize, and D is overconﬁdent in making real/fake decisions after seeing real and fake images.
Such an unbalanced competition motivates us to perform high-frequency alignment (HFA) to alleviate the information asymmetry between G and D. Innovatively, we explicitly exploit the frequency signals of real images induced from D as a self-supervised constraint to guide G to leverage the frequency knowledge properly.
Besides, HFD and HFA are applied on multi-scale features to thoroughly excavate the frequency signals of limited data, mitigating frequency bias and loss of high-frequency information.
The primary contributions of this paper are three-fold: 1) we propose FreGAN to raise the model’s frequency awareness, which successfully mines and exploits frequency information of limited data, and as a byproduct, FreGAN alleviates the unhealthy competition between G and D; 2) we demonstrate the compatibility of our model by combining our method with other techniques like regularization [37] and attention mechanism [22]; 3) we perform extensive experiments on various datasets with limited data, and our FreGAN achieves state-of-the-art performance on these datasets, indicating the effectiveness and superiority of our method for ameliorating synthesize quality, especially when training data is extremely limited.
Related Work
Generative Adversarial Networks.
Generative adversarial networks (GANs) [44] [15] [11], which target at generating plausible and realistic images, have made massive progress since the pioneering work [10].
The capability of GANs enables various visual applications like image [18] [19] and video generation [39] [38], image inpainting [48] [25], image manupulation [9] [35] and superresolution [42], etc.
However, GANs are notoriously difﬁcult to train as several issues like mode collapse and instability happen easily.
Numerous techniques have been proposed to stabilize training and improve the synthesize quality by designing new optimization objectives or network architectures [13].
WGAN [2] and f -GAN [27] minimize the Wasserstein distance and the f -divergence of real and generated distribution instead of minimizing JS divergence in [10].
BigGAN
[7] and
StyleGAN series [15] [18] [19] [16] [17] have made breakthrough progress in producing realistic images.
However, the performance of these models deteriorates when given limited data.
Training GANs under limited data.
Improving the synthesize quality under limited data remains an underexplored problem, which has drawn extensive attention recently.
Insufﬁcient training data leads to discriminator overﬁtting, thus degrading the quality of generated images.
One straightforward way to address such data scarcity is to expand the training set with various augmentations.
In addition to employing conventional augmentation techniques [36] [52] (e.g., ﬂip, crop), ADA [19] and DiffAug [51] propose adaptive and differentiable augmentation to enlarge the training data, respectively.
APA [14] deceives D based on the degree of overﬁtting with an adaptive pseudo augmentation.
InsGen [46] involves instance discrimination as an auxiliary task to encourage D to distinguish every individual image, which improves the discriminative power of the discriminator.
Lecam [37] regularizes the output of the discriminator throughout the training process.
FastGAN [24] employs a skip-layer channel-wise excitation module and a self-supervised discriminator to stabilize and accelerate the training.
The most recently MoCA [22] improves few-shot image generation quality with a prototype memory with an attention mechanism.
Beneﬁt from the signiﬁcant progress of large-scale pre-trained visual recognition models, Vision-aided GAN [20] uses available offthe-shelf models to help the GAN training and ProjectedGAN [31] improve GANs by projecting generated and real images into pre-trained feature spaces.
Another category of methods transfer and reuse knowledge from models that are pre-trained on large-scale data, i.e. , few-shot GAN adaptation [41] [23] [40] [28].
In this paper, we ameliorate the synthesize quality under limited data from the frequency domain perspective.
By raising the frequency awareness of GANs and providing more ﬁne details to G, we facilitate photorealistic image generation.
Our work is complementary to previous model regularization and attention mechanism approaches, and our method promotes equilibrium between G and D.
Wavelet Transformation in GANs.
Schwarz et al.
[32] prove that GANs exhibit a frequency bias and resolving frequency artifacts is necessary for photorealistic image generation.
Consequently,
GANs tend to ignore high-frequency signals as they are hard to generate, compromising the generation quality.
Wavelet transformation [6], which decomposes images into frequency components with different bands, has been wildly used in various applications of GANs, such as style transfer [4] [47], image inpainting [48], image editing [9], etc.
HiFA [9] alleviates the generator’s pressure of producing high-frequency signals by directly feeding high-frequency components to the generator.
WaveFill [48] disentangles different frequency signals and explicitly ﬁlls the missing regions in each frequency band, achieving superior image inpainting.
Zhang et al.
[49] propose wavelet knowledge distillation towards efﬁcient image-to-image translation without a performance drop.
SWAGAN [8] incorporates wavelet with the hierarchical training of StyleGAN2 [19] and performs wavelets at the image level.
Our FreGAN is more ﬂexible by directly decomposing intermediate features of the generator and the discriminator into the wavelet domain, and no additional down/up sampling are required to convert images to higher/lower resolution as in SWAGAN, which makes our method more efﬁcient.
Unlike existing methods that are performed on ample data, this paper addresses the more challenging few-shot generation problem.
In addition to raising the frequency awareness of the model, we also mitigate the unhealthy competition by lessening the frequency gap between G and D.
Methodology
The overall framework of our FreGAN is illustrated in Fig.
To formulate our method, we explicitly utilize wavelet transformation to decompose features into different frequency components.
We then employ high-frequency discriminator (HFD) and frequency skip connection (FSC) to raise the frequency awareness of G and D, respectively.
Moreover, we perform high-frequency alignment (HFA) to further guide G to synthesize adequate frequency signals.
3.1 Wavelet Transformation
To decompose images into different frequency components, we adopt a simple but effective wavelet transformation, i.e. , Haar wavelet.
Haar wavelet consists of two mirror operations: wavelet pooling and wavelet unpooling.
The former converts images into the hlwavelet domain, and the latter inversely reconstructs frequency components into the spatial domain.
There are four kernels in wavelet pooling operation: LL T , LH T , HL T , HH T , where L T = 1 √ 2 [1 , 1] , H T = 1 √ 2 [ − 1 , 1] , L and H denotes the low and high pass ﬁlters, respectively.
The low (L) pass ﬁlter captures the outline and surface of images, while the high (H) pass ﬁlter focuses on detailed information like the edges and delicate textures.
Fig.
3 illustrates the obtained frequency components of given images via Haar wavelet.
We can observe that low-frequency component LL contains the overall surface of images, while components that are decomposed by high pass ﬁlters, i.e., LH, HL, HH , contain more ﬁne details.
Further, by summing up the three high-frequency components, we approximately obtain all details information of images, e.g. , the eyes of the cat and the teeth of Obama.
3.2 High-Frequency Discriminator
To raise the frequency awareness of D, we devise high-frequency discriminator (HFD).
HFD is responsible for distinguishing the real images from the generated images from the perspective of the frequency domain.
Formally, for the i -th layer in the discriminator, we adopt wavelet pooling on the intermediate features and obtain LL iD , LH iD , HL iD , HH iD , then we combine the three highfrequency components by tensor addition, i.e. , HF iD =
LH iD + HL iD + HH iD , which contains sufﬁcient details of features.
By applying traditional convolution and downsampling operations following the original discriminator, we deﬁne the adversarial loss of our HFD as:
[Equation]
where HF real and HF fake are the high-frequency information of real and fake images, respectively.
D H is the high-frequency discriminator.
Since the high-frequency information may be eschewed by
D as the network goes deeper, we perform multi-scale HFD on multi-layers of the discriminator.
The multi-scale operation ensures fully mine and exploit the frequency information of limited data, which further improves D’s frequency awareness.
Notably, being guided by the HFD with Eq. 2, G is also optimized to produce rich high-frequency details.
3.3 Frequency Skip Connection
The generator is capable of producing plausible frequency signals after employing HFD (see Tab. 5).
However, as GANs ﬁt frequency signals from low to high and the high-frequency signals may be ignored as the network goes deeper.
To prevent the loss of high-frequency information and further encourage the generator to produce rich details, we propose frequency skip connection (FSC).
Concretely, we utilize wavelet unpooling operation of the frequency components LL iG , LH iG , HL iG , HH iG obtained from wavelet transformation on the features of G’s i -th layer, which reconstructs the highfrequency representation to the original features.
Then we explicitly feed the reconstructed frequency representations to the next layer of G. Formally,
[Equation]
where F i denotes the features of the i -th layer and Unpooling is the wavelet unpooling operation.
F (cid:48)
i +1 is the obtained features after FSC, which will be fed into the subsequent layer for further operation.
Such skip connection prevents loss of high-frequency information and maintains highfrequency details.
3.4 High-Frequency Alignment
Adding HFD and FSC explicitly raises the frequency awareness of G, but G can only synthesize arbitrary frequency signals.
How G can utilize the frequency signals is still ambiguous, and D still dominates the competition since it learns discriminative knowledge from both real and generated images.
To balance the unhealthy competition between G and D, we propose high-frequency alignment (HFA), which involves high-frequency signals of real images induced from D as a regularizer to guide G, promoting G to synthesize more reasonable and realistic ﬁne details.
Speciﬁcally, we extract the frequency representations of intermediate features of G at different layers.
For the i -th layer of G, we obtain frequency components LL iG , LH iG , HL iG , HH iG .
We ignore LL iG and combine the three high-frequency components, i.e. , HF iG =
LH iG + HL iG + HH iF .
Then we use the high-frequency components of the discriminator HF iD as a self-supervision constraint.
In addition to fool D, G is expected to minimize the distance of high-frequency information between the generated and real images.
The alignment loss is deﬁned as:
[Equation]
where (cid:107) ∗ (cid:107) 1 denotes the L 1 -norm.
Such alignment encourages G to synthesis frequency signals that approach real frequency signals, mitigating the unhealthy competition and facilitating generation quality.
To take full advantage of frequency signals of real images from D, we perform HFA on multiscale features like HFD as shown in Fig.
The ablative experiment results in Sec.
4.2 demonstrate the rationality and effectiveness of employing HFA and HFD on multi-scale features.
3.5 Optimization
Following [24], we adopt the hinge version of adversarial loss to train our model.
[Equation]
We also use the reconstruction loss [24] to encourage the discriminator to extract more representative features.
[Equation]
(7) where f is the intermediate features of D, G and T denote the processing on the features f and the input images x .
In sum, our discriminator is optimized by L D , L recons , and L HFD .
Our generator is optimized by L G , L HFG and L align , the coefﬁcient of each loss is set to 1 .
Our FreGAN improves the overall quality of generated images and raises the model’s frequency awareness, encouraging the generator to produce precise high-frequency signals with ﬁne details.
Experiments
Datasets.
We test the effectiveness of our method on low-shot datasets from various domains with different resolutions.
On 256 × 256 resolution, we use Animal Face Dogs and Cat [33], as well as 100-shot-Panda, Obama, and Grumpy_cat [51].
On 512 × 512 resolution, we use Anime-Face,
Art Paintings, Moongate, Flat-colored, and Fauvism-still-life [24].
On 1024 × 1024 resolution, we use Pokemon, Skulls, Shells, MetFace [16] and BrecaHAD [1].
These datasets contain a limited number of samples (mostly less than 1,000) and cover art paintings, realistic photos, human faces, etc.
For datasets that are not strictly equal to the corresponding resolution, we resize them to the closest resolution in implementation.
Besides, we use AnimalFace HQ (AFHQ) datasets [5] to evaluate the performance of our model when training with more data ( ∼ 5k).
Evaluation metrics and baseline.
We adopt two common metrics to evaluate the synthesize quality:
Fréchet Inception Distance (FID) [12] and Kernel Inception Distance (KID) [3].
The lower FID and
KID is, the better the generation quality is.
FID quantiﬁes the distance between the distribution of the generated and the real images.
KID, which is designed unbiased, has been proven more descriptive for small datasets [16], note that all KID scores reported in our paper need to × 10 − 3 following [16].
Following [24], we calculate FID and KID by measuring the distance between all available training images and 5k generated images.
We also provide the LPIPS [50], IS [30], Precision, Recall [21],
Density, Coverage [26] results in the appendix.
We compare our model with: 1) the state-of-the-art generative model StyleGAN2 [19], and SWAGAN [8], which incorporates wavelet into StyleGAN2; 2) data augmentation-based approaches that is designed for training GANs with limited data, i.e. , ADA [16], DiffAug [51], APA [14]; 3) the state-of-the-art few-shot generative model FastGAN [24].
We reimplement all baselines with their released ofﬁcial code under consistent settings for a fair comparison.
Implementation details of baseline models are given in the appendix.
Implementation details.
We choose the current state-of-the-art few-shot generative model FastGAN [24] as the backbone and implement our proposed techniques upon it.
all other settings remain the same as [24].
We decompose the intermediate 8 × 8, 16 × 16, 32 × 32 features of G and D into frequency components for our frequency skip connection, high-frequency discriminator, and high-frequency alignment.
More implement details are given in the appendix.
4.1 Main Results
Quantitative comparison on datasets with limited data amounts.
The quantitative comparison results of our FreGAN and baseline methods on different resolutions are given in Tab. 1, Tab. 2 and
Tab. 3.
We save the best training snapshots of each method and generate 5k images to compute FID and KID.
The whole training set is adopted as the referenced distribution.
We can observe from the
[Table] [Table]
[Table]
results that, although evaluated on various datasets that have different resolutions and data amounts, our proposed FreGAN achieves superior performance on all these datasets.
Our FreGAN consistently improves both FID and KID metrics on 14 of the 15 datasets, demonstrating the effectiveness and generalizability of our proposed techniques.
Notably, for those datasets with extremely limited data (less than 100), i.e. , Flat (Tab. 2), Shells and Skulls (Tab. 3), our method improves the performance more signiﬁcantly, e.g. , the FID from 216.27 to 178.10 on Flat and from 101.94 to 86.12 on Skulls, and the corresponding KID is improved doubled, further reﬂecting our model’s potential for training
GANs with extremely limited data.
More quantitative results are presented in the appendix.
Qualitative Comparison.
The qualitative results of FastGAN and our FreGAN on various datasets are illustrated in Fig.
For each dataset in Fig. 4, from left to right are generated images, the visualization of the low and high-frequency components of the generated images.
The images generated by FastGAN contain unsatisfactory artifacts and some of them are incongruous, e.g. , the generated images of cat and dog in the bottom right of Fig.4, the cat has artifacts around the head, and the dog’s ears are distorted.
Our FreGAN signiﬁcantly facilitates image quality in coordination, rationality, and ﬁne details.
As can been seen from Fig.
4, the human face Obama generated by our
FreGAN is more photorealistic, the details of the anime face, such as eye color and hair texture, are more realistic, and the synthesized animal faces of cats and dogs are also more plausible.
Besides, the frequency components of the images generated by our FreGAN contain wealthier details.
For example, the generated image of AnimalFace-Cat has a richer background, and the generated image of Skulls has more clear contours of the eye and nose.
Such observation reﬂects that the proposed
FreGAN: 1) ameliorates the quality of generated images under limited data; 2) raises frequency awareness of synthesizing high-frequency signals with richer ﬁne details of images; and 3) takes full advantage of limited data’s frequency information.
More qualitative results are given in the appendix.
[Table] [Table]
Effectiveness under datasets with more data.
To investigate the effectiveness of our FreGAN more comprehensively, we evaluate the performance on datasets with more training data, i.e. , AnimalFaceHQ (AFHQ) [5], which includes 3 sub-datasets with close to 5k images, the results are shown in
Tab. 4.
Similarly, our method yields compelling improvements on both FID and KID metrics when training with more data.
Combined with the generated images in Fig.
4, the results further validate our FreGAN’s contribution to the synthesize quality.
Our method boosts the performance under different amounts of data, suggesting the generalization of our model.
4.2 Ablation Studies
Ablation studies on variants of FreGAN.
There are three ingredients of our FreGAN, i.e. , the highfrequency discriminator (HFD), high-frequency alignments (HFA), and frequency skip connection (FSC).
We evaluate the efﬁcacy of each component by removing each of them from the full version of our FreGAN.
We choose one from each of the different resolution datasets, i.e. , 100-shot-Obama,
Anime face and pokemon for 256, 512 and 1024 resolution, respectively.
As shown in Tab. 5, removing any of the three techniques leads to a performance drop, reﬂecting the contribution of each component.
Still, all these variants outperform baseline FastGAN on both FID and KID, which implies that the combination of different components of our method consistently boosts model performance.
Moreover, the performance drops the most when removing the HFD module, which is reasonable because the HFD raises the frequency awareness of G and D, and the frequency awareness of D serves as a self-supervision to guide G to synthesize adequate and reasonable frequency signals.
Qualitative comparison results of ablation studies are given in the appendix.
Ablation studies on different scale of features.
We employ our proposed HFD and HFA on multiscale features of G and D, namely, 8, 16, and 32 scales of features.
Here we provide the ablation studies on different scales in Tab. 6.
It can be seen that performing HFD and HFA on multi-scale features boosts the model performance.
Besides, when only performing HFD and HFA on single-scale features, the obtained results still outperform the FastGAN baseline, suggesting the effectiveness of HFA and HFD.
Notably, despite adding more scales of features may bring further performance advancement, the required additional convolutional and downsampling layers increases for higher scales features( e.g. , 128, 256), bringing non-negligible computational costs.
Ablation studies on different frequency components.
Three high-frequency components are obtained from wavelet transformation on the features, i.e. , LH , HL , and HH .
Each of them encodes different details of features as shown in Fig.
3, we sum them to fuse all the detail information for further operation in our main experiments.
Here we conduct experiments on the three components respectively to verify their contribution and the necessity of fusing them.
As shown in Tab. 6, each high-frequency component contributes to the model performance compared with the baseline, and fusing them can better promote the generation quality.
[Table] [Table]
4.3 Analysis on Compatibility and GAN Equilibrium
Compatibility of Our Model.
Lecam [37] and MoCA [22] exploit regularization and attention mechanism for training GANs under limited data, respectively.
We implement our proposed techniques on them to test the compatibility of our method.
We keep the original setting unchanged and the set the regularization weight to 0.1.
The FID results are given in Tab. 7, from which we can see that FreGAN can further boost the performance of MoCA and Lecam, demonstrating that our method is complementary to the model regularization and attention mechanism methods.
GAN Equilibrium is improved.
Our HFA module aligns the frequency components of real and generated images, guiding G to synthesize precise instead of arbitrary high-frequency signals.
Meanwhile, as a byproduct, the HFA mitigates the domain gap between G and D, alleviating the unhealthy competition.
As shown in Fig.
5 (a), our discriminator converges to a better point, and our generator can better fool the discriminator, while the discriminator of FastGAN surpass the generator, thus providing less informative guidelines and degrading the synthesize quality.
Besides, we plot the FID and KID curves throughout the training process in Fig.
5 (b), from which we can observe that
our
FreGAN are consistently better.
Moreover, we plot the multi-scale HFA loss curves in Fig.
5 (c), where each line denotes the loss of each scale.
These curves indicate that the frequency signals are well aligned, lessening the domain gaps and promoting the GAN equilibrium.
Discussion
Conclusions.
In this paper, we propose a frequency-aware method for training GANs under limited data, i.e. , FreGAN.
The proposed FreGAN ameliorates the synthesize quality by raising the model’s frequency awareness, encouraging the model to pay more attention to frequency signals, especially high-frequency signals, which encode ﬁne details of images.
We conduct extensive experiments on various datasets with different amounts of data and different resolutions to demonstrate the efﬁcacy of our proposed method.
Qualitative results suggest that our model successfully makes the generator to generate precise high-frequency signals, facilitating high-quality image generation.
Quantitative results indicate that our method 1) substantially boosts the performance, especially when data is
Our generator can better deceive the discriminator.
(b) Our FreGAN yields better performance throughout the training process.
(c)
The aligned HFA loss mitigates the asymmetrical information and facilitates the GAN equilibrium.
extremely limited (less than 100), and 2) is complementary to existing regularization and attention models.
Moreover, the proposed model alleviates the disequilibrium of GANs by lessening the frequency information gap.
In the future, we plan to implement our techniques on more backbones, e.g. , StyleGAN2 [19] and apply our method to more applications.
Limitations.
Despite achieving signiﬁcant improvements on various low-data datasets, our FreGAN still struggles in generating photorealistic images when given datasets with limited data but various contents, e.g. , only dozens of images, and their contents vary widely.
When the low-data datasets are imbalanced [43] or even long-tailed, the proposed method may fail to generalize, which is limited by the intrinsic reasons of the data distribution.
Developing more effective ways to train generative models with insufﬁcient training data still requires more efforts.
Acknowledgment
This work is supported by Shanghai Science and Technology Program “Distributed and generative few-shot algorithm and theory research” under Grant No. 20511100600 and “Federated based cross-domain and cross-task incremental learning” under Grant No. 21511100800, Natural Science
Foundation of China under Grant No. 62076094, Chinese Defense Program of Science and Technology under Grant No.2021-JCJQ-JJ-0041, China Aerospace Science and Technology Corporation
Industry-University-Research Cooperation Foundation of the Eighth Research Institute under Grant
No.SAST2021-007.