  Training generative adversarial networks (GANs) with limited data generally
results in deteriorated performance and collapsed models. To conquer this
challenge, we are inspired by the latest observation of Kalibhat et al. (2020);
Chen et al.(2021d), that one can discover independently trainable and highly
sparse subnetworks (a.k.a., lottery tickets) from GANs. Treating this as an
inductive prior, we decompose the data-hungry GAN training into two sequential
sub-problems: (i) identifying the lottery ticket from the original GAN; then
(ii) training the found sparse subnetwork with aggressive data and feature
augmentations. Both sub-problems re-use the same small training set of real
images. Such a coordinated framework enables us to focus on lower-complexity
and more data-efficient sub-problems, effectively stabilizing training and
improving convergence. Comprehensive experiments endorse the effectiveness of
our proposed ultra-data-efficient training framework, across various GAN
architectures (SNGAN, BigGAN, and StyleGAN2) and diverse datasets (CIFAR-10,
CIFAR-100, Tiny-ImageNet, and ImageNet). Besides, our training framework also
displays powerful few-shot generalization ability, i.e., generating
high-fidelity images by training from scratch with just 100 real images,
without any pre-training. Codes are available at:
https://github.com/VITA-Group/Ultra-Data-Efficient-GAN-Training.
