Distilling Representations from GAN Generator via Squeeze and Span
Abstract
In recent years, generative adversarial networks (GANs) have been an actively studied topic and shown to successfully produce high-quality realistic images in various domains.
The controllable synthesis ability of GAN generators suggests that they maintain informative, disentangled, and explainable image representations, but leveraging and transferring their representations to downstream tasks is largely unexplored.
In this paper, we propose to distill knowledge from GAN generators by squeezing and spanning their representations.
We squeeze the generator features into representations that are invariant to semantic-preserving transformations through a network before they are distilled into the student network.
We span the distilled representation of the synthetic domain to the real domain by also using real training data to remedy the mode collapse of GANs and boost the student network performance in a real domain.
Experiments justify the efﬁcacy of our method and reveal its great signiﬁcance in self-supervised representation learning.
Code is available at https://github.com/yangyu12/squeeze-and-span .
Introduction
Generative adversarial networks (GANs) [23] continue to achieve impressive image synthesis results thanks to large datasets and recent advances in network architecture design [5, 36, 37, 34].
GANs synthesize not only realistic images but also steerable ones towards speciﬁc content or styles [22, 52, 49, 33, 57, 53, 32].
These properties motivate a rich body of works to adopt powerful pretrained GANs for various computer vision tasks, including part segmentation [68, 56, 61], 3D reconstruction [67], image alignment [48, 45], showing the strengths of GANs in the few-label regime.
GANs typically produce ﬁne-grained, disentangled, and explainable representations, which allow for higher data efﬁciency and better generalization [42, 68, 56, 61, 67, 48].
Prior works on GAN-based representation learning focus on learned features from either a discriminator network [50] or an encoder network mapping images back into the latent space [19, 17, 18].
However, there is still inadequate exploration about how to leverage or transfer the learned representations in generators .
Inspired by the recent success of [68, 56, 61], we hypothesize that representations produced in generator networks are rich and informative for downstream discriminative tasks.
Hence, this paper proposes to distill representations from feature maps of a pretrained generator network into a student network (see Fig. 1).
In particular, we present a novel “squeeze-and-span” technique to distill knowledge from a generator into a representation network 2 that is transferred to a downstream task.
Unlike transferring discrimi-
[Figure]
nator network, generator network is not directly transferable to downstream image recognition tasks, as it cannot ingest image input but a latent vector.
Hence, we distill generator network representations into a representation network that can be further transferred to the target task.
When fed in a synthesized image, the representation network is optimized to produce similar representations to the generator network’s.
However, the generator representations are very high-dimensional and not all of them are informative for the downstream task.
Thus, we propose a squeeze module that puriﬁes generator representations to be invariant to semantic-preserving transformations through an MLP and an augmentation strategy.
As the joint optimization of the squeeze module and representation network can lead to a trivial solution ( e.g . mapping representations to zero vector), we employ variance-covariance regularization in [3] while maximizing the agreement between the two networks.
Finally, to address the potential domain gap between synthetic and real images, we span the learned representation of synthetic images by training the representation network additionally on real images.
We evaluate our distilled representations on CIFAR10, CIFAR100 and STL10 with linear classiﬁcation tasks as commonly done in representation learning.
Experimental results show that squeezing and spanning generator representations outperforms methods that build on discriminator and encoding images into latent space.
Moreover, our method achieves better results than discriminative SSL algorithms, including SimSiam [10] and VICReg [3] on CIFAR10 and CIFAR100, and competitive results on STL10, showing signiﬁcant potential for transferable representation learning.
Our contributions can be summarized as follows:
We (1) provide a new taxonomy of representation and transfer learning in generative adversarial networks based on the location of the representations, (2) propose a novel “squeeze-and-span” framework to distill representations in the GAN generator and transfer them for downstream tasks, (3) empirically show the promise of utilizing generator features to beneﬁt self-supervised representation learning.
Related Work
GANs for Representation Learning.
Signiﬁcant progress has been made on the interpretability, manipulability, and versatility of the latent space and representation of GANs [36, 37, 34, 35].
It inspires a broad spectrum of GAN-based applications, such as semantic segmentation [68, 56, 61], visual alignment [48, 45], and 3D reconstruction [67], where GAN representations are leveraged to synthesize supervision signals efﬁciently.
As GAN can be trained unsupervised, its representations are transferred to downstream tasks.
DCGAN [50] proposes a convolutional GAN and uses the pre-trained discriminator for image classiﬁcation.
BiGAN [17] adopts an inverse mapping strategy to transfer the real domain knowledge for representation learning.
While ALI [19] improves this idea with a stochastic network instead of a deterministic one, BigBiGAN [18] extends BiGAN with
BigGAN [5] for large scale representation learning.
GHFeat
[59] trains a post hoc encoder that maps given images back into style codes of style-based GANs [36, 37, 35] for image representation.
These works leverage or transfer representations from either discriminators or encoders.
In contrast, our method reveals that the generator of a pre-trained GAN is typically more suitable for representation transfer with a proper distillation strategy.
[Figure]
Knowledge Distillation (KD) aims at training a small student network, under the supervision of a relatively large teacher network [31].
In terms of the knowledge source, it can be broadly divided into logit-based KD and feature-based KD.
Logit-based KD methods [41, 60, 12] optimize the divergence loss between the predicted class distributions, usually called logits or soft labels, of the teacher and student network.
Feature-based KD methods [38, 2, 54] adopt the teacher model’s intermediate layers as supervisory signals for the student.
FitNet [51] introduces the output of hidden layers of the teacher network as supervision.
AT [63] proposes to match attention maps between the teacher and student.
FSP [62] calculates ﬂow between layers as guidance for distillation.
Likewise, our method distills knowledge from intermediate layers from a pre-trained GAN generator.
Self-Supervised Representation Learning (SSL) pursues learning general transferable representations from unlabelled data.
To produce informative self-supervision signals, the design of handcrafted pretext tasks has ﬂourished for a long time, including jigsaw puzzle completion [46], relative position prediction [15, 16], rotation perception [21], inpainting [47], colorization [40, 65], masked image modeling [27, 58], etc .
Instead of performing intra-instance prediction, contrastive learning-based
SSL methods explore inter-instance relation.
Applying the InfoNCE loss or its variants [26], they typically partition informative positive/negative data subsets and attempt to attract positive pairs while repelling negative ones.
MoCo series [28, 8, 11] introduce an ofﬂine memory bank to store large negative samples for contrast and a momentum encoder to make them consistent.
SimCLR [7] adopts an end-to-end manner to provide negatives in a mini-batch and introduce substantial data augmentation and a projection head to improve the performance signiﬁcantly.
Surprisingly, without negative pairs, BYOL [25] proposes a simple asymmetry SSL framework with the momentum branch applying the stop gradient to avoid model collapse.
It inspires a series of in-deep explorations, such as SimSiam [9], Barlow Twins [64], VICReg [3], etc .
In this paper, despite the same end goal of obtaining transferable representations and the use of techniques from VICReg [3], we study the transferability of generator representations in pretrained GANs to discriminative tasks, use asymmetric instead of siamese networks, and design effective distillation strategies.
Rethinking GAN Representations
Let G : W → X denote a generator network that maps a latent variable in W to an image in X .
An unconditional GAN trains G adversarially against a discriminator network D : X → [0 , 1] that estimates the realness of the given images,
[Equation]
The adversarial learning does not require any human supervision and therefore allows for learning representations in an unsupervised way.
In this paper, we show that the type of GAN representations and how they are obtained has a large effect on their transferability.
In Fig.
2, we illustrate three different representations from an unconditional GAN, which is trained on CIFAR10, by using 2D
UMAP embeddings [44] 3 .
Note that we describe each representation in the following paragraphs.
Discriminator Feature
The discriminator D , which is tasked to distinguish real and fake images, can be transferred to various recognition tasks [50].
Formally, let D =
d ( L ) ◦
d ( L − 1) ◦ · · · ◦ d (1) denote the decomposition of a discriminator into L consecutive layers.
As shown in Fig. 1(a), given an image x , the discriminator representation can be extracted by concatenating the features after average pooling from each discriminator block output,
[Equation]
[Equation]
[Equation]
h
[Equation]
where µ denotes the average pooling operator.
However, Fig. 2(a) shows that the cluster of discriminator features is not signiﬁcantly correlated with class information indicating that real/fake discrimination does not necessarily relate to class separation.
Latent Variable An alternative way of transferring GAN representation is through its latent variable w [19, 17, 18].
Latent variables are assumed to observe simple statistic distribution, e.g . normal distribution, and can be transformed with a given generator into an image.
This suggests that latent variables represent and depict statistical characteristics of images.
Moreover, works on steerable GANs show that speciﬁc dimensions in latent variables correspond to interpretable data variation, suggesting latent variable is an informative representation source.
To extract a latent variable representation of an image, one can invert the generator with an encoder.
While some works integrate training encoder into adversarial learning [19, 17, 18], we consider training a post hoc encoder [6] given a ﬁxed pre-trained generator G for comparison in a uniﬁed setting,
[Equation]
where L percep denotes the LPIPS loss [66] and λ = 1 . 0 is used to balance different loss terms.
Fig. 2(b) visualizes the embedding of latent variables 4 .
It shows the class information in latent variables is entangled despite they contain information on image synthesis factors.
Generator Feature An overlooked practice is to utilize generator features.
Typically, GAN generators transform a low-resolution ( e.g . 4 × 4) feature map to a higher-resolution one ( e.g . 256 × 256) and further synthesize images from the ﬁnal feature map [17, 36] or multi-scale feature maps [37].
The image synthesis is performed hierarchically: feature map from low to high resolution encodes the low-frequency to high-frequency component for composing an image signal [35].
This understanding is also evidenced by image editing works [22, 52, 49, 53, 32] which show that interfering with low-resolution feature maps leads to a structural and high-level change of an image, and altering high-resolution feature maps only induces subtle appearance changes.
Therefore, generator features contain valuable hierarchical knowledge about an image.
Formally, let G
= g ( L ) ◦ g
( L − 1) ◦ · · · ◦ g (1) denote the decomposition of a discriminator into L consecutive layers.
Given a latent variable w ∼ P ( w ) drawn from a prior distribution, we consider the concatenated features average pooled from each generator block output, h g =
[ µ ( h g 1 ) , . .
, µ ( h g L )] , where h g i =
g ( i ) ◦ · · · ◦ g (1) ( w ) .
As Fig. 2(c) shows, generator features within the same class are naturally clustered.
This result suggests that generators contain identiﬁable representations that can be transferred for downstream tasks.
However, as GANs do not initially provide a reverse model for the accurate recovery of generator features, it is still inconvenient to extract generator features for any given image.
This limitation motivates us to distill the valuable features from GAN generators.
Squeeze-and-Span Representations from GAN Generator
This section introduces the “Squeeze-and-Span” technique to distill representation from GANs into a student network, which can then be readily transferred for downstream tasks, e.g . image classiﬁcation.
Let S θ : X → H denote a student network that maps a given image into representation space
A 𝐰~𝑷(𝐰)
Synthesis4x4Synthesis4x
Synthesis8x8Synthesis8x
Synthesis16x16Synthesis16x
Synthesis 32x32 Synthesis 32x toRGBtoRGB
M L P toRGBtoRGB toRGBtoRGB toRGBtoRGB 𝜇 & FC 𝜇 & FC
Const
UpUp UpUp
UpUp 𝜇 & FC 𝜇 & FC 𝜇 & FC 𝜇 & FC 𝜇 & FC 𝜇 & FC 𝐱
𝐠 Squeeze module 𝑇 𝜙
Generator 𝐺 𝑮 𝑻 𝝓 𝑻 𝝓 𝑺 𝜽 𝓛 𝐯𝐚𝐫 & 𝓛 𝐜𝐨𝐯
𝐱 𝐠 𝒂 𝐱 𝐠 𝒂 𝐱 𝐫 𝒂′ 𝐱 𝒓 𝓛 𝐯𝐚𝐫 & 𝓛 𝐜𝐨𝐯 𝓛 𝐯𝐚𝐫 & 𝓛 𝐜𝐨𝐯 𝓛 𝐯𝐚𝐫 & 𝓛 𝐜𝐨𝐯 𝓛 𝐑𝐃′ 𝓛 𝐑𝐃 𝐰~𝑷(𝐰)
Teacher
Student
Squeeze
Span
Teacher naive way of representation learning can be achieved by tasking the student network to predict the teacher representation, which can be formulated as the following optimization problem:
[Equation]
(5) where we use mean squared error to measure the prediction loss and h g ( w ) to denote the dependence of h g on w .
However, this formulation has two problems.
First, representations extracted through multiple layers of the generator are likely to contain signiﬁcantly redundant information for downstream tasks but necessary for image synthesis.
Second, as the student network is only optimized on synthetic images, it is likely to perform poorly in extracting features from real images in the downstream task due to the potential domain gap between real and synthetic images.
To mitigate these issues, we propose the “Squeeze and Span” technique as illustrated in Fig.
4.1 Squeezing Informative Representations
To alleviate the ﬁrst issue that generator representation may contain a big portion of irrelevant information for downstream tasks, we introduce a squeeze (or bottleneck) module T φ (Fig. 3) that squeezes informative representations out of the generator representation.
In addition, we transform the generated image via a semantic-preserving image transformation a ( e.g . color jittering and cropping) before feeding it to the student work.
Equ. 5 can be rewritten as
[Equation]
where image transformation a is randomly sampled from A .
In words, we seek to distill compact representations from the generator among the ones that are invariant to data augmentation A , inspired from the success of recent self-supervised methods [7, 10].
An informal intepretation is that, similar to Chen & He [10], considering one of the alternate subproblems that ﬁx θ and solve φ , the optimal solution would result in the effect of T φ ∗ ( h g ( w ))
≈ E a ∼A S θ ( a [ G ( w )]) , which implies Equencourages T φ to squeeze out transformation-invariant representation.
However, similar to the siamese network in SSL [10], there exists a trivial solution to Equ. 6: both the squeeze module and the student network degenerate to output constant for any input.
Therefore, we consult the techniques from SSL methods and add regularization terms to the distillation loss.
In particular, we employ variance-covariance [3] to explicitly regularize representations to be signiﬁcantly uncorrelated and varied in each dimension.
Formally, in a mini-batch of N samples, we denote the squeezed generator representations and student representations with
[Equation]
1 )]) , S θ ( a 2 [ G ( w 2 )]) , . . .
, S θ ( a N [ G ( w N )])]
∈ R M × N , (8) where w i ∼ P ( w ) and a i ∼ A denote random sample of latent variable and data augmentation operator.
The variance loss is introduced to encourage the standard deviation of each representation dimension to be greater than 1 ,
[Equation]
where z j represents the j -th dimension in representation z .
The covariance loss is introduced to encourage the correlation of any pair of dimensions to be uncorrelated,
[Equation]
To this end, the loss function of squeezing representations from the generator into the student network can be summarized as
[Equation]
Discussion Our work differs from multi-view representation learning methods [3, 10] in the following aspects.
(1) Our work studies the transfer of the generative model that does not originally favor representation extraction, whereas most multi-view representation learning learns representation with discriminative pretext tasks.
(2) Unlike typical Siamese networks in multi-view representation learning, the two networks in our work are asymmetric: one takes in noise and outputs an image and the other works in the reverse fashion.
(3) While most multi-view representation learning methods learn representation networks from scratch, our work distills representations from a pre-trained model.
In speciﬁc, most SSL methods create multiview representations by transforming input images in multiple ways, we instead pursue different representation views from a well-trained data generator.
4.2 Spanning Representations from Synthetic to Real Domain
Here we address the second problem, the domain between synthetic and real domains, due to two factors.
First, the synthesized images may be of low quality.
This aspect has been improved a lot with recent GAN modelling [37, 35] and is out of our concern.
Second, more importantly, GAN is notorious for the mode collapse issue, suggesting the synthetic data can only cover partial modes of real data distribution.
In other words, the synthetic dataset appears to be a subset of the real dataset.
To undermine the harm of mode collapse, we include the real data in the training data of the student network.
In particular, in each training step, synthetic data and real data consist of a mini-batch of training data.
For synthetic data, the aforementioned squeeze loss is employed.
For real data, we employ the original VICReg to compute loss.
Speciﬁcally, given a mini-batch of real data { x ri } Ni =1 , each image x r i is transformed twice with random data augmentation to obtain two views a i ( x r i ) and a (cid:48)
i ( x ri ) , where a i , a (cid:48)
i ∼ A .
The corresponding representations Z r and Z (cid:48) r are obtained by feeding the transformed images into S θ similarly to Equ. 8.
Then the loss on real data is computed as
[Equation]
where L (cid:48)
RD denotes a self-distillation by measuring the distance of two-view representations on real images.
The overall loss is computed by simply combine the generated data loss and real data loss as L total =
α L squeeze + (1 − α )
L span , where α = 0 .
5 denotes the proportion of synthetic data in a mini-batch of training samples.
From a technical perspective, spanning representation seems to be a combination of representation distillation and SSL using VICReg [3].
We interpret this combination as spanning representation from the synthetic domain to the real domain.
The representation is dominantly learned in the synthetic domain and generalized to the real domain.
The student network learns to fuse representation spaces of two domains into a consistent one in the spanning process.
Our experimental evaluation shows that “squeeze and span” can outperform VICReg on real data, suggesting that the squeezed representations do have a nontrivial contribution to the learned representation.
Experiments 5.1 Setup
Dataset and pre-trained GAN Our methods are mainly evaluated on CIFAR10, CIFAR100, and STL10, ImageNet100, and ImageNet.
CIFAR10 and CIFAR100 [39] are two image datasets containing small images at 32 × 32 resolution with 10 and 100 classes, respectively, and both split into 50,000 images for training and 10,000 for validation.
STL-10 [13], which is derived from the
ImageNet [14], includes images at 96 × 96 resolution over 10 classes.
STL-10 contains 500 labeled images per class ( i.e . 5K in total) with an additional 100K unlabeled images for training andlabeled images for testing.
ImageNet100 [55] contains images of 100 classes, among which 126, images are regarded as the train split and 5,000 images are taken as the validation split.
ImageNet [14] is a popular large-scale image dataset of 1000 classes, which is split into 1,281,167 images as training set and 50,000 images as validation set.
We adopt StyleGAN2-ADA 5 for representation distillation since it has good stability and high performance.
GANs are all pre-trained on training split.
More details can be referred in the supplementary material.
Implementation details The squeeze module uses linear layers to transform the generator features into vectors with 2048 dimensions, which are then summed up and fed into a three-layer MLP to get a 2048-d teacher representation.
On CIFAR10 and CIFAR100, we use ResNet18 [30] of the
CIFAR variant as the backbone.
On STL10, we use ResNet18 as the backbone.
On ImageNet100 and
ImageNet, we use ResNet50 as the backbone.
On top of the backbone network, a ﬁve-layer MLP is added for producing representation.
We use SGD optimizer with cosine learning rate decay [43] scheduler to optimize our models.
The actual learning rate is linearly scaled according to the ratio of batch size to 256, i.e . base_lr × batch_size / 256 [24].
We follow the common practice in
SSL [7, 55, 29] to evaluate the distilled representation with linear classiﬁcation task.
More details are available in the supplementary material.
5.2 Transferring GAN Representation
Compared methods In this section, we justify the advantage of distilling generator representations by comparing the performance of different ways of transferring GAN representation.
In particular, we consider the following competitors:
• Discriminator .
As the discriminator network receives image as input and is ready for representation extraction, we directly extract features, single penultimate features, or multiple features (Equ 2), using a pre-trained discriminator and train a linear classiﬁer on top of them.
• Encoding .
We train a post hoc encoder with or without real images involved in the training process as in Equ 3.
• Distilling latent variable .
We employ the vanilla distillation or squeeze method on latent variables with data augmentation engaged.
• Distilling generator feature .
Our method as described in Section 4.
Results Table 1 presents the comparison results, from which we can draw the following conclusions.
(1) Representation distillation, whether from the latent variable or generator feature, signiﬁcantly outperforms discriminator and encoding.
We think this is because image reconstruction and realness discrimination are not suitable pretext tasks for representation learning.
(2) Distillation from latent variable achieve comparable performance to distillation from generator feature, despite that the former one show entangled class information (Fig. 2).
This result can be attributed to a projection head in the student network.
(3) Our method works signiﬁcantly better than vanilla distillation which does not employ a squeeze module.
This result suggests that our method squeeze more informative representation that can help to improve the student performance.
5.3 Comparison to SSL
Linear classiﬁcation We further compare our methods to SSL algorithms such as SimSiam [10] and VICReg [3] in different training data domains: real, synthetic, and a mixture of real and synthetic.
[Table] [Table]
Table 2 presents the linear classiﬁcation results, from which we want to highlight the following points
(1) Both SimSiam and VICReg perform worse when pre-trained on only synthetic data than only real data, indicating the existence of a domain gap between synthetic data and real data.
(2) Our methods outperform SimSiam and VICReg in synthetic and mixture domains, suggesting distillation of generator feature contributes extra improvement SSL.
(3) Our "Squeeze and Span" is the best among all the competitors on CIFAR10, CIFAR100, and STL10.
(4) Our method outperforms VICReg with a large margin (6.90% Top-1 Acc) on ImageNet100 and a clear increase (0.48% TopAcc) on ImageNet.
Transfer learning As one goal of representation learning is its transferability to other datasets, we further conduct a comprehensive transfer learning evaluation.
We follow the protocol in [20] and use its released source code 6 to conduct a thorough transfer learning evaluation for our pre-trained models on ImageNet100/ImageNet.
In particular, the learned representations are mainly evaluated for (1) linear classiﬁcation on 11 datasets including Aircraft, Caltech101, Cars, CIFAR10, CIFAR100, DTD,
Flowers, Food, Pets, SUN397, and VOC2007; (2) ﬁnetuning on three downstream tasks and datasets, including object detection on PASCAL VOC, surface normal estimation on NYUv2, and semantic segmentation on ADEChallenge2016.
Please refer to [20] for the details of evaluation protocol.
The results are presented in Table 3 and Table 4, where we have the following observation (1) As depicted in Table 3, our method achieves better transferability than VICReg on the mixed data no matter pre-trained on ImageNet100 or ImageNet. Our method beats VICReg on nearly all other datasets and the improvement on average accuracy is 3.40 with models pre-trained on ImageNet and 1.00 with models pre-trained on ImageNet. (2) As depicted in Table 4, representations learned with our method can be well transferred to various downstream tasks such as object detection, surface normal estimation and semantic segmentation, and consistently show higher performance than VICReg. [Table]
[Table] [Figure] [Table] We believe these results suggest that generator features have strong transferability and great promise to contribute to self-supervised representation learning.
5.4 Ablation Study
Effect of squeeze and span The effect of our method is studied by adding modules to the vanilla version of representation distillation (a) one by one.
(a) → (b): After added data augmentation, signiﬁcant improvement can be observed, suggesting that invariant representation to data augmentation is crucial for linear classiﬁcation performance.
This result inspires us to make teacher representation more invariant.
(b) → (c): the learnable T φ is introduced to squeeze out invariant representation as teacher.
However, trivial performance (10% top-1 accuracy, no better than random guess) is obtained, implying models learn trivial solutions, probably constant output.
(c) → (d) & (e): regularization terms are added, and the student network now achieves meaningful performance, which indicates the trivial solution is prevented.
Moreover, using both regularizations achieves the best performance, which outperforms (b) without "squeeze".
(e) → (f): training data is supplemented with real data, i.e . adding "span", the performance is further improved.
Domain gap issue We calculate the squared MMD [4] of representations of synthetic and real data to measure their gap in representation space.
Table 6 shows “Squeeze and Span” (Sq&Sp) reducesthe
MMD compared to “Squeeze” by an order of magnitude on CIFAR10 and CIFAR100 and a large margin on STL10, clearly justifying the efﬁcacy of “span” as reducing the domain gap.
[Figure]
accuracy) versus generator quality (FID) on CIFAR100.
A better GAN has a lower FID.
Impact of generator We further compare the performance of our method when we use
GAN checkpoints of different quality.
Fig.shows the top-1 accuracy with respect to FID, which indicates the quality of GAN.
It is not surprising that GAN quality signiﬁcantly impacts our method.
The higher the quality of generator we utilize, the higher performance of learned representation our method can attain.
It is noteworthy that a moderately trained
GAN (FID < 11.03) is already able to contribute additional performance improvement on CIFAR100 when compared to VICReg trained on a mixture of synthetic and real data.
In the appendix, we further analyze the impact of generator feature choices and GAN architectures on the distillation performance. [Table]
and A denote whether to introduce the learnable squeeze module and data augmentation, respectively.
L RD , L var , and L cov represent whether to enable the correpsonding losses.
[Table]
Conclusions
This paper proposes to "squeeze and span" representation from the GAN generator to extract transferable representation for downstream tasks like image classiﬁcation.
The key techniques, "squeeze" and "span", aim to mitigate issues that the GAN generator contains the information necessary for image synthesis but unnecessary for downstream tasks and the domain gap between synthetic and real data.
Experimental results justify the effectiveness of our method and show its great promise in self-supervised representation learning.
We hope more attention can be drawn to studying GAN for representation learning.
Limitation and future work The current form of our work still maintains several limitations that need to be studied in the future.
(1) Since we distill representation from GANs, the performance of learned representation relies on the quality of pre-trained GANs and thus is limited by the progress of GAN techniques.
Therefore, whether a pre-maturely trained GAN can also contribute to selfsupervised representation learning and how to effectively distill them will be an interesting problem.
(2) In this paper, the squeeze module sets the transformation-invariance as the learning objective.
We hope there will be evolving materialization of our framework to continuously improve the effectiveness and efﬁciency of the distillation procedure of GAN generator representations.
(3) We hope more scalable methods may emerge to further arouse the potential of our method
Acknowledgments and Disclosure of Funding
This work was supported by the National Key R&D Program of China under Grant 2018AAA0102801,
National Natural Science Foundation of China under Grant 61620106005, EPSRC Visual AI grant
EP/T028572/
Checklist
The checklist follows the references.
Please read the checklist guidelines carefully for information on how to answer these questions.
For each question, change the default [TODO] to [Yes] , [No] or [N/A] .
You are strongly encouraged to include a justiﬁcation to your answer , either by referencing the appropriate section of your paper or providing a brief inline description.
For example:
• Did you include the license to the code and datasets?
[Yes] See Section ?? .
• Did you include the license to the code and datasets?
[No] The code and the data are proprietary.
• Did you include the license to the code and datasets?
[N/A]
Please do not modify the questions and only use the provided macros for your answers.
Note that the
Checklist section does not count towards the page limit.
In your paper, please delete this instructions block and only keep the Checklist section heading above along with the questions/answers below.
For all authors... (a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s contributions and scope?
[Yes] (b) Did you describe the limitations of your work?
[Yes] See Section(c)
Did you discuss any potential negative societal impacts of your work?
[No] (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes] 2.
If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
[N/A] (b) Did you include complete proofs of all theoretical results?
[N/A] 3.
If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[Yes] See Section 5.
and supplementary for thorough implementation details and the source code.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
[Yes] See supplementary material.
(c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[No] (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
[No] 4.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
[Yes] (b) Did you mention the license of the assets?
[No] (c) Did you include any new assets either in the supplemental material or as a URL?
[No] (d) Did you discuss whether
and how consent was obtained from people whose data you’re using/curating?
[No] (e) Did you discuss whether the data you are using/curating contains personally identiﬁable information or offensive content?
[Yes] See Section 5.
If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?
[N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
[N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
[N/A]