L EARNING D ISENTANGLED R EPRESENTATION BY E XPLOITING P RETRAINED G ENERATIVE M ODELS : A C ONTRASTIVE L EARNING V IEW
Anonymous authors
Paper under double-blind review
A BSTRACT
From the intuitive notion of disentanglement, the image variations corresponding to different factors should be distinct from each other, and the disentangled representation should reﬂect those variations with separate dimensions.
To discover the factors and learn disentangled representation, previous methods typically leverage an extra regularization term when learning to generate realistic images.
However, the term usually results in a trade-off between disentanglement and generation quality.
For the generative models pretrained without any disentanglement term, the generated images show semantically meaningful variations when traversing along different directions in the latent space.
Based on this observation, we argue that it is possible to mitigate the trade-off by ( i ) leveraging the pretrained generative models with high generation quality, ( ii ) focusing on discovering the traversal directions as factors for disentangled representation learning.
To achieve this, we propose
Dis entaglement via Co ntrast ( DisCo ) as a framework to model the variations based on the target disentangled representations, and contrast the variations to jointly discover disentangled directions and learn disentangled representations.
DisCo achieves the state-of-the-art disentangled representation learning and distinct direction discovering, given pretrained non-disentangled generative models including GAN, VAE, and Flow.
I NTRODUCTION
Disentangled representation learning aims to identify and decompose the underlying explanatory factors hidden in the observed data, which is believed by many to be the only way to understand the world for AI fundamentally (Bengio & LeCun, 2007).
To achieve the goal, as shown in Figure 1 (a), we need an encoder and a generator.
The encoder to extract representations from images with each dimension corresponds to one factor individually.
The generator (decoder) decodes the changing of each factor into different kinds of image variations.
With supervision, we can constrain each dimension of the representation only sensitive to one kind of image variation caused by changing one factor respectively.
However, this kind of exhaustive supervision is often not available in real-world data.
The typical unsupervised methods are based on a generative model to build the above encoder and generator framework, e.g., VAE (Kingma & Welling, 2014) provides encoder and generator, and GAN (Goodfellow et al., 2014; Miyato et al., 2018; Karras et al., 2019) provides generator.
During the training process of the encoder and generator, to achieve disentangled representation, the typical methods rely on an additional disentanglement regularization term, e.g., the total correlation for VAE-based methods (Higgins et al., 2017; Burgess et al., 2018;
Kumar et al., 2017; Kim & Mnih, 2018; Chen et al., 2018) or mutual information for InfoGAN-based methods (Chen et al., 2016; Lin et al., 2020).
However, the extra terms usually result in a trade-off between disentanglement and generation quality (Burgess et al., 2018;
Khrulkov et al., 2021).
Furthermore, those unsupervised methods have been proved to have an inﬁnite number of entangled solutions without introducing inductive bias (Locatello et al., 2019).
Recent works (Shen & Zhou, 2021; Khrulkov et al., 2021; Karras et al., 2019; Härkönen et al., 2020; Voynov & Babenko, 2020) show that, for GANs purely trained for image generation, traversing along different directions in the latent space causes different variations
[Figure]
of the generated image.
This phenomenon indicates that there is some disentanglement property embedded in the latent space of the pretrained GAN.
The above observations indicate that training the encoder and generator simultaneous may not be the best choice.
We provide an alternative route to learn disentangled representation: ﬁx the pretrained generator, jointly discover the factors in the latent space of the generator and train the encoder to extract disentangled representation, as shown in Figure 1(b).
From the intuitive notion of disentangled representation, similar image variations should be caused by changing the same factor, and different image variations should be caused by changing different factors.
This provide a novel contrastive learning view for disentangled representation learning and inspires us to propose a framework:
Dis entanglement via Co ntrast ( DisCo ) for disentangled representation learning.
In DisCo , changing a factor is implemented by traversing one discovered direction in the latent space.
For discovering the factors, DisCo adopts a typical network module, Navigator , to provides candidate traversal directions in the latent space (Voynov & Babenko, 2020; Jahanian et al., 2020;
Shen et al., 2020).
For disentangled representation learning, to model the various image variations, we propose a novel ∆ -Contrastor to build a Variation Space where we apply the contrastive loss.
In addition to the above architecture innovations, we propose two key techniques for DisCo : ( i ) an entropy-based domination loss to encourage the encoded representations to be more disentangled, ( ii ) a hard negatives ﬂipping strategy for better optimization of Contrastive Loss.
We evaluate DisCo on three major generative models (GAN, VAE, and Flow) on three popular disentanglement datasets.
DisCo achieves
the state-of-the-art (SOTA) disentanglement performance compared to all the previous discovering-based methods and typical (VAE/InfoGAN-based) methods.
Furthermore, we evaluate DisCo on the real-world dataset FFHQ (Karras et al., 2019) to demonstrate that it can discover SOTA disentangled directions in the latent space of pretrained generative models.
Our main contributions can be summarized as: ( i ) To our best knowledge, DisCo is the ﬁrst uniﬁed framework for jointly learning disentangled representation and discovering the latent space of pretrained generative models by contrasting the image variations.
( ii )
We propose a novel ∆ -Contrastor to model image variations based on the disentangled representations for utilizing
Contrastive Learning.
( iii ) DisCo is an unsupervised and model-agnostic method that endows non-disentangled VAE, GAN, or Flow models with the SOTA disentangled representation learning and latent space discovering.
( iv ) We propose two key techniques for DisCo : an entropy-based domination loss and a hard negatives ﬂipping strategy.
R ELATED W ORK
Typical unsupervised disentanglement.
There have been a lot of studies on unsupervised disentangled representation learning based on VAE (Higgins et al., 2017; Burgess et al., 2018; Kumar et al., 2017; Kim & Mnih, 2018;
Chen et al., 2018) or InfoGAN (Chen et al., 2016; Lin et al., 2020).
These methods achieve disentanglement via an extra regularization, which often sacriﬁces the generation quality (Burgess et al., 2018; Khrulkov et al., 2021).
VAE-based methods disentangle the variations by factorizing aggregated posterior, and InfoGAN-based methods maximize the mutual information between latent factors and related observations.
VAE-based methods achieve relatively good disentanglement performance but have low-quality generation.
InfoGAN-based methods have a relatively high quality of generation but poor disentanglement performance.
Our method supplements generative models pretrained without disentanglement regularization term with contrastive learning in the Variation Space to achieve both high-ﬁdelity image generation and SOTA disentanglement.
Interpretable directions in the latent space.
Recently, researchers have been interested in discovering the interpretable directions in the latent space of generative models without supervision, especially for GAN (Goodfellow et al., 2014; Miyato et al., 2018; Karras et al., 2020).
Based on the fact that the
GAN latent space often possesses semantically meaningful directions
(Radford et al., 2015;
Shen et al., 2020; Jahanian et al., 2020), Voynov & Babenko (2020) propose a regression-based method to explore interpretable directions in the latent space of a pretrained GAN.
The subsequent works focus on extracting the directions from a speciﬁc layer of GANs.
Härkönen et al.
(2020) search for important and meaningful directions by performing PCA in the style space of StyleGAN (Karras et al., 2019; 2020).
Shen & Zhou (2021) propose to use the singular vectors of the ﬁrst layer of a generator as the interpretable directions, and Khrulkov et al.
(2021) extend this method to the intermediate layers by Jacobian matrix.
All the above methods only discover the interpretable directions in the latent space, except for Khrulkov et al.
(2021) which also learns disentangled representation of generated images by training an extra encoder in an extra stage.
However, all these methods can not outperform the typical disentanglement methods.
Our method is the ﬁrst to jointly learn the disentangled representation and discover the directions in the latent spaces.
Contrastive Learning.
Contrastive Learning gains popularity due to its effectiveness in representation learning (He et al., 2020;
Grill et al., 2020; van den Oord et al., 2018; Hénaff, 2020; Li et al., 2020; Chen et al., 2020).
Typically, contrastive approaches bring representations of different views of the same image (positive pairs) closer, and push representations of views from different images (negative pairs) apart using instance-level classiﬁcation with Contrastive Loss.
Recently, Contrastive
Learning is extended to various tasks, such as image translation (Liu et al., 2021; Park et al., 2020) and controllable generation (Deng et al., 2020).
In this work, we focus on the variations of representations and achieve SOTA disentanglement with Contrastive Learning in the Variation Space .
Contrastive
Learning is suitable for disentanglement due to: ( i ) the actual number of disentangled directions is usually unknown, which is similar to Contrastive Learning for retrieval (
Le-Khac et al., 2020), ( ii ) it works in the representation space directly without any extra layers for classiﬁcation or regression.
D ISENTANGLEMENT VIA C ONTRAST 3. O VERVIEW OF D IS C O
From the contrastive view of the intuitive notion of disentangled representation learning, we propose a
DisCo to leverage pretrained generative models to jointly discover the factors embedded as directions in the latent space of the generative models and learn to extract disentangled representation.
The beneﬁts of leveraging a pretrained generative model are two-fold: ( i ) the pretrained models with high-quality image generation are readily available, which is important for reﬂecting detailed image variations and downstream tasks like controllable generation; ( ii ) the factors are embedded in the pretrained model, severing as an inductive bias for unsupervised disentangled representation learning.
DisCo consists of a Navigator to provides candidate traversal directions in the latent space and a ∆ -Contrastor to extract the representation of image variations and build a Variation Space based on the target disentangled representations.
More speciﬁcally, ∆ -Contrastor is composed of two sharedweight Disentangling Encoders.
The variation between two images is modeled as the difference of their corresponding encoded representations extracted by the Disentangling Encoders.
In the Variation Space , by pulling together the variation samples resulted from traversing the same direction and pushing away the ones resulted from traversing different directions, the Navigator learns to discover disentangled directions as factors, and Disentangling Encoder learns to extract disentangled representations from images.
Thus, traversing along the discovered directions causes distinct image variations, which causes separated dimensions of disentangled representations respond.
Different from VAE-based or InfoGAN-based methods, our disentangled representations and factors are in two separate spaces, which actually does not affect the applications.
Similar to the typical methods, the Disentangling Encoder can extract disentangled representations from images, and the pretrained generative model with discovered factors can be applied to controllable generation.
Moreover, DisCo can be applied to different types of generative models.
Here we provide a detailed workﬂow of DisCo .
As Figure 2 shows, given a pretrained generative model G : Z → I , where Z ∈ R L denotes the latent space, and I denotes the image space,
the
[Figure]
workﬂow is: 1) A Navigator A provides a total of D candidate traversal directions in the latent space Z , e.g., in the linear case, A ∈ R L × D is a learnable matrix, and each column is regarded as a candidate direction.
2) Image pairs G ( z ) , G ( z (cid:48) ) are generated.
z is sampled from Z and z (cid:48)
=
z + A ( d, ε ) , where d ∈ { 1 , ..., D } and ε ∈ R , and A ( d, ε ) denotes the shift along the dth direction with ε scalar.
The ∆ -Contrastor , composed of two shared-weight Disentangling
Encoders E , encodes the image pair to a sample v ∈ V as
[Equation]
where V ∈ R J + denotes the Variation Space .
Then we apply Contrastive Learning in V to optimize the Disentangling Encoder E to extract disentangled representations and simultaneously enable
Navigator A to ﬁnd the disentangled directions in the latent space Z .
3. D ESIGN OF D IS C O
We present the design details of DisCo , which include: ( i ) the collection of query set Q = { q i } Bi
=1 , positive key set K
+ = { k + i } Ni =1 and negative key set K − = { k − i } Mi =1 , which are three subsets of the
Variation Space V , ( ii ) the formulation of the Contrastive Loss.
According to our goal of contrasting the variations, the samples from Q and K + share the same traversal direction and should be pulled together, while the samples from Q and K − have different directions and should be pushed away.
Recall that each sample v in V is determined as v ( z , d, ε ) .
To achieve the contrastive learning process, we construct the query sample q i = v ( z i , d i , ε i ) , the key sample k
+ i = v ( z + i , d
+ i , ε + i ) and the negative sample k − i = v ( z − i , d − i , ε − i ) .
Speciﬁcally, we randomly sample a direction index ˆ d from a discrete uniform distribution U{ 1 , D } for { d i } Bi
=
and { d + i } Ni =1 to guarantee they are the same.
We randomly sample { d − i } Mi =1 from the set of the rest of the directions U{ 1 , D } \ { ˆ d } individually and independently to cover the rest of directions in Navigator
A .
Note that the discovered direction should be independent with the starting point and the scale of variation, which is in line with the disentangled factors.
Therefore, { z i } Bi =1 , { z + i } Ni =1 , { z − i } Mi = are all sampled from latent space Z , and { ε i } Bi =1 , { ε + i } Ni =1 , { ε − i } Mi =1 are all sampled from a shared continuous uniform distribution U [ − (cid:15), (cid:15) ] individually and independently.
We normalize each sample in Q , K + , and K − to a unit vector to eliminate the impact caused by different shift scalars.
For the design of Contrastive Loss, a well-known form of Contrastive Loss is InfoNCE (van den Oord et al., 2018):
[Equation]
where τ is a temperature hyper-parameter and { k i } N + M i = =
{ k + i } Ni =1 (cid:83) { k − i } Mi =1 . The InfoNCE is originate from BCELoss (Gutmann & Hyvärinen, 2010). BCELoss has been used to achieve contrastive learning (Wu et al., 2018;
Le-Khac et al., 2020; Mnih & Kavukcuoglu, 2013; Mnih &
Teh, 2012). We choose to follow them to use BCELoss L logits for reducing computational cost:
[Equation]
where σ denotes the sigmoid function, l + i denotes the part for positive samples, and l − i denotes the part for the negative ones.
Note that we use a shared positive set for B different queries to reduce the computational cost.
3. K EY T ECHNIQUES FOR D IS C O
Entropy-based domination loss.
By optimizing the Contrastive Loss, Navigator A is optimized to ﬁnd the disentangled directions in the latent space, and Disentangling Encoder E is optimized to extract disentangled representations from images.
To further make the encoded representations more disentangled, i.e., when traversing along one disentangled direction, only one dimension of the encoded representation should respond, we thus propose an entropy-based domination loss to encourage the corresponding samples in the Variation Space to be one-hot.
To implement the entropy-based domination loss, we ﬁrst get the mean c of Q and K + as
[Equation]
We then compute the probability as p i = exp c
( i ) / (cid:80)
Jj =1 exp c ( j ) , where c ( i ) is the i -th element of c and J is the number of dimensions of c .
The entropy-based domination loss L ed is calculated as
[Equation]
Hard negatives ﬂipping.
Since the latent space of the generative models is a high-dimension complex manifold, many different directions carry the same semantic meaning.
These directions with the same semantic meaning result in hard negatives during the optimization of Contrastive
Loss.
The hard negatives here are different from the hard negatives in the works of self-supervised representation learning (He et al., 2020;
Coskun et al., 2018), where they have reliable annotations of the samples.
Here, our hard negatives are more likely to be “false” negatives, and we choose to ﬂip these hard negatives into positives.
Speciﬁcally, we use a threshold T to identify the hard negative samples, and use their similarity to the queries as the pseudo-labels for them:
[Equation]
where ˆ l − i denotes the modiﬁed l − i , and α ij = q i · k − j /τ .
Therefore, the modiﬁed ﬁnal BCELoss is:
[Equation]
Full objective.
With the above two techniques, the full objective is:
[Equation]
where λ is the weighting hyper-parameter for entropy-based domination loss L ed .
[Table]
[Figure]
E XPERIMENT
In this section, we ﬁrst follow the well-accepted protocol (Locatello et al., 2019; Khrulkov et al., 2021) to evaluate the learned disentangled representation, which also reﬂects the performance of discovered directions implicitly (Lin et al., 2020)
(Section 4.1).
Secondly, we follow Li et al.
(2021a) to directly evaluate the discovered directions (Section 4.2).
Finally, we conduct ablation study (Section 4.3).
4. E VALUATIONS ON D ISENTANGLED R EPRESENTATION 4.1. E XPERIMENTAL S ETUP
Datasets.
We consider the following popular datasets in the disentanglement areas:
Shapes3D (Kim & Mnih, 2018) with 6 ground truth factors, MPI3D (Gondal et al., 2019) with 7 ground truth factors, and Cars3D (Reed et al., 2015) with 3 ground truth factors.
In the experiments of the above datasets, images are resized to the 64x64 resolution.
[Figure] [Figure]
[Figure]
Pretrained generative models.
For GAN, we use the StyleGAN2 model (Karras et al., 2020).
For
VAE, we use a common structure with convolutions (Locatello et al., 2019).
For Flow, we use
Glow (Kingma & Dhariwal, 2018).
Baseline.
For the typical disentanglement baselines, we choose FactorVAE (Kim & Mnih, 2018), β -TCVAE
(Chen et al., 2018) and InfoGAN-CR (Lin et al., 2020).
For discovering-based methods, we consider serveral recent methods: GANspace (GS) (Härkönen et al., 2020), LatentDiscovery (LD) (Voynov & Babenko, 2020), ClosedForm (CF) (Shen & Zhou, 2021) and DeepSpectral (DS) (Khrulkov et al., 2021).
For these methods, we follow Khrulkov et al.
(2021) to train an additional encoder to extract disentangled representation.
We are the ﬁrst to extract disentangled representations from pretrained VAE and Flow, so we extend LD to VAE and Flow as a baseline.
Disentanglement metrics.
We mainly consider two representative ones: the Mutual Information
Gap (MIG) (Chen et al., 2018) and the Disentanglement metric (DCI) (Eastwood & Williams, 2018).
MIG requires each factor to be only perturbed by changes of a single dimension of representation.
DCI requires each dimension only to encode the information of a single dominant factor.
We evaluate the disentanglement in terms of both representation and factors.
We also provide results for β -VAE score (Higgins et al., 2017) and FactorVAE score (Kim & Mnih, 2018) in Appendix B.3.
Randomness.
We consider the randomness caused by random seeds and the strength of the regularization term (Locatello et al., 2019).
For random seeds, we follow the same setting as the baselines.
Since DisCo does not have a regularization term, we consider the randomness of the pretrained generative models.
For all methods, we ensure there are 25 runs, except that Glow only has one run, limited by GPU resources.
More details are presented in Appendix A.
4.1. E XPERIMENTAL R ESULTS
The quantitative results are summarized in Table 1 and Figure 3.
More details about the experimental settings and results are presented in Appendix A & C.
DisCo vs. typical baselines.
Our DisCo achieves the SOTA performance consistently in terms of MIG and DCI scores.
The variance due to randomness of DisCo tends to be smaller than those typical baselines.
We demonstrate that the method, which extracts disentangled representation from pretrained non-disentangled models, can outperform typical disentanglement baselines.
DisCo vs. discovering-based methods.
Among the baselines based on discovering pretrained
GAN, CF achieves the best performance.
DisCo outperforms CF in almost all the cases by a large margin.
Besides, these baselines need an extra stage (Khrulkov et al., 2021) to get disentangled representation, while our Disentangling Encoder can directly extract disentangled representation.
4. E VALUATIONS ON D ISCOVERED D IRECTIONS
[Table]
To evaluate the discovered directions, we compare DisCo on StyleGAN2 with GS , LD , CF and DS on the real-world dataset FFHQ (Karras et al., 2019) 1 . and adopt the comprehensive Manipulation Disentanglement Score (MDS)
(Li et al., 2021a) as a metric.
To calculate MDS, we use 40 CelebaHQ-Attributes predictors released by StyleGAN.
Among them, we select
Young , Smile , Bald and Blonde Hair , as they are attributes with an available predictor and commonly found by all methods at the same time.
The results are summarized in Table 3.
DisCo has shown better overall performance compared to other baselines, which veriﬁes our assumption that learning disentangled representation beneﬁts latent space discovering.
We also provide qualitative comparisons in Figure 4.
Finally, we provide an intuitive analysis in Appendix D for why DisCo can ﬁnd those disentangled directions.
4. A BLATION S TUDY
In this section, we perform ablation study of DisCo only on GAN, limited by the space.
For the experiments, we use the Shapes3D dataset, and the random seed is ﬁxed.
Choice of latent space.
For style–based GANs (Karras et al., 2019; 2020), there is a style space W , which is the output of style network (MLP) whose input is a random latent space Z .
As demonstrated in Karras et al.
(2019), W is more interpretable than Z .
We conduct experiments on W and Z respectively to see how the latent space inﬂuences the performance.
As shown in Table 4, DisCo on
W is better, indicating that the better the latent space is organized, the better disentanglement DisCo can achieve.
[Table]
Choices of A .
Following the setting of Voynov & Babenko (2020), we mainly consider three options of A : a linear operator with all matrix columns having a unit length, a linear operator with orthonormal matrix columns, or a nonlinear operator of 3 fully-connected layers.
The results are shown in Table 4.
For latent spaces W and Z , A with unit-norm columns achieves nearly the best performance in terms of MIG and DCI scores.
Compared to A with orthonormal matrix columns, using A with unitnorm columns is more expressive with less constraints.
Another possible reason is that A is global without conditioned on the latent code z .
A non-linear operator is more suitable for a local navigator A .
For such a much more complex local and non-linear setting, more inductive bias or supervision should be introduced.
Entropy-based domination loss.
Here, we verify the effectiveness of entropy-based domination loss L ed for disentanglement.
For a desirable disentangled representation, one semantic meaning corresponds to one dimension.
As shown in Table 4, L ed can improve the performance by a large margin.
We also visualize the Variation Space to further demonstrate the effectiveness of our proposed [Figure] loss in Figure 5.
Adding the domination loss makes the samples in the Variation Space to be one-hot, which is desirable for disentanglement.
Hard negatives ﬂipping.
We run our DisCo with or without the hard negatives ﬂipping strategy to study its inﬂuence.
As shown in Table 4, ﬂipping hard negatives can improve the disentanglement ability of DisCo .
The reason is that the hard negatives have the same semantics as the positive samples.
In this case, treating them as the hard negatives does not make sense.
Flipping them with pseudo-labels can make the optimization of Contrastive Learning easier.
Hyperparmeter N & M. We run DisCo with different ratios of N : M with a ﬁxed sum of 96, and different sum of N + M with a ﬁxed ratio 1 : 2 to study their impacts.
As shown in Figure 6 (a), the best ratio is N : M = 32 : 64 = 1 : 2 , as the red line (MIG) and blue line (DCI) in the ﬁgure show that larger or smaller ratios will hurt DisCo , which indicates DisCo requires a balance between
N and M .
As shown in Figure 6 (b), the sum of N + M has slight impact on DisCo .
For other hyperparameters, we set them empirically, and more details are presented in Appendix A.
Contrast vs. Classiﬁcation.
To verify the effectiveness of Contrast, we substitute it with classiﬁcation by adopting an additional linear layer to recover the corresponding direction index and the shift along this direction.
As Table 2 shows, Contrastive Learning outperforms Classiﬁcation signiﬁcantly.
Concatenation vs. Variation.
We further demonstrate that the Variation Space is crucial for DisCo .
By replacing the difference operator with concatenation, the performance drops signiﬁcantly (Table 2), indicating that the encoded representation is not well disentangled.
On the other hand, the disentangled representations of images are achieved by Contrastive Learning in the Variation Space .
4. A NALYSIS OF D IFFERENT G ENERATIVE M ODELS
As shown in Table 1, DisCo can be well generalized to different generative models (GAN, VAE, and Flow).
DisCo on GAN and VAE can achieve relative good performance, while DisCo on Flow is not as good.
The possible reason is similar to the choice of latent space of GAN.
We assume the disentangled directions are global linear and thus use a linear navigator.
In contrast to GAN and VAE, we suspect that Flow may not conform to this assumption well.
Furthermore, Flow has the problems of high GPU cost and unstable training, which limit us to do further exploration.
C ONCLUSION
In this paper, we present an unsupervised and model-agnostic method DisCo , which is a Contrastive
Learning framework to learn disentangled representation by exploiting pretrained generative models.
We propose an entropy-based domination loss and a hard negatives ﬂipping strategy to achieve better disentanglement.
DisCo outperforms typical unsupervised disentanglement methods while maintaining high image quality.
We pinpoint a new direction that Contrastive Learning can be well applied to extract disentangled representation from pretrained generative models.
There may be some speciﬁc complex generative models, for which the global linear assumption of disentangled directions in the latent space could be a limitation.
For future work, extending DisCo to the existing
VAE-based disentanglement framework is an exciting direction.
A I MPLEMENTATION D ETAILS A. S ETTING FOR D IS C O
We provide an example code at https://bit.ly/3innjnr .
For the hyperparameters, we empirically set the temperature τ to 1 , threshold T to 0 .
95 , batch size B to 32 , the number of positives N to 32 , the number of negatives K to 64, the loss weight λ for L ed to 1 , the number of directions D to 64 and the dimension of the representation J to 32.
We use an
Adam optimizer (Kingma & Ba, 2015) in the training process, as shown in Table 5.
Besides N and
M , we empirically ﬁnd that DisCo is not sensitive to threshold T ≥ 0 .
9 and other hyperparameters.
For the randomness, there is no regularization term for DisCo , thus the disentanglement performance is mainly inﬂuenced by the pretrained generative models.
We follow Khrulkov et al.
(2021) to run 5 random seeds to pretrain the GAN and 5 random seeds for training DisCo .
We have the same setting for DisCo on GAN and VAE on all three datasets.
For Flow, we only use one random seed to pretrain the Glow and use one random seed for DisCo .
Compare with the baselines, for DisCo on StyleGAN, our modiﬁcation happens globally on all layers in the W space without any manual selection.
[Table] [Table]
T < 0 .
9 , we may ﬂip true hard negative and thus lead the optimization of Contrastive Loss collapse.
A. S ETTING FOR BASELINES
In this section, we introduce the implementation setting for the baselines (including randomness).
VAE-based methods.
We choose FactorVAE and β -TCVAE as the SOTA VAE-based methods, we follow Locatello et al.
(2019) to use the same architecture of encoder and decoder.
For the hyperparameter s, we use the the best settings by grid search.
We set the latent dimension of representation to 10 .
For FactorVAE, we set the hyperparameter γ to 10 .
For β -TCVAE, we set the hyperparameter β to 6 .
For the random seed s, considering our method has 25 run, we run 25 times with different random seeds for each model to make the comparison fair.
InfoGAN-based methods.
We choose InfoGAN-CR as a baseline.
We use the ofﬁcial implementation 2 with the best hyperparameter settings by grid search.
For the random seed s, we runtimes with different random seeds
Discovering-based methods.
We follow Khrulkov et al.
(2021) to use the same settings for the following four baselines: LD (GAN), CF , GS , and DS .
Similar to our method ( DisCo ), discoveringbased methods do not have a regularization term.
Thus, for the randomness, we adopt the same strategy with DisCo .
We take the top-10 directions for 5 different random seeds for GAN anddifferent random seeds for the additional encoder to learn disentangled representations.
LD (VAE) & LD (Flow).
We follow LD (GAN) to use the same settings and substitute the GAN with VAE / Glow.
The only exception is the randomness for LD (Flow).
We only run one random seed to pretrain the Glow and use one random seed for the encoder.
A. M ANIPULATION D ISENTANGLEMENT S CORE
As claimed in Li et al.
(2021a), it is difﬁcult to evaluate the performance on discovering the latent space among different methods, which often use model-speciﬁc hyper-parameters to control the editing strength.
Thus, Li et al.
(2021a) propose a comprehensive metric called Manipulation Disentanglement Score (MDS), which takes both the accuracy and the disentanglement of manipulation into consideration.
For more details, please refer to Li et al.
(2021a).
A. D OMAIN GAP PROBLEM
Please note that there exists a domain gap between the generated images of pretrained generative models and the real images.
However, the good performance on disentanglement metrics shows that the domain gap has limited inﬂuence on DisCo .
A. A RCHITECTURE
Here, we provide the model architectures in our work.
For the architecture of StyleGAN2, we follow
Khrulkov et al.
For the architecture of Glow, we use the open-source implementation 3 .
[Table]
[Table]
B M ORE E XPERIMENTS B. M ORE QUALITATIVE COMPARISON
We provide some examples for qualitative comparison.
We ﬁrst demonstrate the trade-off problem of the VAE-based methods.
As shown in Figure 7, DisCo leverages the pretrained generative model and does not have the trade-off between disentanglement and generation quality.
[Figure]
Furthermore, as shown in Figure 8 and Figure 9, VAE-based methods suffer from poor image quality.
When changing one attribute, the results of discovering-based methods tend to also change other attributes.
[Figure]
[Figure]
We also provide qualitative comparisons between DisCo and InfoGAN-CR.
Note that the latent space of InfoGAN-CR is not aligned with the pretrained StyleGAN2.
InfoGAN-CR also suffers from the trade-off problem, and its disentanglement ability is worse than DisCo .
[Figure]
[Figure]
We explain the comparison in the main paper and show more manipulation comparisons here.
[Figure]
[Figure] [Figure]
B. A NALYSIS OF THE LEARNED DISENTANGLED REPRESENTATIONS
We feed the images traversing the three most signiﬁcant factors (wall color, ﬂoor color, and object color) of Shapes3D into the Disentangling Encoders and plot the corresponding dimensions of the encoded representations to visualize the learned disentangled space.
The location of each point is the disentangled representation of the corresponding image.
An ideal result is that all the points form a cube, and color variation is continuous.
We consider three baselines that have relatively higher MIG and DCI: CF , DS , LD .
As the ﬁgures below show, the points in the latent space of CF and DS are not well organized, and the latent space of all the three baselines are not well aligned with the axes, especially for LD .
DisCo learns a well-aligned and well-organized latent space, which signiﬁes a better disentanglement.
[Equation]
Ours
B. M ORE QUANTITATIVE COMPARISON
We provide additional quantitative comparisons in terms of β -VAE score and FactorVAE score.
DisCo on pretrained GAN is comparable to discovering-based baselines in terms of β -VAE score and FactorVAE score, suggesting that some disagreement between these two scores and MIG/ DCI.
However, note that the qualitative evaluation in Figure 8, Figure 9 and Section B.2 shows that the disentanglement ability of DisCo is better than all the baselines on Shapes3D dataset.
[Table]
We also provide an additional experiment on Noisy-DSprites dataset.
We compare DisCo with β -TCVAE
(the best typical method) and CF (the best discovering-based method) in terms of MIG and DCI metrics.
[Table]
C L ATENT T RAVERSALS
In this section, we visualize the disentangled directions of the latent space discovered by DisCo on each dataset.
For Cars3D, Shapes3D, Anime and MNIST, the iamge resolution is 64 × 64 . For FFHQ, LSUN cat and LSUN church, the image resolution is 256 × 256 . Besides StyleGAN2, we also provide results of Spectral Norm GAN (Miyato et al., 2018) 4 on MNIST (LeCun et al., 2010) and Anime Face (Jin et al., 2017) to demonstrate that DisCo can be well generalized to other types of GAN.
[Figure]
[Figure] [Figure]
[Figure] [Figure]
[Figure] [Figure]
[Figure] [Figure]
[Figure] [Figure]
[Figure] [Figure]
[Figure]
D A N I NTUITIVE A NALYSIS FOR D IS C O
DisCo works by contrasting the variations resulted from traversing along the directions provided by the Navigator.
Is the method sufﬁcient to converge to the disentangled solution?
Note that it is very challenging to answer this question.
To our best knowledge, for unsupervised disentangled representation learning, there is no sufﬁcient theoretical constraint to guarantee the convergence to a disentangled solution Locatello et al.
Here we provide an intuitive analysis for DisCo and try to provide our thoughts on how DisCo ﬁnd the disentangled direction in the latent space, which is supported by our observations on pretrained GAN both quantitatively and qualitatively.
The intuitive analysis consists of two part: ( i )
The directions that can be discovered by DisCo have different variation patterns compared to random directions.
( ii ) DisCo hardly converges to the an entangled solution.
D. W HAT KIND OF DIRECTIONS D IS C O CAN CONVERGE TO ?
[Figure]
First, we visualize the latent space and show that there are some variation patterns in the latent space for disentangled factors.
We design the following visualization method.
Given a pretrained GAN and two directions in the latent space, we traverse along the plane expanded by the two directions to generate a grid of images.
The range is large enough to include all values of these disentangled factors, and the step is small enough to obtain a dense grid.
Then, we input these images into an encoder that trained with ground truth factors labels.
We get a heatmap of each factor (the value is the response value corresponding dimension of the factor).
In this way, we can observe the variation pattern that emerged in the latent space.
We take the pretrained StyleGAN on Shapes3D (synthetic) and FFHQ (real-world).
For Shapes3D, we take background color and ﬂoor color as the two factors (since they refer to different areas in the image, these two factors are disentangled).
For FFHQ, we take smile (mouth) and bald (hair) as the two factors (disentangled for referring to different areas).
We then choose random directions and the directions discovered by DisCo .
The results are shown in Figure 27 and Figure 28.
We ﬁnd a clear difference between random directions and directions discovered by DisCo .
This is because DisCo can learn the directions by separating the variations resulted from traversing along with them.
However, not all directions can be separated.
For those directions in which the variations are not able to be recognized or clustered by the encoder E , it is nearly impossible for DisCo to converge to them.
Conversely, for those directions that can be easily recognized and clustered,
DisCo will converge to them with a higher probability.
From the following observations, we ﬁnd that the variation patterns resulting from the directions corresponding to disentangled factors are easily recognized and clustered.
[Figure]
D. W HY D IS C O HARDLY CONVERGES TO THE ENTANGLED CASES ?
In the previous section, we show that DisCo can discover the directions with distinct variation patterns and exclude random directions.
Here we discuss why DisCo can hardly converge to the following entangled case (trivial solution based on disentangled one).
For example, suppose there is an entangled direction of factors A and B (A and B change with the same rate when traversing along with it) in the latent space of generative models, and DisCo can separate the variations resulting from the direction of A and the entangled direction.
In that case, DisCo has no additional bias to update these directions to converge to disentangled ones.
In the following text, for ease of referring to, we denote the entangled direction of factors A and B (A and B change with the same rate when traversing along with it) as A+B direction, and direction of factor
A (only A change when we traverse along with it).
The reasons for why DisCo is hardly converged to the case of A and A+B are two-fold:
( i )
Our encoder is a lightweight network (5 CNN layers + 3 FC layers).
It is nearly impossible for it to separate the A and A+B directions.
( ii )
In the latent space of the pretrained generative models, the disentangled directions (A, B) are consistent at different locations.
In contrast, the entangled directions (A+B) are not, as shown in Figure 29.
We conduct the following experiments to verify them.
For ( i ) , we replace our encoder in DisCo with a ResNet-50 and train DisCo from scratch on the Shapes3D dataset.
The loss, MIG, and DCI are presented in Table 11.
The trivial solution is possible when the encoder is powerful enough to ﬁt the A and A+B directions to “become orthogonal”.
With this consideration, in DisCo we adopt a lightweight encoder to avoid this issue.
[Table]
For ( ii ) , as the sketch Figure 29 demonstrates, the disentangled directions (”A“- blue color or “B”- green color) are consistent, which is invariant to the location in the latent space, while the entangled directions (”A+B“- red color) is not consistent on different locations.
The fundamental reason is that: the directions of the disentangled variations are invariant with the position in the latent space.
However, the “rate” of the variation is not.
E.g., at any point in the latent space, going “up” constantly changes the camera’s pose.
However, at point a, going “up” with step 1 means rotating 10 degrees.
At point b, going “up” with step 1 means rotating 5 degrees.
When the variation “rate” of “A” and “B” are different, the “A+B” directions at different locations are not consistent.
Based on the different properties of disentangled and entangled directions in the latent space, DisCo can discover the disentangled directions with contrastive loss.
The contrastive loss can be understood from the clustered view (Wang & Isola, 2020; Li et al., 2021b).
The variations from the disentangled directions are more consistent and can be better clustered compared to the variations from the entangled ones.
Thus, DisCo can discover the disentangled directions in the latent space and learn disentangled representations from images.
We further provide the following experiments to support our above analysis.
D.2. Q UANTITATIVE EXPERIMENT
We compare the losses of three different settings:
• A : For a navigator with disentangled directions, we ﬁx the navigator and train the encoder until convergence.
• A + B
For a navigator with entangled directions (we use the linear combination of the disentangled directions to initialize the navigator), we ﬁx it and train the encoder until convergence.
• A + B → A : After A+B is convergent, we update both the encoder and the navigator until convergence.
The Contrastive loss after convergence is presented in Table 12.
[Table]
[Equation]
[Table]
The results show that: ( i ) The disentangled directions (A) can lead to lower loss and better performance than entangled directions (A+B), indicating no trivial solution.
( ii ) Even though the encoder with A+B is converged, when we optimize the navigator, gradients will still backpropagate to the navigator and converge to A.
D.2. Q UALITATIVE EXPERIMENT
We visualize the latent space of GAN in Figure 30 to verify the variation “rate” in the following way: in the latent space, we select two ground truth disentangled directions: ﬂoor color (A) and background color (B) obtained by supervision with InterFaceGAN (Shen et al., 2020), we conduct equally spaced sampling along the two disentangled directions:
A (labeled with green color variation),
B (labeled with gradient blue color) and composite direction (A+B, labeled with gradient red color) as shown in Figure 30 (a).
Then we generate the images (also include other images on the grid as shown in Figure 30 (b) ), and feed the images in the bounding boxes into a “ground truth” encoder (trained with ground truth disentangled factors) to regress the “ground truth” disentangled representations of the images.
In Figure 30 (c), the points labeled with green color are well aligned with the x-axis indicating only ﬂoor color change, points labeled with blue variation are well aligned with the y-axis indicating only background color change.
However, the points labeled with red color are NOT aligned with any line, which indicates the directions of A+B are not consistent.
Further, the variation “rate” is relevant to the latent space locations for the two disentangled directions.
This observation well supports our idea shown in Figure 29.
The different properties between disentangled and entangled directions enable
DisCo to discover the disentangled directions in the latent space.
E E XTENSION : B RIDGE THE PRETRAINED VAE AND PRETRAINED GAN
Researchers are recently interested in improving image quality given the disentangled representation generated by typical disentanglement methods.
Lee et al.(Lee et al., 2020) propose a post-processing stage using a GAN based on disentangled representations learned by VAE-based disentanglement models.
This method scariﬁes a little generation ability due to an additional constraint.
Similarly,
Srivastava et al.
(Srivastava et al., 2020) propose to use a deep generative model with AdaIN (Huang & Belongie, 2017) as a post-processing stage to improve the reconstruction ability.
Following this setting, we can replace the encoder in DisCo (GAN) with an encoder pretrained by VAE-based disentangled baselines.
In this way, we can bridge the pretrained disentangled VAE and pretrained
GAN, as shown in Figure 31.
Compared to previous methods, our method can fully utilize the state-of-the-art GAN and the state-of-the-art VAE-based method and does not need to train a deep generative model from scratch.
[Figure]
F D ISCUSSION ON R ELATION BETWEEN BCEL OSS AND NCEL OSS
We would like to present a deep discussion on the relation between the BCELoss L logits and NCELoss
L NCE .
This discussion is related to the NCE paper Gutmann & Hyvärinen (2010), and InfoNCE paper van den Oord et al.
The discussion is as following: ( i ) we ﬁrst provide a formulation of a general problem and get two objectives, L 1 and L 2 , and L 1 is the upper bound of L 2 .
( ii )
Following Gutmann & Hyvärinen (2010), we show that L 1 is aligned with L BCE under the setting of Gutmann & Hyvärinen (2010).
( iii )
Following van den Oord et al.
, we prove L 2 is aligned with L NCE under the setting of van den Oord et al.
( iii )
We discuss the relation between these objectives and the loss in our paper.
Part I. Assume we have S observations { x i } Si =1 from a data distribution p ( x ) , each with a label
C i ∈ { 0 , 1 } .
The we denote the posterior probabilities as p + ( x ) =
p ( x | C = 1) and p − ( x ) = p ( x | C = 0) .
We deﬁne two objectives as follow:
[Equation]
(10) and
L
[Equation]
{+Since − (cid:80) Si =1 (1 − C i ) log p ( C i = 
+}
| x i ) ≥ 0 , we have
L
[Equation]
L 2 .
L 1 is the upper bound of L 2 .
This a general formulation of a binary classiﬁcation problem.
{+In the context of our paper, we have a paired observation x i : ( q, k i ) , with q as the query, and the key k i is either from a positive key set { k + j } Nj =1 or as negative key set { k − m } Mm =
+}
(i.e., { k i } N + M i = =
{ k + j } Nj =1 (cid:83) { k − m } Mm =1 ), where
M = S − N .
And C i is assigned as:
[Equation]
In our paper, we have h ( x ) =
exp( q · k/τ ) .
Part II.
In this part, following Gutmann & Hyvärinen (2010), we show that L 1 is aligned with
L logits (Equation 3 in the main paper) under the setting of Gutmann & Hyvärinen (2010).
Following
Gutmann & Hyvärinen (2010))
, we assume the prior distribution P ( C = 0)
=
P ( C = 1) = 1 / 2 , according to the Bayes rule, we have
[Equation]
And P ( C = 0 | x ) = 1 − P ( C = 1 | x ) .
On the other hand, we have a general form of BCELoss, as
[Equation]
15) where σ ( · ) is the sigmoid function.
We have
[Equation]
From Gutmann & Hyvärinen (2010) Theorem 1, we know that when L BCE is minimized, we have
[Equation]
Thus, we know the BCELoss L BCE is a approximation of the objective L 1 .
Part.
III
Following van den Oord et al.
, we prove L 2 is aligned with L NCE (Equation 2 in the main paper) under the setting of van den Oord et al.
From the typical contrastive setting (one positive sample, others are negative samples, following van den Oord et al.
, we assume there is only one positive sample, others are negatives in { x i } Si =1 .
Then, the probability of x i sample from p + ( x ) rather then p − ( x ) is as follows,
[Equation]
From van den Oord et al.
(2018), we know that when minimize Equation 11, we have h ( x ) =
exp( q · k/τ ) ∝ p + ( x ) p − ( x ) .
In this case, we get the form of L NCE as
[Equation]
L NCE is a approximate of L 2 .
Part.
IV When generalize the contrastive loss into our setting ( N positive samples, M negative samples).
The BCELoss (Equation 15) can be reformulated as
The BCELoss (Equation 15) can be reformulated as
[Equation]
Similarly, the NCEloss (Equation 19) can be reformulated as
[Equation]
ˆ L BCE is aligned with L logits (Equation 3 in our main paper), and ˆ L NCE is aligned with L NCE (Equation 2 in the main paper).
Now we have L 1 (approximated by L BCE ) is the upper bound of L 2 (approximated by L NCE ).
However, as you may notice, the assumptions we made in Part II and Part III are different, one is
P ( C = 0) =
P ( C = 1) , the other one is only one positive sample, others are negative.
Also the extent to our situation is more general case ( N positives, others are negatives).
However, they have the same objective, which is by contrasting positives and negatives, we can use h ( x ) =
exp ( q · k/τ ) to estimate p + /p − .
We can think the h ( x ) as a similarity score, i.e. if q and k are from a positive pair (they have the same direction in our paper), h ( x ) should be as large as possible ( p + /p − > 1 ) and vice versa.
From this way, we can learn the representations ( q, k ) to reﬂect the image variation, i.e., similar variations have higher score h ( x ) , while different kinds of variation have lower score h ( x ) .
Then with this meaningful representation, in the latent space, can help to discover the directions carrying different kinds of image variation.
This is an understanding, from a contrastive learning view, of how our method works.