ADIS-GAN: A FFINE D ISENTANGLED GAN
Anonymous authors
Paper under double-blind review
A BSTRACT
This paper proposes Afﬁne Disentangled GAN (ADIS-GAN), which is a Generative Adversarial Network that can explicitly disentangle afﬁne transformations in a self-supervised and rigorous manner.
The objective is inspired by InfoGAN, where an additional afﬁne regularizer acts as the inductive bias.
The afﬁne regularizer is rooted in the afﬁne transformation properties of images, changing some properties of the underlying images, while leaving all other properties invariant.
We derive the afﬁne regularizer by decomposing the afﬁne matrix into separate transformation matrices and inferring the transformation parameters by maximum likelihood estimation.
Unlike the disentangled representations learned by existing approaches, the features learned by ADIS-GAN are axis-aligned and scalable, where transformations such as rotation, horizontal and vertical zoom, horizontal and vertical skew, horizontal and vertical translation can be explicitly selected and learned.
ADIS-GAN successfully disentangles these features on the MNIST,
CelebA, and dSprites datasets.
I NTRODUCTION
In a disentangled representation, observations are interpreted in terms of a few explanatory factors.
Examples of such factors are the rotation angle(s), scale, or position of an object in an image.
Disentangled variables are generally considered as the abstraction of interpretable semantic information and reﬂection of separatable factors of variation in the data.
Many studies have explored the effectiveness of disentangled representations (Bengio et al., 2013; N et al., 2017; LeCun et al., 2015;
Lake et al., 2017; Tschannen et al., 2018).
The information presented in observations is encoded in an interpretable and compact manner, e.g., the texture style and the orientation of the objects (Bengio et al., 2013; LeCun et al., 2015; Lake et al., 2017; Tschannen et al., 2018).
The learned representation are more generalizable and can be useful for downstream tasks, such as classiﬁcation and visualization (Bengio et al., 2013; N et al., 2017; Chen et al., 2016).
The concept of disentangled representation has been deﬁned in several ways in the literature (Locatello et al., 2019; Higgins et al., 2018; Eastwood & Williams, 2018).
The necessity of explicit inductive biases both for learning approaches and the datasets is discussed in Locatello et al.
Inductive bias refers to the set of assumptions that the learner uses to predict outputs of given inputs that it has not encountered.
For instance, in the dSprites dataset objects are displayed at different angles and positions; such prior knowledge helps to detect and classify the objects.
However, the inductive biases in existing deep learning models are mostly implicit.
The proposed ADIS-GAN utilizes relative afﬁne transformations (see Section 4) as the explicit inductive bias, leading to axisaligned and scalable disentangled representations.
Axis-alignment: The issue of axis-alignment is addressed in Higgins et al.
(2018), where each latent dimension should have a pre-deﬁned unique axis-alignment.
Without axis-alignment, the features learned by disentangled representation need to be identiﬁed with expert knowledge after the training, which could be a cumbersome process when dealing with a large number of features.
The axis-alignment property also helps to discover desired but non-dominant attributes (e.g., roll angle of face in CelebA dataset).
Scalability: The scalability property allows us to make a trade-off between the compactness and expressivity of the disentangled representation.
For example, the zoom attribute can be decomposed as horizontal and vertical zoom.
A more compact representation encodes the zoom attribute by one latent dimension, while a more expressive representation decomposes the zoom attribute as
[Figure]
horizontal and vertical zoom, encoded by two latent dimensions.
The scalability property provides a solution to an open question related to disentangled representations: Should we capture a data generative factor (e.g., zoom) by one or multiple latent dimensions?
Many disentanglement metrics Higgins et al.
; Chen et al.
; Kim & Mnih (2018); Eastwood & Williams (2018) rely on a single latent dimension, while others Denton & Birodkar (2017); Ridgeway & Mozer (2018);
Higgins et al.
(2018) encode latent factors via multiple latent dimensions.
We motivate the importance of axis-alignment and scalability in particular for afﬁne transformations (see Figure 1), where disentangling object poses from texture and shape is an attractive property of an algorithm in the imaging domain (Jaderberg et al., 2015; Bepler et al., 2019; Engstrom et al., 2019).
In supervised learning tasks, Spatial Transformer Network Jaderberg et al.
(2015) can actively spatially transform an image by providing a proper afﬁne transformation matrix.
In unsupervised learning tasks, few algorithms have successfully disentangled the afﬁne transformation.
In
Bepler et al.
(2019), an algorithm is introduced that disentangles rotation and translation but not an entire afﬁne transformation.
We propose ADIS-GAN, which is a Generative Adversarial Network that utilizes the afﬁne regularizer (see Section 4) as an inductive bias to explicitly disentangle the afﬁne transformation.
The afﬁne regularizer is rooted in the afﬁne transformation properties of the images, that affect certain properties of the underlying images, while leaving all other properties invariant.
We derive the afﬁne regularizer by decomposing the afﬁne matrix into separate transformations and inferring the transformation parameters by maximum likelihood estimation.
Unlike the disentangled representations learned by existing approaches, the features learned by ADIS-GAN are axis-aligned and scalable, where transformations such as rotation, horizontal and vertical zoom, horizontal and vertical skew, horizontal and vertical translation can be explicitly selected and learned (see Figure 1).
In the remainder of the paper, we review related work in Section 2.
We then compare the difference between GAN, InfoGAN and the proposed method in Section 3.
We introduce the ADIS-GAN in
Section 4, while in Section 5, we show numerical results showing the axis-aligned and scalable disentangled representation learned by ADIS-GAN.
We offer concluding remarks in Section 6.
Our contributions:
To the best of our knowledge, ADIS-GAN is the ﬁrst algorithm that can disentangle an entire afﬁne transformation, including rotation, horizontal and vertical zoom, horizontal and vertical skew, horizontal and vertical translation in a self-supervised manner.
The disentangled representations obtained by ADIS-GAN are axis-aligned.
The advantages are two-folds: a.
The attributes to be learned can be pre-deﬁned, which saves the effort to identify the attributes after the training.
b. Desired but non-dominant attributes can be learned, in parallel with the dominant attributes.
The disentangled representations obtained by ADIS-GAN are scalable.
The scalability property makes it possible to make a trade-off between the compactness and expressivity of the learned representation.
R ELATED LITERATURE
Recent approaches learning disentangled representations are largely based on Variational Autoencoders (VAEs) Kingma & Welling (2013) and InfoGAN Chen et al.
In a standard VAE , observation x is encoded to latent representation z from variational distribution Q ( z | x ) using a deep neural network.
In the generative process, the latent variable z is sampled from a prior distribution
P ( z ) and uses another deep neural network to reconstruct the observation x from conditional distribution P ( x | z ) .
To achieve the disentanglement, a factorized aggregated posterior (cid:82) x Q ( z | x ) P ( x ) dx is encouraged during the training.
In a standard GAN Goodfellow et al.
(2014), latent representation z is sampled from a prior distribution P ( z ) .
The fake data x fake is generated by z from distribution
P ( x | z ) .
A discriminator is introduced to encourage the generated data x fake to be close to the observation x real .
To achieve disentanglement, InfoGAN Chen et al.
(2016) proposes to maximize the mutual information between a subset c of latent representation z and the generated data x fake .
Recently, much attention has been paid to regularizers that promote disentanglement.
The β -VAE
Higgins et al.
(2017) encourages the disentanglement by increasing the weight of the KL regularizer, thus promoting the factorization of the posterior Q ( z | x ) .
Both FactorVAE Kim & Mnih (2018) and β -TCVAE Chen et al.
(2018) penalize the total correlation, while the former relies on adversarial training and the latter directly calculates the total correlation through the decomposition of the β -VAE objective function. The HFVAE Esmaeili et al.
(2019) proposes a two-level hierarchical objective to control relative degree of statistical independence.
In the ChyVAE Ansari & Soh (2019), an inverse-Wishart (IW) prior on the covariance matrix of the latent code is augmented to promote statistical independence.
The DIP-VAE Kumar et al.
(2018) penalizes the difference between the aggregated posterior and a factorized prior.
In the AnnealedVAE Huang et al.
(2018), the encoder can concentrate on learning individual factors and variation by gradually increasing the bottleneck capacity.
The IB-GAN Jeon et al.
(2019) is an extension to InfoGAN rooted in information bottleneck theory, which includes a mutual information upper bound and forms a mutual information bottleneck.
The InfoGAN-CR Lin et al.
(2019) adds a contrastive regularizer on top on InfoGAN, that compares the changes between the image and latent space.
Even though the aforementioned methods have achieved better disentanglement performance compared to the baseline established by VAE and InfoGAN, none of them have successfully disentangled an entire afﬁne transformations in a scalable and axis-aligned way, which is a desirable property in the imaging domain.
Moreover, our afﬁne regularizer is orthogonal to those approaches, thus makes it possible to integrate our afﬁne regularizer with other methods.
In Gidaris et al.
; Chen et al.
; Wang et al.
; Zhang et al.
(2019), selfsupervised regularization is applied, where they compare the difference of images before and after the afﬁne/projective transformation.
The transformation loss is deﬁne as: L = || M ( θ (cid:48) )
− M ( θ ) ||and a parameterized matrix M ∈ R 3 × 3 .
We make a more speciﬁc assumption about the matrix and further decompose it to achieve disentanglement (see Section 4).
B ACKGROUND : GAN, I NFO GAN AND ADIS-GAN
In a standard GAN Goodfellow et al.
(2014), latent representation z is sampled from a prior distribution P ( z ) .
The fake data x fake is generated from z from conditional distribution P ( x | z ) .
A discriminator is introduced to encourage the generated data x fake to be close to the observation x real :
[Equation]
To achieve disentanglement, InfoGAN Chen et al.
(2016) maximizes the mutual information between a subset c of latent representation z and the generated data x fake :
[Equation]
To explicitly disentangle afﬁne transformation attributes in a scalable and axis-aligned manner,
ADIS-GAN has one additional term called afﬁne regularizer on top of InfoGAN:
[Equation]
A FFINE REGULARIZER
Compared to InfoGAN, the major difference of ADIS-GAN is the afﬁne regularization loss L afﬁne (see Figure 2). To calculate L afﬁne , three modiﬁcations to the network have been made. 1: Besides generating x fake and calculating mutual information loss I ( c (cid:48) ; x fake ) like InfoGAN, the semantic latent vector c of ADIS-GAN has an additional task: to generate random afﬁne transformation parameters. We now rename the latent vector c as c realtransform to describe the state before and after the afﬁne transformation, where superscript “real” stands for the initial state before the transformation and subscript “transform” stands for the ﬁnal state after the transformation, more details regarding the transformation states will be introduced in Section 4.2.
c realtransform is sampled from a uniform distribution, then converted to an afﬁne matrix M realtransform using Flow 1 (see Figure 4). 2: Afﬁne transformation augmented image x transform is introduced. x transform is obtained by multiplying M realtransform and x real sampled from training data (see Figure 3). Unlike InfoGAN where “ x real ” is the positive sample fed to the discriminator.
In ADIS-GAN,“ x transform ” is the positive sample fed to the discriminator. This guarantees the afﬁne transformations are observed by the network. 3: Afﬁne reugularization loss L afﬁne is calculated by comparing the ground truth afﬁne transformation latent vector c realtransform and the reconstructed afﬁne transformation latent vector c real (cid:48) transform , which is obtained by calculating the relative difference between x real and its afﬁne augmented pair x transform (see Figure 3).
[Equation]
x fake is the generated image, x real is the image sampled from training dataset, x transform is the afﬁne transformed image. The generation process of x transform is demonstrated in
Figure 3. z is latent noise sampled from normal distribution. c realtransform is the semantic latent vectors sampled from uniform distribution.
c realtransform has two usages: 1.
It is used to generate x fake and compute mutual information loss. It is used to generate x transform together with x real and compute afﬁne regularization loss (see Figure 3).
The meaning of the superscript “real” refers to the initial state before the transformation and subscript “transform” refers to the ﬁnal state after the transformation. I ( c (cid:48) info ; x fake ) is the mutual informaion loss. L adv ( D, G ) is the GAN loss. L afﬁne is the afﬁne regularization loss.
Figure 3 illustrates how to calculate the afﬁne regularization loss L afﬁne . The working principle of afﬁne regularizer is illustrated in Figure 3. First, the latent vector c realtransform is sampled from a uniform distribution.
It is converted to an afﬁne matrix M realtransform through “Flow 1” (see Section 4.1.1). Next, the real image “ x real ” sampled from training data is afﬁne transformed with M realtransform to obtain x transform .
Both x real and its transformed pair x transform are encoded to latent vectors c basisreal and c basistransform respectively. The encoded latent vectors c basisreal and c basistransform are further converted to afﬁne matrix M basisreal and M basistransform through “Flow 1” (see Section 4.1.1).
Then the relative transformation matrix M real (cid:48) transform is computed using both M basisreal and M basistransform (see Section 4.2 Equation 9).
Finally, the reconstructed latent vector c real (cid:48) transform is obtained using M real (cid:48) transform through “Flow 2” (see Section 4.1.2).
Here we use rotation ( θ ), horizontal and vertical zoom ( p, q ), horizontal and vertical translation ( x, y ) as an example (more afﬁne parameter combinations can be found in Appendix H).
The afﬁne regularizer loss is L afﬁne
[Equation]
You may refer to Algorithm 1 and Figure 3 for more details.
[Figure]
Input: Sampled images: x latent vectors c i
i
Output
L
Matrix from sampled latent
M y }
Afﬁne transformation: x M x Latent vectors encoded from images: c basistransform = Encoder ( x transform ) , c basisreal =
Encoder ( x real ) Matrices from encoded latent vector: M basistransform =
Flow 1 ( c ( i ) basistransform ) , M basisreal = Flow 1 ( c ( i ) basisreal ) , i ∈ { θ, p, q, x, y } Relative transform matrix: M real (cid:48) transform =
M basistransform × M basisreal − Reconstructed latent vectors: c ( i ) real (cid:48) transform =
Flow 2 ( M real (cid:48) transform )
Afﬁne reuglarizer loss: L afﬁne =
(cid:80)
i || c
( i ) realtransform − c ( i ) real (cid:48) transform || 22 , i ∈ { θ, p, q, x, y } c c Norm LA
Norm AL ˜ c ˜ c M Initialization MLE M
M Flow 1: Flow 2:  C ONVERSION BETWEEN LATENT VECTORS AND AFFINE MATRIX 4.1. L ATENT VECTORS TO AFFINE MATRIX : F LOW
To conduct afﬁne transformation and calculate the relative difference, we need to convert latent vector c to an afﬁne matrix M . The “Flow 1” in Figure 4 illustrates the process. Norm LA (Latent to Afﬁne): First, we need to normalize the latent vector c to the afﬁne transformation parameter ˜ c . For instance, if we set the afﬁne transformation range as rotation θ ∈ [ − π/ 10 , π/ 10] , horizontal and vertical zoom p, q ∈ [0 . 8 , 1 . 2] , horizontal and vertical translation x, y ∈ [ − 0 . 2 , 0 . 2] , and a 5-d latent vector c is sampled from uniform distribution Unif ( − 1 , 1) .
Norm LA (Latent to Afﬁne) refers to the following normalization: ˜ c ( θ ) = c ( θ ) × ( π/ 10) , ˜ c ( p ) = c ( p ) × 0 .
2 + 1 , ˜ c
( q ) =
c ( q ) × 0 .
2 + 1 , ˜ c
( x ) =
c ( x ) × 0 .
2 , ˜ c
( y )
=
c ( y ) × 0 .
M Initialization : Next, the afﬁne parameter ˜ c needs to be properly arranged in the afﬁne matrix M .
By default, the afﬁne matrix M is organized as in Equation 4.
[Equation]
[Equation]
[Equation]
{+1 0 x 0y 
+}
[Equation]
4.1. AFFINE MATRIX TO LATENT VECTORS : FLOW
To obtain the reconstructed afﬁne transformation latent vector “ c real (cid:48) transform ”, we need to convert the afﬁne matrix to latent vector.
The “Flow 2” in Figure 4 illustrates the process.
MLE (Maximum Likelihood Estimation) :
First, we need to calculate the afﬁne parameter ˜ c using maximum likelihood estimation (see Appendix A).
[Equation]
[Equation]
where i ∈ θ, p, q, x, y .
Norm AL (Afﬁne to Latent):
Next, we need to normalize the afﬁne parameter ˜ c to latent vector c .
If we apply the same afﬁne transformation setting as mentioned in Section 4.1.1 Flow 1.
Norm AL (Afﬁne to Latent) refers to the following normalization: c ( θ ) =
˜ c
( θ ) × ( 1 0 /π ) , c ( p ) =
(˜ c ( p ) − 1) × 5 , c ( q ) =
(˜ c ( q ) − 1) × 5 , c ( x ) =
˜ c ( x ) × 5 , c ( y ) =
˜ c
( y ) × 5 .
4. R ELATIVE AFFINE TRANSFORMATION
[Equation] [Figure] [Equation]
[Figure]
[Equation]
We assume, without loss of generality, that images can be expressed as the multiplication of the afﬁne transformation M basisreal
[Equation]
3 (afﬁne matrix is a 3 × 3 matrix by default) and an image basis x basis (see Figure 5):
[Equation]
The superscript “basis” of M basisreal stands for the initial state and subscript “real” stands for the transformed state of the image x .
The image basis x basis is the average manifold of all images within the same category.
For instance, the digits “0” , “1”, ... “9” in MNIST are different categories.
x basis does not refer to a particular image in the dataset and it is purely learned from the data.
If we purposely apply a known afﬁne transformation M realtransform on a sampled image x real , we can obtain an new image x transform :
[Equation]
According to the deﬁnition in equation 6, the transformed image can also be expressed as the multiplication of the M basistransform and the image base x basis :
[Equation]
By solving the simultaneous formula from equation 6 to 8:
[Equation]
M realtransform is the relative afﬁne transformation between afﬁne matrices M basistransform and M basisreal .
The goal of the experiments in this section is to investigate, both qualitatively and quantitatively, the disentangled representations obtained by ADIS-GAN.
The axis-alignment of the disentangled representations is demonstrated on CelebA dataset Liu et al.
(2015), while the scalability of the disentangled representation is illustrated on MNIST dataset.
Next quantitative results for ADISGAN are presented and compared to benchmarks on the dSprites Matthey et al.
(2017) dataset.
The parameters of the afﬁne transformation are selected as follows: rotation range: [ − 18 ◦ , 18 ◦ ] , zoom range: [0.8, 1.2], and translation range: [-0.2, 0.2].
For all the experiments we use Adam optimizer with the learning rate of 0.0002 for discriminator and 0.0001 for generator and encoder.
The batch size is 128 for MNIST, 64 for dSprites, and 16 for CelebA.
The regularization weight in Equationis 1 for both λ and β .
You may refer to Appendix G for network structure details.
5. Q UALITATIVE R ESULTS
The axis-alignment and scalability properties are achieved by selecting different matrix initializations and their corresponding maximum likelihood estimation.
You may refer to Appendix I for various ways of matrix initializations and their maximum likelihood estimation.
5.1. A XIS - ALIGNMENT
In CelebA dataset, typical dominant attributes are azimuth, sunglasses, emotion, etc.
Existing methods Chen et al.
; Kim & Mnih (2018);
Chen et al.
(2018) have successfully disentangled those attributes (see Appendix E).
However, non-dominant attributes such as the roll, width and length of the face and relative position of face in the frame are rarely tackled.
Thanks to the axis-aligned property, ADIS-GAN can explicitly learn those non-dominant attributes (see Figure 6, 7 and 8).
It is interesting to note that ADIS-GAN does not rigidly perform afﬁne transformations on the given images but understands the relationship between different facial components.
For example, when we try to shorten the length of the face (vertical zoom and translation, see left images on both Figure 7 and 8), ADIS-GAN knows how to complete the bottom of the images with a neck to make the images look realistic.
To disentangle facial attributes on the CelebA data, we choose to model the latent space with one categorical code, c cat ∼ Cat ( K = 10 , p = 0 . 1) and 5 continuous codes c cont ∼ Unif(-1,1).
5.1. S CALABILITY
Scalability refers to the ﬂexibility to encode one attribute via one latent vector or, alternatively, to decompose one attribute into sub-attributes and encode them by multiple latent vectors.
For example, both zoom and translation attributes can be encoded by one latent vector, or they can be decomposed horizontally and vertically and encoded by two latent vectors (see Figure 9).
[Figure]
5. Q UANTITATIVE R ESULTS
To disentangle object style on dSprites we choose to model the latent space by one categorical code c cat ∼ Cat ( K = 3 , p = 0 . 33) and 4 continuous codes c cont ∼ Unif(-1,1) that represent rotation, zoom, horizontal and vertical translation.
To the best of our knowledge, ADIS-GAN is the ﬁrst algorithm that can disentangle shape attributes with categorical codes on dSprites datasets.
We have observed that shape disentanglement is easier for larger images.
We conducted experiments on the [Table] “Four Shapes” dataset (see Appendix D).
Compared to dSprites, the major difference of the “Four Shapes” dataset is that the images are larger.
We found that ADIS-GAN can successfully disentangle shape attribute on “Four Shapes” dataset with categorical codes (see Appendix D).
The image enlargement procedures the disentanglement of dSprites
[Table]
C ONCLUSION
This paper introduces the Afﬁne Disentangled GAN (ADIS-GAN) that can explicitly learn the afﬁne transformations via disentangled representations.
In contrast to earlier approaches to disentanglement, where inductive biases are not explicit, the disentangled representation obtained by ADISGAN is axis-aligned and scalable.
The afﬁne regularizer introduced in this paper can be applied to other methods such as the VAE family Kingma & Welling (2013); Higgins et al.
Kim &
Mnih (2018); Chen et al.
Another attractive property of the afﬁne regularizer is that it supports the possibility to utilize expert knowledge as the inductive bias, as it is model based and exploits maximum likelihood estimation.
For example, we can disentangle the 2D afﬁne transformations with the decomposed relationships.
Task-speciﬁc explicit regularizers may provide an alternative pathway for disentanglement compared to existing general implicit regularizers.