M ANIFOLD L EARNING AND A LIGNMENT WITH G ENERATIVE A DVERSARIAL N ETWORKS
Anonymous authors
Paper under double-blind review
A BSTRACT
We present a generative adversarial network (GAN) that conducts manifold learning and alignment (MLA):
A task to learn the multi-manifold structure underlying data and to align those manifolds without any correspondence information.
Our main idea is to exploit the powerful abstraction ability of encoder architecture.
Speciﬁcally, we deﬁne multiple generators to model multiple manifolds, but in a particular way that their inverse maps can be commonly represented by a single smooth encoder.
Then, the abstraction ability of the encoder enforces semantic similarities between the generators and gives a plausibly aligned embedding in the latent space.
In experiments with MNIST, 3D-Chair, and UT-Zap50k datasets, we demonstrate the superiority of our model in learning the manifolds by FID scores and in aligning the manifolds by disentanglement scores.
Furthermore, by virtue of the abstractive modeling, we show that our model can generate data from an untrained manifold, which is unique to our model.
I NTRODUCTION
Real-world datasets generally have multi-manifold structure.
Smoothly varying features such as the size or the pose of an animal introduce smooth manifold structure, while the existence of discrete features such as species makes the structure not a single manifold but multiple disconnected manifolds (e.g., one manifold per species)
(Khayatkhoei et al., 2018).
Importantly, these manifolds are correlated to each other in their structure since the transformation rules according to the size (stretching the foreground patch) or the pose (rotating the patches of body parts) are similar even for different species.
Hence, our desire is to build a model that learns not only the multi-manifold structure but also the correlations between the manifolds.
As for the learning of manifold structure, generative adversarial networks (GANs) (Goodfellow et al., 2014) have been known for their remarkable performance.
In particular, recently proposed
GAN models have successfully demonstrated multi-manifold learning by using multiple generator networks (Khayatkhoei et al., 2018; Ghosh et al., 2017; Hoang et al., 2018) or giving a mixture density on the latent space (Xiao et al., 2018; Gurumurthy et al., 2017).
However, none of these models have taken the correlations between the manifolds into account.
On the other hand, a ﬁeld of research named manifold alignment (Ham et al., 2003) has its primary concern in learning the correlations between the manifolds.
In particular, unsupervised manifold alignment (Cui et al., 2014; Wang & Mahadevan, 2009) aims to learn plausible correspondences between the data points of different manifolds, without any ground-truth correspondence information is given.
However, most existing methods focus only on ﬁnding the correspondences not on learning the structure of the manifolds itself.
In this work, we propose a model that performs both of the tasks, which we call as manifold learning and aligning GAN (MLA-GAN).
MLA-GAN is similar to the GANs with multiple generators (Khayatkhoei et al., 2018; Ghosh et al., 2017; Hoang et al., 2018)
but the generators are restricted such that their inverse maps can be represented by a single deep encoder.
Due to the abstracting property of deep encoders, this restriction guides the inverse maps to learn semantically similar mappings, and the model achieves a semantically plausible manifold alignment.
The main contributions of this work can be summarized as follows.
[Figure]
Latent Space 𝑍
S i ze P
o s e 𝑓 $(&) 𝑓 $(() 𝑓 $& )&
S i z e P
o s e 𝑓 $( )& 𝑟 +(&) 𝑟
+(()
S i ze P
o s e 𝑓 $(&) 𝑓 $(() ℎ +
S i ze P
o s e 𝑓 $& )&
[Figure]
$( )
[Figure]
• The MLA problem: Up to our knowledge, MLA-GAN is the ﬁrst model that performs both multi-manifold learning and alignment with real-world datasets.
It shows state-ofthe-art FID scores in data generation and plausible alignments with disentangled features.
• Generalization to an untrained manifold: Due to the abstracting nature of an encoder, the smooth features can be inferred even for data lying on an untrained manifold.
Exploiting this, one can generate other data points lying on this manifold.
• Easily applicable : Due to the new regularizer we propose, the inverse maps used for restricting the generators do not need to be actually computed.
This makes MLA-GAN easily applicable to any
existing GAN architectures.
M ANIFOLD L EARNING AND A LIGNMENT WITH GAN 2. B ACKGROUNDS
Multi-Manifold Learning GANs Conventional GANs (Goodfellow et al., 2014) model data by transforming a latent-space distribution p ( z ) with a generating map f G : Z → X .
If f G is smooth and injective 1 , the image of f G is a smooth manifold M ⊂ X , thus the data are modeled as they are lying on this manifold.
To deal with multiple manifolds, recent extensions (Khayatkhoei et al., 2018; Ghosh et al., 2017; Hoang et al., 2018) employ multiple generating maps { f ( i ) G } Ai =1 where each f ( i ) G accounts for the i -th manifold M ( i ) .
Unsupervised Manifold Alignment Most existing methods formulate the manifold alignment problem as ﬁnding embeddings of manifolds into a common low-dimensional space Y .
This set of embeddings { r ( i ) E : M ( i ) → Y } i essentially deﬁnes an alignment of manifolds; data points that are mapped to the same point in Y correspond to each other.
To achieve a plausible alignment, the embeddings are trained to minimize several loss terms guiding geometrical similarity (see Sec. 3).
2. M AIN I DEA
To model multiple manifolds, we use multiple generating maps { f ( i ) G : Z → M ( i ) } i as with the aforementioned GAN models.
Interestingly, the inverses 2 of these maps { ( f ( i ) G ) − 1 : M ( i )
→ Z } i can be regarded as a set of embeddings (where Y is replaced by Z ), which means that the manifold alignment is automatically deﬁned as we deﬁne the manifolds .
However, this alignment is not at all semantically plausible unless properly regulated.
Here, we regulate it by restricting the generating maps such that their inverses to be commonly represented by a single encoder h E : X → Z .
Formally, we restrict the generating maps as follows:
[Equation]
Here, H is a certain set of functions (to be clearer soon), and h E being an element of H is an important constraint.
Without this constraint, one can always ﬁnd h E satisfying the above condition, for any sets of { f ( i ) G } i , because h E is merely a union of functions { ( f ( i ) G ) − 1 } i extending the domain from (cid:83)
i M ( i ) ⊂
X to X .
However, if H is, for example, a set of smooth functions and h E ∈ H , then { ( f ( i ) G ) − 1 } i will be restricted to be similar to each other and the alignment will become more plausible.
In our model, H is deﬁned as a set of functions represented by deep encoder networks .
We explain the intuition behind this design in what follows.
Intuition Consider a dataset containing images of cats and dogs.
It is not difﬁcult to imagine a deep encoder network that takes an image from this dataset and outputs the size and pose regardless of the species.
At least, we could have trained this encoder easily if the size and pose labels were given.
Also, even if we train the encoder using the cat data only, we still expect it works fairly well for the dogs due to the powerful abstraction and generalization abilities of deep encoders.
We want to exploit these abilities of deep encoders for aligning the manifolds.
If we let H be a set of deep encoders, its element h E , once well-trained, will abstract the species away and outputs the size and pose.
This gives a nice manifold alignment, where a cat and a dog images with the same size and pose correspond to each other.
Remaining Question Is H being a set of deep encoders enough to obtain the abstraction and generalization?
An answer to this question is related to how deep networks generalize so well despite the large number of parameters they have.
Several works conjecture that it is due to the SGD (Zhang et al., 2016; Brutzkus et al., 2018), but this is not a fully developed theory yet.
At least, we can assume that deep encoders learn simpler, generalization-friendly maps among others, based on various cases (e.g., VGG-Net (Simonyan & Zisserman, 2014), Fast R-CNN (Girshick, 2015)).
2. MLA-GAN
What remains is to implement the generating maps with deep neural networks.
The main challenge is to make the deep networks comply with the constraint given in Eq.
In this section, we explain how this challenge can be achieved in the proposing MLA-GAN model in detail.
We ﬁrst focus on a single linear layer and prove the following (detailed in Proposition 1):
If linear generating maps are deﬁned with the same weight and the same tangential components of bias, their inverses can always be represented by a single linear encoder.
Thus, the maps comply with Eq. 1.
Then, to
realize the above
we introduce a new form of regularizer.
After, we move on to the entire network and derive MLA-GAN by applying the same principles as the linear layer
2.3. S INGLE L INEAR L AYER
Consider a set of linear generating maps { f ( i ) G : Z → M ( i ) ⊂
X } i ; each of the maps is deﬁned as f
[Equation]
𝑀 (#) 𝑀
(%) 𝒂 (𝟏) 𝒂 (𝟐) 𝑧
𝑈𝑧 𝒙 (𝟏) =
𝑼𝒛 + 𝒂 (𝟏) 𝒙 (𝟐) =
𝑼𝒛 + 𝒂 (𝟐)
G e n er a t i o n 𝑎 ∥ 𝑥 3(#) 𝑥 3(%) 𝑥
∥ 𝒙 (𝟐) 𝑊
# 𝑊 % 𝒙 (𝟏) 𝑀 (#) 𝑀 (%) 𝑂 (Origin)
𝑊 6 𝑥 ∥ =
𝑊 6 𝑥 (#) 𝑧 =
𝑊 6 𝑥 ∥ + 𝑏 𝑏
E n c o d i n g 𝑥 ∥ =
𝑊 6 𝑥 (%) 𝑈 #
[Figure]
where U ( i ) ∈ R
d X × d Z ( d X > d Z ) and a ( i ) ∈
R
d X are the weight and bias parameters respectively.
We assume U ( i ) is a full-rank matrix (rank- d Z ) such that f ( i ) G is injective (see Footnote 1).
Then, the inverse ( f ( i ) G ) − 1 : M ( i ) → Z can be derived as
[Equation]
(2) where ( U ( i ) )
+ := (cid:0) ( U ( i ) )
(cid:62) U (
[Equation]
) (cid:1) − 1 ( U ( i ) )
[Equation]
denotes the pseudo-inverse of U ( i ) .
As noted in Eq.
1, our desire is to restrict the inverse maps { ( f ( i ) G ) − 1 } i such that they can be represented by a single encoder h E : X → Z .
One simple way to achieve this is to use the following proposition.
Proposition 1.
If the linear generating maps { f ( i ) G ( z ) } i are restricted to have the same weight
U and to have the same tangential components of bias a (cid:107) (i.e., a ( i ) =
a (cid:107) + a ( i ) ⊥ ), then their inverses { ( f ( i ) G ) − 1 } i can be represented by a single linear encoder h E ( x ) :
= W (cid:62) x
+ b where
W =
U ( U (cid:62) U ) − 1 and b = − W (cid:62) a (cid:107) .
Proof.
Note the bias a ( i ) can be decomposed into the tangential component a ( i ) (cid:107) and the normal component a ( i ) ⊥ to the column space of U (cid:62) .
Substituting the condition a ( i ) = a (cid:107) + a ( i ) ⊥ and the same weight U , Eq. 2 is written as
[Equation]
Bias Regularizer When implementing Proposition 1, making { U ( i ) } i the same is as trivial as setting the same weight U for all f ( i ) G , but making { a ( i ) (cid:107) } i the same is not trivial since the tangential direction keeps changing while training.
One solution would be to use a regularizer minimizing the sum of the variance: trace ( cov ( a ( i ) (cid:107) )) .
However, computing this term is intractable due to the inversion ( U (cid:62) U ) − 1 inside of a ( i ) (cid:107) =
U ( U (cid:62) U ) − 1 U (cid:62) a ( i ) .
Theorem 1.
The following inequality holds:
[Equation]
where { λ k } d z k =1 denotes the eigenvalues of U (cid:62) U and H ( · ) denotes harmonic mean.
Proof.
See Appendix B.
As the harmonic mean in Theorem 1 is constant from the perspective of a ( i ) (cid:107) , we can minimize the original term by minimizing the upper bound instead.
With an additional log function to match the scale due to the dimensionality, we propose the upper bound as a regularizer to make a ( i ) (cid:107) the same:
Regularizer: R bias =
[Equation]
(cid:16)
[Equation]
(cid:16)
[Equation] [Equation]
E NTIRE N ETWORK
A single linear layer is, of course, insufﬁcient as linear generators can only model the linear manifolds and a linear encoder can align the manifolds only linearly by removing the normal components.
So, we implement the generating maps with deep networks to model complex nonlinear manifolds.
As usual, the deep networks are constructed by interweaving linear layers and nonlinear activations, and optionally batch-normalization layers.
But, each linear layer here implements the conditions of
Proposition 1.
That means, for the l -th linear layer of each deep generating map
f
( i ) G , their weights are set as the same, denoted as U l , and their biases { a ( i ) l } i are regularized to have the same tangential components using Eq. 3.
Note that the above set of deep generating maps comply with Eq. 1.
As all the linear layers, nonlinear activations (we use LeakyReLU; see Appendix C) and batch-norm layers are injective, the deep generating maps have inverses.
These deep inverse maps can be represented by a single deep encoder, as the inverses of all the component linear maps can be represented by single encoders.
Training As above, let us denote the weight of the l -th linear layer as U l and the biases as { a ( i ) l } Ai = .
Then, we can express the data generating distribution as an ancestral sampling:
[Equation]
where
z ∼ p ( z ) , i ∼ π i
[Equation]
Here, π i stands for the probability of selecting the i -th bias.
This probability could be also learned using the method proposed in Khayatkhoei et al.
, but it is beyond our scope and we ﬁx it as 1 /A .
Now, denoting the real data distribution as p R and the fake distribution that the above sampling presents as p G , we deﬁne our GAN losses as:
MLA-GAN Loss
[Equation]
where L G and L D are the generator and the discriminator losses respectively and λ is a regularization weight.
We use Wasserstein GAN (WGAN)
Arjovsky et al.
so the discriminator D ( x ) is restricted to a k -Lipschitz function.
Encoding Due to the bias regularizer, we do not need to concretize the encoder h E by inverting f
( i ) G during training.
But, when encoding is needed, we can obtain the latent codes and biases by minimizing the distance along with the bias regularizer as follows (as suggested in Ma et al.
[Equation]
Importantly, one can encode an image that is considerably different from the trained images using this method.
By virtue of the abstraction ability of the encoder, the encoded latent code is actually semantically plausible.
Moreover, as the encoded biases fully describe the generating map, one can virtually generate any data lying on the same manifold.
This is unique to our model and to be further discussed in Sec. 4.3.
[Table]
R ELATED W ORK
Multi-Manifold Learning GANs There exist roughly three different methods to handle multiple manifolds with GANs (see also Appendix A).
Using multiple generators (DMWGAN (Khayatkhoei et al., 2018); Ghosh et al.
; Hoang et al.
(2018)) is the most simplest one, but it cannot align the manifolds as the generators are independent to each other without any regulations.
Giving a mixture density for the latent space prior (Xiao et al., 2018; Gurumurthy et al., 2017) is another method; although this method uses a single generating map, it hardly aligns the manifolds since the manifolds are modeled by different regions in the latent space where the mappings are effectively different to each other.
The last method (InfoGAN (Chen et al., 2016)) introduces discrete latent variables and concatenate them with the original continuous latent variables.
Here, the discrete information is slowly blended in, layer by layer, and modulates the generating map.
As the overall structure is shared among the manifolds to some degree, this method does align the manifolds; but as it is not explicitly guided, the performance is not as good as MLA-GAN (Table 1).
Unsupervised Manifold Alignment The existing methods (Cui et al., 2014; Wang & Mahadevan, 2009) ﬁnd a correspondence between the points of different manifolds by learning embeddings, with a consideration on geometrical similarity.
As the geometry is captured by adjacency matrices, one shortcoming is that the alignment can be done only in a point-to-point manner.
Moreover, as the embeddings are modeled as linear functions, it is inevitable to lose information about the manifolds.
Deep generative models involving an encoder Utilization of an encoder is a notable feature of
MLA-GAN, but there are several other deep generative models using encoders, such as adversarial auto-encoders (Makhzani et al., 2015), variational auto-encoders (Kingma & Welling, 2013; Higgins et al., 2016) and BiGAN (Donahue et al., 2016).
However, there is a large difference: The encoders of these models captures all the necessary information in order to reconstruct the data, while the encoder of MLA-GAN removes the discrete information in the latent space to abstract the manifolds.
E XPERIMENTS
Datasets We experiment on MNIST (Lecun et al., 1998), 3D-Chair (Aubry et al., 2014) and UTZap50k (Yu & Grauman, 2014) image datasets.
3D-Chair contains 1393 distinct chairs rendered from 62 different viewing angles (total 86,366 images); in experiments, only front-looking 44, images are used and rescaled to 64x64 grayscale images.
UT-Zap50k contains images of 4 different types of shoes (total 50,025 images); the images are rescaled to 32x32.
Model Architecture We use DCGAN (Radford et al.
, 2015)-like model architectures for all the datasets (see Appendix C for the complete information).
For each of the linear layers in the generator, the number of biases A are set as 10 (MNIST), 20 (3D-Chair) and 4 (UT-Zap50k).
Although our multi-biased linear layer can be applied to both fully-connected and transposed-convolution layers, we apply it only to the former.
This is sufﬁcient for our purpose since discrete features rarely exist for such small sized kernel patches.
In the discriminator, we use spectral normalization (Miyato et al., 2018) to achieve the k -Lipschitz condition for WGAN.
For training and encoding, Adam (Kingma & Ba, 2014) is used with the defaults except for the learning rate, 0.0002.
M ULTI -M ANIFOLD L EARNING
MLA-GAN shows great performance in multi-manifold learning and alignment, both qualitatively and quantitatively.
Looking at Fig.
3 row-wise, we can see that MLA-GAN learns distinct manifolds well, where each manifold accounts for a different discrete class: e.g., different digits; rolling vs. arm chairs; boots vs. ﬂat shoes.
Column-wise, we can see that the smooth features are well aligned among the manifolds:
e.g., stroke weight, slant of numbers; viewing angle of chairs; colors of shoes.
Note that these alignments are qualitatively much better than that of the existing alignment methods given in Fig.
D.2.
For quantitative evaluation, we compute FID scores (Heusel et al., 2017), which is widely used to measure the diversity and quality of the generated image samples, reﬂecting the manifold learning performance overall.
Table 1 shows that our model gives better or comparable scores than others.
4. M ANIFOLD A LIGNMENT AND D ISENTANGLED F EATURES
To further investigate the manifold alignment performance, we examine how much the learned latent features are disentangled.
We take a few images from the dataset and manually change one of the smooth features that corresponds to a known transformation (e.g., sheer transform).
Then, we encode these images to the latent codes using our model, analyze the principal direction of the change, and compute the linearity of the change as the disentanglement score (see Appendix E).
Fig.
4 shows that the learned smooth features are well disentangled along the principal changing directions.
Table 1 shows that our model gets better scores than other models, but sometimes not as good as β -VAE.
However, it should be noted that the FID scores of β -VAE is signiﬁcantly worse.
4. S TYLE T RANSFER
We demonstrate a style-transfer between images using our model, by matching the latent features (Fig. 5).
Speciﬁcally, we encode the source and target images, then use the inferred biases of the source and the inferred latent codes of the target to generate the style-transferred image.
Interestingly, MLA-GAN is even able to style-transfer the images that are off the trained distribution (see the red boxes).
This is an exclusive property of MLA-GAN due to the encoder.
If the encoder generalizes well (which apparently does), it can recognize the smooth features of an image even if the image has noises such as added frames or color modiﬁcation.
Once the smooth features are recognized, the model knows how to transform this image smoothly, similar to the original data, but on this new manifold involving noises (as described in Eq. 4 and below).
C ONCLUSION
We proposed MLA-GAN that performs multi-manifold learning and alignment.
To our knowledge,
MLA-GAN is the ﬁrst model that performs both of the tasks for real-world datasets, and it mostly showed superior performance.
Unique to MLA-GAN is that it utilizes the abstraction ability of an encoder, from which it showed its potential to infer the untrained manifolds in the style-transfer experiment.
If it is trained with larger datasets such as ImageNet in the future, we expect MLA-GAN would become a more versatile tool that can infer the manifold structure of general images.
Appendix A C OMPARISON OF V ARIOUS GAN M ODELS
[Figure]
B T HE P ROOF OF T HEOREM
Theorem 1.
The following inequality holds
[Equation]
where { λ k } d z k =1 are the eigenvalues of U (cid:62) U and H ( · ) denotes a harmonic mean.
Proof.
Note that
[Equation]
where the second and the fourth lines use the deﬁnition of the covariance, third line is obtained from the cyclic property of trace and the last line is obtained from the Cauchy-Schwarz inequality of the positive semi-deﬁnite matrices.
Thus,
[Equation]
where { λ d } D d =1 are the eigenvalues of U (cid:62) U and H ( · ) denotes the harmonic mean.
C M ODEL A RCHITECTURE AND E XPERIMENTING E NVIRONMENTS
We used machines with one NVIDIA Titan Xp for the training and the inference of all the models.
C. MNIST
We use A = 10 distinct decoding biases in the model.
In the training, we set the regularization weight λ = 0 .
and use the Adam optimizer with learning rate 0.0002.
In the encoding, we use the
Adam optimizer with learning rate 0.1, and the set the regularization weight µ = 0 .
[Table]
C.
[Table]
. N OTES ON THE O THER C OMPARED M ODELS
Overall
[Table]
we match the architecture of other models with our model for fair comparison.
Some differences to note are:
• DMWGAN : We used 10 generators.
Each generator has the same architecture as ours except the number of features or the channels are divided by 4, to match the number of trainable parameters.
Note that 4 is the suggested number from the original paper.
• β -VAE : We used Bernoulli likelihood.
• InfoGAN : Latent dimensions consist of 1 discrete variable (10 categories), 2 continuous variables and 8 noise variables.
C. 3D-C HAIR
We use A = 20 distinct decoding biases in the model.
In the training, we set the regularization weight λ = 0 .
and use the Adam optimizer with learning rate 0.0002.
In the encoding, we use the
Adam optimizer with learning rate 0.1, and the set the regularization weight µ = 0 .
[Table]
C
[Table]
N OTES ON THE O THER C OMPARED M ODELS
• DMWGAN : We used 20 generators.
Each generator has the same architecture as ours except the BatchNorms are removed and the number of features or the channels are divided by 4, to match the number of trainable parameters.
Note that 4 is the suggested number from the original paper.
Note that this was the best setting among what we have tried (division number 2; ones with the BatchNorms).
• β -VAE : We used Bernoulli likelihood.
• InfoGAN : Latent dimensions consist of 3 discrete variables (20 categories), 1 continuous variable and 10 noise variables.
C. UT-Z AP 50 K
We use A = 4 distinct decoding biases in the model.
For the regularization weight in the training, we start with λ = 5 e − 6 then raise to λ = 5 e − 4 after 300 epochs.
[Table]
D M ANIFOLD A LIGNMENT R ESULTS OF THE E XISTING M ETHODS
We show the manifold alignment results of Wang & Mahadevan (2009) for MNIST dataset.
E D ISENTANGLEMENT S CORE
To compute the disentanglement score, we ﬁrst take 500 images from the dataset and manually change one of the smooth features that corresponds to a known transformation.
For example, we change the slant of the MNIST digits by taking a sheer transform.
With 11 different degrees of the transformation, we obtain 5500 transformed images in total.
We encode these images to obtain the corresponding latent codes and subtract the mean for each group of the images (originates from the same image) to align all the latent codes.
Then, we conduct Principal Component Analysis (PCA) to obtain the principal direction and the spectrum of variations of the latent codes.
If the latent features are well disentangled, the dimensionality of the variation should be close to one.
To quantify how much it is close to one, we compute the ratio of the ﬁrst eigenvalue to the second eigenvalue of the
PCA covariance, and set it as the disentanglement score.
F S CORES A CCORDING TO D IFFERENT N UMBER OF G ENERATORS , A
To investigate the effect of the number of generators (or biases), A , we train our model on MNIST with different A values then compute the FID and the disentanglement scores (Fig.
F.3).
It can be seen that our model, regardless of the different A values, performs consistently better than the baseline, WGAN.
G E FFECT OF THE B IAS R EGULARIZER
To examine the effectiveness of our bias regularizer, we visualize the raw values of biases { a ( i ) l } i,l and their (pseudo-)tangential component { U (cid:62) l a ( i ) l } i,l (see Fig.
G
H S AMPLES G ENERATED FROM V ARIOUS M ODELS
[Figure] [Figure]