Original: D UAL SPACE LEARNING WITH VARIATIONAL AUTOENCODERS
Revised: D UAL SPACE LEARNING WITH VARIATIONAL AUTOENCODERS

Original: A BSTRACT
Revised: A BSTRACT

Original: This paper proposes a dual variational autoencoder (DualVAE), a framework for generating images corresponding to multiclass labels.
Revised: This paper proposes a dual variational autoencoder (DualVAE), a framework for generating images corresponding to multiclass labels.

Original: Recent research on conditional generative models, such as the Conditional VAE, exhibit image transfer by changing labels.
Revised: Recent research on conditional generative models, such as the Conditional VAE, exhibit image transfer by changing labels.

Original: However, when the dimension of multiclass labels is large, these models cannot change images corresponding to labels, because learning multiple distributions of the corresponding class is necessary to transfer an image.
Revised: However, when the dimension of multiclass labels is large, these models cannot change images corresponding to labels, because learning multiple distributions of the corresponding class is necessary to transfer an image.

Original: This leads to the lack of training data.
Revised: This leads to the lack of training data.

Original: Therefore, instead of conditioning with labels, we condition with latent vectors that include label information.
Revised: Therefore, instead of conditioning with labels, we condition with latent vectors that include label information.

Original: DualVAE divides one distribution of the latent space by linear decision boundaries using labels.
Revised: DualVAE divides one distribution of the latent space by linear decision boundaries using labels.

Original: Consequently, DualVAE can easily transfer an image by moving a latent vector toward a decision boundary and is robust to the missing values of multiclass labels.
Revised: Consequently, DualVAE can easily transfer an image by moving a latent vector toward a decision boundary and is robust to the missing values of multiclass labels.

Original: To evaluate our proposed method, we introduce a conditional inception score (CIS) for measuring how much an image changes to the target class.
Revised: To evaluate our proposed method, we introduce a conditional inception score (CIS) for measuring how much an image changes to the target class.

Original: We evaluate the images transferred by DualVAE using the CIS in CelebA datasets and demonstrate state-of-the-art performance in a multiclass setting.
Revised: We evaluate the images transferred by DualVAE using the CIS in CelebA datasets and demonstrate state-of-the-art performance in a multiclass setting.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Recent conditional generative models have shown remarkable success in generating and transferring images.
Revised: Recent conditional generative models have shown remarkable success in generating and transferring images.

Original: Speciﬁcally, a conditional variational autoencoder (CVAE) (Kingma et al., 2014) can generate conditional images by learning the latent space Z that corresponds to multiclass labels.
Revised: Speciﬁcally, a conditional variational autoencoder (CVAE) (Kingma et al., 2014) can generate conditional images by learning the latent space Z that corresponds to multiclass labels.

Original: In addition, StarGAN (Choi et al., 2017) and FaderNetworks (Lample et al., 2017) can generate images corresponding to multiple domains by conditioning with domains such as attributes.
Revised: In addition, StarGAN (Choi et al., 2017) and FaderNetworks (Lample et al., 2017) can generate images corresponding to multiple domains by conditioning with domains such as attributes.

Original: However, when the dimension of the multiclass is increased, these models cannot transfer the images corresponding to one arbitrary domain (an element of a multiclass label).
Revised: However, when the dimension of the multiclass is increased, these models cannot transfer the images corresponding to one arbitrary domain (an element of a multiclass label).

Original: The possible reasons are the following.
Revised: The possible reasons are the following.

Original: For simplicity, we consider a binary multiclass classiﬁcation.
Revised: For simplicity, we consider a binary multiclass classiﬁcation.

Original: To transfer an image of a certain class, it is necessary to learn the distributions of the corresponding class.
Revised: To transfer an image of a certain class, it is necessary to learn the distributions of the corresponding class.

Original: That is, assuming that the number of classes in the multiclass is N, conditional models need to create 2 N distributions.
Revised: That is, assuming that the number of classes in the multiclass is N, conditional models need to create 2 N distributions.

Original: However, when N is large, training is difﬁcult as O (2 N ) training samples will be required.
Revised: However, when N is large, training is difﬁcult as O (2 N ) training samples will be required.

Original: Hence, instead of conditioning with labels, we propose DualVAE, which conditions with latent vectors that include label information.
Revised: Hence, instead of conditioning with labels, we propose DualVAE, which conditions with latent vectors that include label information.

Original: DualVAE divides one distribution of the latent space by N linear decision boundaries which need to learn only O ( N ) parameters by adding another decoder p w ( y | z ) to a variational autoencoder (VAE) (Kingma & Welling, 2013).
Revised: DualVAE divides one distribution of the latent space by N linear decision boundaries which need to learn only O ( N ) parameters by adding another decoder p w ( y | z ) to a variational autoencoder (VAE) (Kingma & Welling, 2013).

Original: DualVAE assumes that a label is a linear combination of vectors of the latent space and the dual latent space.
Revised: DualVAE assumes that a label is a linear combination of vectors of the latent space and the dual latent space.

Original: There are two advantages to the DVAE decoder p w ( y | z ) being a linear model.
Revised: There are two advantages to the DVAE decoder p w ( y | z ) being a linear model.

Original: First, DualVAE can easily transfer an image by moving a latent vector toward a decision boundary.
Revised: First, DualVAE can easily transfer an image by moving a latent vector toward a decision boundary.

Original: Next, DualVAE is robust to the missing values of multiclass labels.
Revised: Next, DualVAE is robust to the missing values of multiclass labels.

Original: In addition to this method, we propose the conditional inception score (CIS), a new metric for conditional transferred images.
Revised: In addition to this method, we propose the conditional inception score (CIS), a new metric for conditional transferred images.

Original: Although the evaluation methods often used in the generation models are the Inception Score (IS) (Salimans et al., 2016) and the Fr´echet Inception Distance (Heusel et al., 2017), they are used for evaluating the diversity of images and not suitable for evaluating transferred images conditioned with domains such as attributes or classes.
Revised: Although the evaluation methods often used in the generation models are the Inception Score (IS) (Salimans et al., 2016) and the Fr´echet Inception Distance (Heusel et al., 2017), they are used for evaluating the diversity of images and not suitable for evaluating transferred images conditioned with domains such as attributes or classes.

Original: Therefore, we propose a new metric to evaluate two properties: the ﬁrst property pertains to whether images in one domain are transferred properly to images in another domain; the second property pertains to whether images in one domain transferred to images in another domain can preserve the original properties.
Revised: Therefore, we propose a new metric to evaluate two properties: the ﬁrst property pertains to whether images in one domain are transferred properly to images in another domain; the second property pertains to whether images in one domain transferred to images in another domain can preserve the original properties.

Original: By using the CIS, we compare DualVAE with other methods that can perform image-to-image translations for multiple domains.
Revised: By using the CIS, we compare DualVAE with other methods that can perform image-to-image translations for multiple domains.

Original: In summary, the contributions from this study are as follows: 1) We introduce DualVAE, a method for transferring images corresponding to multiclass labels and demonstrate that images can be transferred quantitatively and qualitatively.
Revised: In summary, the contributions from this study are as follows: 1) We introduce DualVAE, a method for transferring images corresponding to multiclass labels and demonstrate that images can be transferred quantitatively and qualitatively.

Original: We propose the CIS, a new metric that can evaluate transferred images corresponding to multiclass labels.
Revised: We propose the CIS, a new metric that can evaluate transferred images corresponding to multiclass labels.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: Conditional model Several studies have been conducted to generate or transfer images conditioned with labels.
Revised: Conditional model Several studies have been conducted to generate or transfer images conditioned with labels.

Original: For example, conditional VAE (Kingma et al., 2014) is an extension of a VAE (Kingma & Welling, 2013) where latent variables z are inferred using image x and label y , and image x is reconstructed with y , z .
Revised: For example, conditional VAE (Kingma et al., 2014) is an extension of a VAE (Kingma & Welling, 2013) where latent variables z are inferred using image x and label y , and image x is reconstructed with y , z .

Original: Further, a CGAN (Mirza & Osindero, 2014) is a conditional model using a GAN, where a noise z and a class label y are input to the generator, and learning is performed similarly to the GAN using image x corresponding to class label y .
Revised: Further, a CGAN (Mirza & Osindero, 2014) is a conditional model using a GAN, where a noise z and a class label y are input to the generator, and learning is performed similarly to the GAN using image x corresponding to class label y .

Original: FaderNetworks
Revised: FaderNetworks

Original: (Lample et al., 2017) learns latent variables from which label information is eliminated by using adversarial learning and assigns attributes to images by providing labels to the decoder.
Revised: (Lample et al., 2017) learns latent variables from which label information is eliminated by using adversarial learning and assigns attributes to images by providing labels to the decoder.

Original: Furthermore, StarGAN (Choi et al., 2017), a method of domain transfer, had succeeded in outputting a beautiful image corresponding to an attribute by conditioning with a domain (attribute).
Revised: Furthermore, StarGAN (Choi et al., 2017), a method of domain transfer, had succeeded in outputting a beautiful image corresponding to an attribute by conditioning with a domain (attribute).

Original: However, all these methods are models conditioned with labels; therefore, as the dimension of the labels becomes larger, the number of training samples becomes insufﬁcient.
Revised: However, all these methods are models conditioned with labels; therefore, as the dimension of the labels becomes larger, the number of training samples becomes insufﬁcient.

Original: Connection to the Information Bottleneck As with DualVAE, there are several papers related to ﬁnding a latent variable z that predicts label y.
Revised: Connection to the Information Bottleneck As with DualVAE, there are several papers related to ﬁnding a latent variable z that predicts label y.

Original: For example, Information Bottleneck (IB) (Tishby et al., 2000) is a method for obtaining a latent expression z that solves task y. IB is a method which leaves the latent information z for solving the task y by maximizing the mutual information amount
Revised: For example, Information Bottleneck (IB) (Tishby et al., 2000) is a method for obtaining a latent expression z that solves task y. IB is a method which leaves the latent information z for solving the task y by maximizing the mutual information amount

Original: I(Z; Y).
Revised: I(Z; Y).

Original: At the same time, extra information about input x is discarded by minimizing I(Z; X).
Revised: At the same time, extra information about input x is discarded by minimizing I(Z; X).

Original: Variational Information Bottleneck (VIB) (Alemi et al., 2016) succeeded in parameterizing the IB with a neural network, by performing a variational approximation.
Revised: Variational Information Bottleneck (VIB) (Alemi et al., 2016) succeeded in parameterizing the IB with a neural network, by performing a variational approximation.

Original: VIB can also be considered as a kind of extension of VAE.
Revised: VIB can also be considered as a kind of extension of VAE.

Original: VAE minimizes the mutual information I(Z; i) between individual data i and latent variable z while maximizing I(Z; X).
Revised: VAE minimizes the mutual information I(Z; i) between individual data i and latent variable z while maximizing I(Z; X).

Original: DualVAE can be regarded as a framework of VIB as well, and it minimizes I(Z; i) while maximizing I(Z; Y) and I(Z; X).
Revised: DualVAE can be regarded as a framework of VIB as well, and it minimizes I(Z; i) while maximizing I(Z; Y) and I(Z; X).

Original: Connection to the Probabilistic Matrix Factorization We can also regard DualVAE as a probabilistic matrix factorization (PMF) (Mnih & Salakhutdinov, 2008) extended to a generative model.
Revised: Connection to the Probabilistic Matrix Factorization We can also regard DualVAE as a probabilistic matrix factorization (PMF) (Mnih & Salakhutdinov, 2008) extended to a generative model.

Original: A PMF is used in several application areas, primarily in collaborative ﬁltering, which is a typical recommendation algorithm.
Revised: A PMF is used in several application areas, primarily in collaborative ﬁltering, which is a typical recommendation algorithm.

Original: It can predict missing ratings of users by assuming that the user’s ratings are modeled by a linear combination of the item and user latent factors.
Revised: It can predict missing ratings of users by assuming that the user’s ratings are modeled by a linear combination of the item and user latent factors.

Original: Similarly, we experimentally show that DualVAE is also robust to missing labels.
Revised: Similarly, we experimentally show that DualVAE is also robust to missing labels.

Original:  D UAL VAE
Revised:  D UAL VAE

Original: We devised DualVAE by adding a decoder p w ( y | z )
Revised: We devised DualVAE by adding a decoder p w ( y | z )

Original: = p ( y | z , u ) to the VAE to learn the decision boundaries between classes.
Revised: = p ( y | z , u ) to the VAE to learn the decision boundaries between classes.

Original: Here, z is a vector of the latent space Z , u is a vector of the dual latent space Z ∗ and y is a label.
Revised: Here, z is a vector of the latent space Z , u is a vector of the dual latent space Z ∗ and y is a label.

Original: Unlike the CVAE, this model does not require label y at the time of inference of z corresponding to x , and the difference is shown in Figure 2.
Revised: Unlike the CVAE, this model does not require label y at the time of inference of z corresponding to x , and the difference is shown in Figure 2.

Original: The objective function of the VAE is as follows:
Revised: The objective function of the VAE is as follows:

Original: [Equation]
Revised: [Equation]

Original: where φ , θ are parameters of the encoder and decoder of the VAE, respectively.
Revised: where φ , θ are parameters of the encoder and decoder of the VAE, respectively.

Original: The lower bound of
Revised: The lower bound of

Original: DualVAE is as follows:
Revised: DualVAE is as follows:

Original: [Equation]
Revised: [Equation]

Original: where p w ( y | z )
Revised: where p w ( y | z )

Original: =
Revised: =

Original: Bern ( y | σ ( Uz )) .
Revised: Bern ( y | σ ( Uz )) .

Original: Here, U is a domain feature matrix whose row vector is a dual vector u and Bern is a Bernoulli distribution.
Revised: Here, U is a domain feature matrix whose row vector is a dual vector u and Bern is a Bernoulli distribution.

Original: As you can see from Equation 2, the objective function of DualVAE is the objective function of the VAE plus the expectation of log-likelihood of p w ( y | z )
Revised: As you can see from Equation 2, the objective function of DualVAE is the objective function of the VAE plus the expectation of log-likelihood of p w ( y | z )

Original: Speciﬁcally, training is performed such that the inner product of z j ∈ Z and u
Revised: Speciﬁcally, training is performed such that the inner product of z j ∈ Z and u

Original: i ∈
Revised: i ∈

Original: Z ∗ predicts the label y ij where j is the index of a sample and i is the index of a domain.
Revised: Z ∗ predicts the label y ij where j is the index of a sample and i is the index of a domain.

Original: At the same time, we ﬁnd the values of θ and φ that maximize the lower bound in Equation 1.
Revised: At the same time, we ﬁnd the values of θ and φ that maximize the lower bound in Equation 1.

Original: We transfer the images on domain i by performing the following operation.
Revised: We transfer the images on domain i by performing the following operation.

Original: We calculated the following vector w i :
Revised: We calculated the following vector w i :

Original: [Equation]
Revised: [Equation]

Original: where λ ( ∈ R ) is a parameter.
Revised: where λ ( ∈ R ) is a parameter.

Original: Image transfer can be demonstrated by changing λ and decoding w i .
Revised: Image transfer can be demonstrated by changing λ and decoding w i .

Original: Equation 3 corresponds to moving a latent vector toward a decision boundary.
Revised: Equation 3 corresponds to moving a latent vector toward a decision boundary.

Original: Although IS (Salimans et al., 2016) is a score for measuring generated images, it can only measure the diversity of the images, and cannot be used for evaluating the domain transfer of the images. Therefore, we proposed using a CIS, a score for evaluating the transformation of images into multiclass target domains. The CIS is a scalar value calculated from the sum of two elements.
Revised: Algorithm

Original: The ﬁrst is whether the domain transfer of the original image has been successful (transfer score), and the second is whether the

Original: Require: images ( x j ) mj
Revised: Require: images ( x j ) mj

Original: =1 , batch size M , indicator function I ij VAE/encoder optimizers: g , g e , hyper parameter α , and the label matrix Y =
Revised: =1 , batch size M , indicator function I ij VAE/encoder optimizers: g , g e , hyper parameter α , and the label matrix Y =

Original: ( y ij ) .
Revised: ( y ij ) .

Original: Initialize encoder parameter, decoder parameter and dual vector: θ , φ , U = ( u i ) ni
Revised: Initialize encoder parameter, decoder parameter and dual vector: θ , φ , U = ( u i ) ni

Original: = repeat
Revised: = repeat

Original: [Table] [Equation] [Table] [Equation]
Revised: Randomly select batch ( x j ) j ∈ B of size M Sample

Original: [Figure]
Revised: z

Original: [Equation]
Revised: [Equation]

Original: [Figure]
Revised: until convergence of parameters θ , φ , U  CIS Although IS (Salimans et al., 2016) is a score for measuring generated images, it can only measure the diversity of the images, and cannot be used for evaluating the domain transfer of the images. Therefore, we proposed using a CIS, a score for evaluating the transformation of images into multiclass target domains.

Original: [Equation]
Revised: The CIS is a scalar value calculated from the sum of two elements.

Original: [Figure] until convergence of parameters θ , φ , U features other than the transferred domain are retained (reconstruction score).
Revised: The ﬁrst is whether the domain transfer of the original image has been successful (transfer score), and the second is whether the features other than the transferred domain are retained (reconstruction score).

Original: The computation ﬂow of these scores can be found in Figure 3.
Revised: The computation ﬂow of these scores can be found in Figure 3.

Original: We calculated the CIS using Algorithm 2.
Revised: We calculated the CIS using Algorithm 2.

Original: First, we assumed that the number of domains is n and the domain that each image belonged to was known.
Revised: First, we assumed that the number of domains is n and the domain that each image belonged to was known.

Original: We ﬁnetuned Inception-v3 (Szegedy et al., 2016) using train images as inputs and domains as outputs.
Revised: We ﬁnetuned Inception-v3 (Szegedy et al., 2016) using train images as inputs and domains as outputs.

Original: To enable the model to classify the images with the domains, we replaced the last layer of the model with a new layer that had n outputs.
Revised: To enable the model to classify the images with the domains, we replaced the last layer of the model with a new layer that had n outputs.

Original: Next, we transferred test images into n domain images and loaded the transferred images into the pretrained Inception-v3.
Revised: Next, we transferred test images into n domain images and loaded the transferred images into the pretrained Inception-v3.

Original: Through this process, we obtained an n × n matrix for every original image because one image was transferred into n domain images and each domain image was mapped to an n-dimension vector.
Revised: Through this process, we obtained an n × n matrix for every original image because one image was transferred into n domain images and each domain image was mapped to an n-dimension vector.

Original: We subsequently mapped the original image into an n-dimension vector using
Revised: We subsequently mapped the original image into an n-dimension vector using

Original: Inception-v3 and subtracted this vector from each row of the n × n matrix.
Revised: Inception-v3 and subtracted this vector from each row of the n × n matrix.

Original: We named this matrix
Revised: We named this matrix

Original: M. The key points are the following: (1) the diagonal elements of M should be large because the speciﬁed domain should be changed signiﬁcantly, and (2) the off-diagonal elements of M should be small because the transferred images should preserve the original features.
Revised: M. The key points are the following: (1) the diagonal elements of M should be large because the speciﬁed domain should be changed signiﬁcantly, and (2) the off-diagonal elements of M should be small because the transferred images should preserve the original features.

Original: Require: observation x ∈ X , Inception-v3 f : X → R n , domain transfer model m .
Revised: Require: observation x ∈ X , Inception-v3 f : X → R n , domain transfer model m .

Original: for i = 1 . . .
Revised: for i = 1 . . .

Original: n do x ( i )
Revised: n do x ( i )

Original: ← m i ( x )
Revised: ← m i ( x )

Original: M i ← f
Revised: M i ← f

Original: ( x ( i ) )
Revised: ( x ( i ) )

Original: − f ( x ) end for ts ← avg(diag( M ))
Revised: − f ( x ) end for ts ← avg(diag( M ))

Original: rs ← − avg(abs(notdiag( M )))
Revised: rs ← [Table] avg(abs [Table]

Original: CIS ← ts + rs
Revised: CIS ← ts + rs

Original: In the algorithm, abs denotes taking the absolute value, diag denotes taking the diagonal elements of the matrix, notdiag denotes taking the nondiagonal elements, avg denotes taking the mean of the multiclass values.
Revised: In the algorithm, abs denotes taking the absolute value, diag denotes taking the diagonal elements of the matrix, notdiag denotes taking the nondiagonal elements, avg denotes taking the mean of the multiclass values.

Original: x is an original image, and x ( i ) denotes a transferred image on domain i.
Revised: x is an original image, and x ( i ) denotes a transferred image on domain i.

Original:  E XPERIMENT
Revised:  E XPERIMENT

Original: We performed a standard image transfer task with the 40 attributes in the CelebA (Liu et al., 2015) dataset, which comprises approximately 200,000 images of faces of celebrities.
Revised: We performed a standard image transfer task with the 40 attributes in the CelebA (Liu et al., 2015) dataset, which comprises approximately 200,000 images of faces of celebrities.

Original: Comparison of DualVAE and several models DualVAE was compared with several models capable of performing image-to-image translations for multiclass labels using a single model.
Revised: Comparison of DualVAE and several models DualVAE was compared with several models capable of performing image-to-image translations for multiclass labels using a single model.

Original: In each model, we calculated the CIS several times when applying Algorithm 2 on 160 CelebA test images; subsequently, the average and standard deviation were obtained.
Revised: In each model, we calculated the CIS several times when applying Algorithm 2 on 160 CelebA test images; subsequently, the average and standard deviation were obtained.

Original: DualVAE obtained a higher CIS than the other models and the results are shown in Table 1 and Figure 4.
Revised: DualVAE obtained a higher CIS than the other models and the results are shown in Table 1 and Figure 4.

Original: [Table]
Revised: [Table]

Original: Visualization of transferred images We visualized transferred images of the 40 attributes by the proposed method and other models in Figure 5.
Revised: Visualization of transferred images We visualized transferred images of the 40 attributes by the proposed method and other models in Figure 5.

Original: Although StarGAN and FaderNetworks retained the characteristics of the original image considerably, it was qualitatively understood that domain transfer was poor.
Revised: Although StarGAN and FaderNetworks retained the characteristics of the original image considerably, it was qualitatively understood that domain transfer was poor.

Original: [Figure]
Revised: [Figure]

Original: Robustness to sparsity To demonstrate experimentally that DualVAE is robust to the missing values of multiclass labels, the following steps were performed.
Revised: Robustness to sparsity To demonstrate experimentally that DualVAE is robust to the missing values of multiclass labels, the following steps were performed.

Original: We calculated the rs and ts values when applying Algorithm 2 on 160 CelebA test images and plotted the ﬁgure below when we changed the missing ratio of CelebA’s domain labels and the λ in Equation 3.
Revised: We calculated the rs and ts values when applying Algorithm 2 on 160 CelebA test images and plotted the ﬁgure below when we changed the missing ratio of CelebA’s domain labels and the λ in Equation 3.

Original: DualVAE
Revised: DualVAE

Original: (a) s = 0 .
Revised: (a) s = 0 .

Original: Image transfer was still well-conducted.
Revised: Image transfer was still well-conducted.

Original: StarGAN
Revised: StarGAN

Original: (b) s = 0 .
Revised: (b) s = 0 .

Original: All identical images were generated, and image transfer was not conducted properly.
Revised: All identical images were generated, and image transfer was not conducted properly.

Original: As shown in Figure 6, DualVAE is robust in terms of the sparseness of domain labels, and the CIS does not decrease even when 90% of the labels are missing.
Revised: As shown in Figure 6, DualVAE is robust in terms of the sparseness of domain labels, and the CIS does not decrease even when 90% of the labels are missing.

Original: Meanwhile, we found that StarGAN is not as robust as DualVAE with respect to sparseness.
Revised: Meanwhile, we found that StarGAN is not as robust as DualVAE with respect to sparseness.

Original: When 90% of the domain labels are missing,
Revised: When 90% of the domain labels are missing,

Original: StarGAN cannot learn at all and generates identical images.
Revised: StarGAN cannot learn at all and generates identical images.

Original:  C ONCLUSION AND FUTURE WORK
Revised:  C ONCLUSION AND FUTURE WORK

Original: We proposed DualVAE, a simple framework for generating and transferring images corresponding to multiclass labels.
Revised: We proposed DualVAE, a simple framework for generating and transferring images corresponding to multiclass labels.

Original: Further, we introduced the CIS, a new metric for measuring how much of an image corresponding to the change of labels could be generated.
Revised: Further, we introduced the CIS, a new metric for measuring how much of an image corresponding to the change of labels could be generated.

Original: The decoder of DualVAE was a simple linear model in this study; however, we would like to test more complex models in the future.
Revised: The decoder of DualVAE was a simple linear model in this study; however, we would like to test more complex models in the future.