Original: M ULTI -V IEW
Revised: M ULTI -V IEW

Original: D ATA
Revised: D ATA

Original: G ENERATION
Revised: G ENERATION

Original: W ITHOUT
Revised: W ITHOUT

Original: V IEW
Revised: V IEW

Original: S UPERVISION
Revised: S UPERVISION

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: The development of high-dimensional generative models has recently gained a great surge of interest with the introduction of variational auto-encoders and generative adversarial neural networks.
Revised: The development of high-dimensional generative models has recently gained a great surge of interest with the introduction of variational auto-encoders and generative adversarial neural networks.

Original: Different variants have been proposed where the underlying latent space is structured, for example, based on attributes describing the data to generate.
Revised: Different variants have been proposed where the underlying latent space is structured, for example, based on attributes describing the data to generate.

Original: We focus on a particular problem where one aims at generating samples corresponding to a number of objects under various views.
Revised: We focus on a particular problem where one aims at generating samples corresponding to a number of objects under various views.

Original: We assume that the distribution of the data is driven by two independent latent factors: the content, which represents the intrinsic features of an object, and the view, which stands for the settings of a particular observation of that object.
Revised: We assume that the distribution of the data is driven by two independent latent factors: the content, which represents the intrinsic features of an object, and the view, which stands for the settings of a particular observation of that object.

Original: Therefore, we propose a generative model and a conditional variant built on such a disentangled latent space.
Revised: Therefore, we propose a generative model and a conditional variant built on such a disentangled latent space.

Original: This approach allows us to generate realistic samples corresponding to various objects in a high variety of views.
Revised: This approach allows us to generate realistic samples corresponding to various objects in a high variety of views.

Original: Unlike many multiview approaches, our model doesn’t need any supervision on the views but only on the content.
Revised: Unlike many multiview approaches, our model doesn’t need any supervision on the views but only on the content.

Original: Compared to other conditional generation approaches that are mostly based on binary or categorical attributes, we make no such assumption about the factors of variations.
Revised: Compared to other conditional generation approaches that are mostly based on binary or categorical attributes, we make no such assumption about the factors of variations.

Original: Our model can be used on problems with a huge, potentially inﬁnite, number of categories.
Revised: Our model can be used on problems with a huge, potentially inﬁnite, number of categories.

Original: We experiment it on four image datasets on which we demonstrate the effectiveness of the model and its ability to generalize.
Revised: We experiment it on four image datasets on which we demonstrate the effectiveness of the model and its ability to generalize.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Multi-view learning aims at developing models that are trained over datasets composed of multiple views over different objects.
Revised: Multi-view learning aims at developing models that are trained over datasets composed of multiple views over different objects.

Original: The problem of handling multi-view inputs has mainly been studied from the predictive point of view where one wants, for example, to learn a model able to predict/classify over multiple views of the same object (Su et al.
Revised: The problem of handling multi-view inputs has mainly been studied from the predictive point of view where one wants, for example, to learn a model able to predict/classify over multiple views of the same object (Su et al.

Original: ; Qi et al.
Revised: ; Qi et al.

Original: For example, using deep learning approaches, different strategies have been explored to aggregate multiple views but a general common idea is based on the (early or late) fusion of the different views at a particular level of a deep architecture.
Revised: For example, using deep learning approaches, different strategies have been explored to aggregate multiple views but a general common idea is based on the (early or late) fusion of the different views at a particular level of a deep architecture.

Original: Few other studies have proposed to predict missing views from one or multiple remaining views as in Arsalan Soltani et al.
Revised: Few other studies have proposed to predict missing views from one or multiple remaining views as in Arsalan Soltani et al.

Original: Recent research has focused on identifying factors of variations from multiview datasets.
Revised: Recent research has focused on identifying factors of variations from multiview datasets.

Original: The underlying idea is to consider that a particular data sample may be thought as the mix of a content information (e.g. related to its class label like a given person in a face dataset) and of a side information, the view, which accounts for factors of variability (e.g. exposure, viewpoint, with/wo glasses...).
Revised: The underlying idea is to consider that a particular data sample may be thought as the mix of a content information (e.g. related to its class label like a given person in a face dataset) and of a side information, the view, which accounts for factors of variability (e.g. exposure, viewpoint, with/wo glasses...).

Original: All samples of a given class share the same content information while they differ on the view information.
Revised: All samples of a given class share the same content information while they differ on the view information.

Original: A number of approaches have been proposed to disentangle the content from the view, also referred as the style in some papers (Mathieu et al.
Revised: A number of approaches have been proposed to disentangle the content from the view, also referred as the style in some papers (Mathieu et al.

Original: ; Denton & Birodkar (2017)).
Revised: ; Denton & Birodkar (2017)).

Original: For instance, different models have been built to extract from a single photo of any object both the characteristics of the object but also the camera position.
Revised: For instance, different models have been built to extract from a single photo of any object both the characteristics of the object but also the camera position.

Original: Once such a disentanglement is learned, one may build various applications like predicting how an object looks like under different viewpoints (Mathieu et al.
Revised: Once such a disentanglement is learned, one may build various applications like predicting how an object looks like under different viewpoints (Mathieu et al.

Original: ; Zhao et al.
Revised: ; Zhao et al.

Original: In the generative domain, models with a disentangled latent space (Louizos et al.
Revised: In the generative domain, models with a disentangled latent space (Louizos et al.

Original: ; Edwards & Storkey (2015)) have been recently proposed with applications to image editing, where one wants to modify an input sample by preserving its content while changing its view (Lample et al. (2017);
Revised: ; Edwards & Storkey (2015)) have been recently proposed with applications to image editing, where one wants to modify an input sample by preserving its content while changing its view (Lample et al. (2017);

Original: Kim et al.
Revised: Kim et al.

Original: (2017b))
Revised: (2017b))

Original: (see Section 6).
Revised: (see Section 6).

Original: Yet most existing controlled generative approaches have two strong limitations: (i) they usually consider discrete views that are characterized by a domain or a set of discrete (binary/categorical) attributes (e.g. face with/wo glasses, the color of the hairs, etc.) and could not easily scale to a large number of attributes or to continuous views.
Revised: Yet most existing controlled generative approaches have two strong limitations: (i) they usually consider discrete views that are characterized by a domain or a set of discrete (binary/categorical) attributes (e.g. face with/wo glasses, the color of the hairs, etc.) and could not easily scale to a large number of attributes or to continuous views.

Original: (ii) most models are trained using view supervision (e.g. the view attributes), which of course greatly helps learning such model, yet prevents their use on many datasets where this information is not available.
Revised: (ii) most models are trained using view supervision (e.g. the view attributes), which of course greatly helps learning such model, yet prevents their use on many datasets where this information is not available.

Original: Recently, some attempts have been made to learn such models without any supervision (Chen et al.
Revised: Recently, some attempts have been made to learn such models without any supervision (Chen et al.

Original: ; Higgins et al.
Revised: ; Higgins et al.

Original: , but they cannot disentangle high level concept as only simple features can be reliably captured without any guidance.
Revised: , but they cannot disentangle high level concept as only simple features can be reliably captured without any guidance.

Original: In this paper, we are interested in learning generative models that build and make use of a disentangled latent space where the content and the view are encoded separately.
Revised: In this paper, we are interested in learning generative models that build and make use of a disentangled latent space where the content and the view are encoded separately.

Original: We propose to take an original approach by learning such models from multi-view datasets, where (i) samples are labeled based on their content, and without any view information, and (ii) where the generated views are not restricted to be one view in a subset of possible views.
Revised: We propose to take an original approach by learning such models from multi-view datasets, where (i) samples are labeled based on their content, and without any view information, and (ii) where the generated views are not restricted to be one view in a subset of possible views.

Original: Following with our same example above, it means ﬁrst, learning from a face dataset including multiple photos of multiple persons taken in various conditions related to exposure, viewpoint etc.
Revised: Following with our same example above, it means ﬁrst, learning from a face dataset including multiple photos of multiple persons taken in various conditions related to exposure, viewpoint etc.

Original: and second, being able to generate an inﬁnity of views of an imaginary person (or the same views of an inﬁnity of imaginary persons) – see Figure 1.
Revised: and second, being able to generate an inﬁnity of views of an imaginary person (or the same views of an inﬁnity of imaginary persons) – see Figure 1.

Original: This contrast with most current approaches that use information about the style, and cannot generate multiple possible outputs.
Revised: This contrast with most current approaches that use information about the style, and cannot generate multiple possible outputs.

Original: More precisely, we propose two models to tackle this particularly difﬁcult setting: a generative model (GMV - Generative Multi-view Model) that generates objects under various views ( multiview generation ), and a conditional extension (C-GMV) of this model that generates a large number of views of any input object ( conditional multi-view generation ).
Revised: More precisely, we propose two models to tackle this particularly difﬁcult setting: a generative model (GMV - Generative Multi-view Model) that generates objects under various views ( multiview generation ), and a conditional extension (C-GMV) of this model that generates a large number of views of any input object ( conditional multi-view generation ).

Original: These two models are based on the adversarial training schema of Generative Adversarial Networks (GAN) proposed in Goodfellow et al.
Revised: These two models are based on the adversarial training schema of Generative Adversarial Networks (GAN) proposed in Goodfellow et al.

Original: The simple but strong idea is to focus on distributions over pairs of examples (e.g. images representing a same object in different views ) rather than distribution on single examples as we will explain later.
Revised: The simple but strong idea is to focus on distributions over pairs of examples (e.g. images representing a same object in different views ) rather than distribution on single examples as we will explain later.

Original: Our contributions are the following: (i)
Revised: Our contributions are the following: (i)

Original: We propose a new generative model able to generate data with various content and high view diversity using a supervision on the content information only.
Revised: We propose a new generative model able to generate data with various content and high view diversity using a supervision on the content information only.

Original: (ii)
Revised: (ii)

Original: We extend the model to a conditional model that allows generating new views over any input sample.
Revised: We extend the model to a conditional model that allows generating new views over any input sample.

Original: (iii)
Revised: (iii)

Original: We report experimental results on four different images datasets that show the ability of our models to generate realistic samples and to capture (and generate with) the diversity of views.
Revised: We report experimental results on four different images datasets that show the ability of our models to generate realistic samples and to capture (and generate with) the diversity of views.

Original: The paper is organized as follows.
Revised: The paper is organized as follows.

Original: We ﬁrst recall useful background on GANs and their conditional variant (Section 2).
Revised: We ﬁrst recall useful background on GANs and their conditional variant (Section 2).

Original: Then we successively detail our proposal for a generative multiview model (Section 3.2) and for its conditional extension (Section 4).
Revised: Then we successively detail our proposal for a generative multiview model (Section 3.2) and for its conditional extension (Section 4).

Original: Finally, we report experimental results on the various generative tasks allowed by our models in Section 5.2.
Revised: Finally, we report experimental results on the various generative tasks allowed by our models in Section 5.2.

Original:  B ACKGROUND
Revised:  B ACKGROUND

Original: Our work is inspired by the Generative Adversarial Network (GAN) model proposed in Goodfellow et al.
Revised: Our work is inspired by the Generative Adversarial Network (GAN) model proposed in Goodfellow et al.

Original: We brieﬂy review the principle of GAN and of one of its conditional version called
Revised: We brieﬂy review the principle of GAN and of one of its conditional version called

Original: Conditional GAN (CGAN) (Mirza & Osindero (2014)) that are the fundations of this work.
Revised: Conditional GAN (CGAN) (Mirza & Osindero (2014)) that are the fundations of this work.

Original: 2. G ENERATIVE A DVERSARIAL N ETWORK
Revised: 2. G ENERATIVE A DVERSARIAL N ETWORK

Original: Let us denote X an input space composed of multi-dimensional samples x e.g vector, matrix or tensor.
Revised: Let us denote X an input space composed of multi-dimensional samples x e.g vector, matrix or tensor.

Original: Given a latent space R n and a prior distribution p z ( z ) over this latent space, any generator function G : R n → X deﬁnes a distribution p G on X which is the distribution of samples G( z ) where z ∼ p z .
Revised: Given a latent space R n and a prior distribution p z ( z ) over this latent space, any generator function G : R n → X deﬁnes a distribution p G on X which is the distribution of samples G( z ) where z ∼ p z .

Original: A GAN deﬁnes, in addition to G , a discriminator function D : X → [0; 1] which aims at differentiating between real inputs sampled from the training set and fake inputs sampled following p G , while the generator is learned to fool the discriminator D .
Revised: A GAN deﬁnes, in addition to G , a discriminator function D : X → [0; 1] which aims at differentiating between real inputs sampled from the training set and fake inputs sampled following p G , while the generator is learned to fool the discriminator D .

Original: Usually both G and D are implemented with neural networks.
Revised: Usually both G and D are implemented with neural networks.

Original: The objective function is based on the following adversarial criterion:
Revised: The objective function is based on the following adversarial criterion:

Original: [Equation]
Revised: [Equation]

Original: where p x is the empirical data distribution on X .
Revised: where p x is the empirical data distribution on X .

Original: It has been shown in Goodfellow et al.
Revised: It has been shown in Goodfellow et al.

Original: (2014) that if G ∗ and D ∗ are optimal for the above criterion, the Jensen-Shannon divergence between p G ∗ and the empirical distribution of the data p x in the dataset is minimized, making GAN able to estimate complex continuous data distributions.
Revised: (2014) that if G ∗ and D ∗ are optimal for the above criterion, the Jensen-Shannon divergence between p G ∗ and the empirical distribution of the data p x in the dataset is minimized, making GAN able to estimate complex continuous data distributions.

Original: 2. C ONDITIONAL G ENERATIVE A DVERSARIAL N ETWORK
Revised: 2. C ONDITIONAL G ENERATIVE A DVERSARIAL N ETWORK

Original: A conditional version of GAN (CGAN) has been proposed in Mirza & Osindero (2014).
Revised: A conditional version of GAN (CGAN) has been proposed in Mirza & Osindero (2014).

Original: Instead of learning from an unsupervised dataset composed of datapoints x , a CGAN is learned to implement a conditional distribution p( x | y ) using a training set that consists of pairs of inputs/conditions ( x , y ) where x is a target and y is the condition.
Revised: Instead of learning from an unsupervised dataset composed of datapoints x , a CGAN is learned to implement a conditional distribution p( x | y ) using a training set that consists of pairs of inputs/conditions ( x , y ) where x is a target and y is the condition.

Original: The conditionality of a CGAN is obtained by deﬁning a generator function G that takes as inputs both a noise vector z and a condition y .
Revised: The conditionality of a CGAN is obtained by deﬁning a generator function G that takes as inputs both a noise vector z and a condition y .

Original: A target x from a given input y may be obtained by ﬁrst sampling the latent vector z ∼ p
Revised: A target x from a given input y may be obtained by ﬁrst sampling the latent vector z ∼ p

Original: z , then by computing
Revised: z , then by computing

Original: G( y , z ) .
Revised: G( y , z ) .

Original: The discriminator in a CGAN takes as inputs both the condition y and the (generated or real) datapoint x .
Revised: The discriminator in a CGAN takes as inputs both the condition y and the (generated or real) datapoint x .

Original: It is learned to discriminate between fake input/target pairs (where the target is sampled using the generator) from real pairs drawn from the empirical distribution.
Revised: It is learned to discriminate between fake input/target pairs (where the target is sampled using the generator) from real pairs drawn from the empirical distribution.

Original: The objective function can be written as:
Revised: The objective function can be written as:

Original: [Equation]
Revised: [Equation]

Original: where p x , y stands for the empirical distribution on ( x , y ) .
Revised: where p x , y stands for the empirical distribution on ( x , y ) .

Original: As with regular GAN, with optimal
Revised: As with regular GAN, with optimal

Original: G and D denoted respectively G ∗ and D ∗ ), the distribution p ∗ G ﬁts the true empirical conditional distribution of the input/target pairs.
Revised: G and D denoted respectively G ∗ and D ∗ ), the distribution p ∗ G ﬁts the true empirical conditional distribution of the input/target pairs.

Original: Unfortunately, many studies have reported that on when dealing with high-dimensional input spaces, CGAN tends to collapse the modes of the data distribution, mostly ignoring the latent factor z and generating x only based on the condition y , exhibiting an almost deterministic behavior.
Revised: Unfortunately, many studies have reported that on when dealing with high-dimensional input spaces, CGAN tends to collapse the modes of the data distribution, mostly ignoring the latent factor z and generating x only based on the condition y , exhibiting an almost deterministic behavior.

Original: In most cases, CGAN is unable to produce targets with a satisfying diversity (see Section 5.2).
Revised: In most cases, CGAN is unable to produce targets with a satisfying diversity (see Section 5.2).

Original:  G ENERATIVE M ULTI - VIEW M ODEL 3. O BJECTIVE AND N OTATIONS
Revised:  G ENERATIVE M ULTI - VIEW M ODEL 3. O BJECTIVE AND N OTATIONS

Original: We consider the multi-view setting where data samples represent a number of objects that have been observed in various views.
Revised: We consider the multi-view setting where data samples represent a number of objects that have been observed in various views.

Original: The distribution of the data x ∈ X is assumed to be driven by two latent factors: a content factor denoted c which corresponds to the invariant proprieties of the object, and a view factor denoted v which corresponds to the factor of variations.
Revised: The distribution of the data x ∈ X is assumed to be driven by two latent factors: a content factor denoted c which corresponds to the invariant proprieties of the object, and a view factor denoted v which corresponds to the factor of variations.

Original: Typically, if X is the space of person’s faces, c stands for the intrinsic features of a person’s face while v stands for the transient features and the viewpoint of a particular photo of the face, including the photo exposure and additional elements like hat, glasses, etc....
Revised: Typically, if X is the space of person’s faces, c stands for the intrinsic features of a person’s face while v stands for the transient features and the viewpoint of a particular photo of the face, including the photo exposure and additional elements like hat, glasses, etc....

Original: We assume that these two factors c and v are independent.
Revised: We assume that these two factors c and v are independent.

Original: This a key property of the factors we want to learn.
Revised: This a key property of the factors we want to learn.

Original: We focus on two different tasks: (i) Multi View Generation : we want to be able to sample over
Revised: We focus on two different tasks: (i) Multi View Generation : we want to be able to sample over

Original: X by controlling the two factors c and v .
Revised: X by controlling the two factors c and v .

Original: Said otherwise, we want to be able to generate different views of the same object, or the same view of different objects.
Revised: Said otherwise, we want to be able to generate different views of the same object, or the same view of different objects.

Original: Given two priors, p( c ) and p( v ) , this sampling will be possible if we are able to estimate p( x | c , v ) from a training set.
Revised: Given two priors, p( c ) and p( v ) , this sampling will be possible if we are able to estimate p( x | c , v ) from a training set.

Original: (ii) Conditionnal
Revised: (ii) Conditionnal

Original: Multi-View Generation : the second objective, is to be able to sample different views of a given object.
Revised: Multi-View Generation : the second objective, is to be able to sample different views of a given object.

Original: For instance, it can be different views of a particular person based on one of his photos.
Revised: For instance, it can be different views of a particular person based on one of his photos.

Original: Given a prior p( v ) , this sampling will be achieved by learning the probability p( c | x ) , in addition to p( x | c , v )
Revised: Given a prior p( v ) , this sampling will be achieved by learning the probability p( c | x ) , in addition to p( x | c , v )

Original: The key issue for tackling these two tasks lies in the ability to accurately learn generative models able to generate from a disentangled latent space where the content factor and the view factor are encoded (and thus sampled) separately.
Revised: The key issue for tackling these two tasks lies in the ability to accurately learn generative models able to generate from a disentangled latent space where the content factor and the view factor are encoded (and thus sampled) separately.

Original: This would allow controlling the sampling on the two different axis, the content and the view.
Revised: This would allow controlling the sampling on the two different axis, the content and the view.

Original: The originality of our work is to learn such generative models without using any view labeling information.
Revised: The originality of our work is to learn such generative models without using any view labeling information.

Original: Let us denote by N the set of different objects and n i the number of views available for object number i (not necessarily the same sets of views nor number of views for every object) such that { x i 1 , x i 2 , ..., x in i } is the set of views for object i ∈ [1; N ] .
Revised: Let us denote by N the set of different objects and n i the number of views available for object number i (not necessarily the same sets of views nor number of views for every object) such that { x i 1 , x i 2 , ..., x in i } is the set of views for object i ∈ [1; N ] .

Original: Moreover, we consider that x is a tensor (e.g an image) and c and v are (latent) vectors in R C and R V , C and V being the sizes of the content and view latent spaces.
Revised: Moreover, we consider that x is a tensor (e.g an image) and c and v are (latent) vectors in R C and R V , C and V being the sizes of the content and view latent spaces.

Original: Note that this setting is not restrictive and corresponds for instance to categorization datasets where multiple photos of objects are available.
Revised: Note that this setting is not restrictive and corresponds for instance to categorization datasets where multiple photos of objects are available.

Original: 3. G ENERATIVE M ULTIVIEW M ODEL
Revised: 3. G ENERATIVE M ULTIVIEW M ODEL

Original: Let us consider two prior distributions over the content and view factors denoted as p c and p v , these distributions typically being isotropic Gaussian distribution.
Revised: Let us consider two prior distributions over the content and view factors denoted as p c and p v , these distributions typically being isotropic Gaussian distribution.

Original: These two distributions correspond to the prior distribution over content and latent factors.
Revised: These two distributions correspond to the prior distribution over content and latent factors.

Original: Moreover, we consider a generator G that implements a distribution over samples x , denoted as p G .
Revised: Moreover, we consider a generator G that implements a distribution over samples x , denoted as p G .

Original: By computing G( c , v ) with c ∼ p c and v ∼ p v .
Revised: By computing G( c , v ) with c ∼ p c and v ∼ p v .

Original: Our objective is to learn this generator so that its ﬁrst input c corresponds to the content of the generated sample while its second input, v , captures the underlying view of the sample.
Revised: Our objective is to learn this generator so that its ﬁrst input c corresponds to the content of the generated sample while its second input, v , captures the underlying view of the sample.

Original: Doing so would allow one to control the output sample of the generator by tuning its content or its view (i.e. c and v ).
Revised: Doing so would allow one to control the output sample of the generator by tuning its content or its view (i.e. c and v ).

Original: Yet it is expected that learning G using a standard GAN approach would not allow to accurately disentangle the latent space.
Revised: Yet it is expected that learning G using a standard GAN approach would not allow to accurately disentangle the latent space.

Original: Indeed, without constraint, the content and view factors are going to be diluted in the input latent factor z of the GAN, without possibility to know which dimensions of z capture the content and which capture the view factor.
Revised: Indeed, without constraint, the content and view factors are going to be diluted in the input latent factor z of the GAN, without possibility to know which dimensions of z capture the content and which capture the view factor.

Original: We propose a novel way to achieve this desired feature.
Revised: We propose a novel way to achieve this desired feature.

Original: The key idea of our model is to focus on the distribution of pairs of inputs rather than on the distribution over individual samples.
Revised: The key idea of our model is to focus on the distribution of pairs of inputs rather than on the distribution over individual samples.

Original: We explain now which pairs we are talking about and why considering pairs might be useful.
Revised: We explain now which pairs we are talking about and why considering pairs might be useful.

Original: First, which pairs are we considering?
Revised: First, which pairs are we considering?

Original: When no view supervision is available the only valuable pairs of samples that one may build from the dataset consist of two samples of a given object under two different views.
Revised: When no view supervision is available the only valuable pairs of samples that one may build from the dataset consist of two samples of a given object under two different views.

Original: Indeed, choosing randomly two samples of a given object will most of the time correspond to different views of this object, especially when considering continuous views as we do.
Revised: Indeed, choosing randomly two samples of a given object will most of the time correspond to different views of this object, especially when considering continuous views as we do.

Original: Fortunately, considering a generator G as discussed above (operating on a couple of a content vector c and of a view vector v ), one can generate corresponding artiﬁcial (fake) pairs of samples by sampling a single content vector c ∼ p c to combine with two different view vectors v 1 ∼ p v and v 2 ∼ p v , i.e. constructing G( c , v 1 ) and G( c , v 2 ) – see Figure 2.
Revised: Fortunately, considering a generator G as discussed above (operating on a couple of a content vector c and of a view vector v ), one can generate corresponding artiﬁcial (fake) pairs of samples by sampling a single content vector c ∼ p c to combine with two different view vectors v 1 ∼ p v and v 2 ∼ p v , i.e. constructing G( c , v 1 ) and G( c , v 2 ) – see Figure 2.

Original: Second, why working on such pairs would be interesting?
Revised: Second, why working on such pairs would be interesting?

Original: Following the GAN idea, we propose to learn a discriminator to distinguish between such real and fake pairs.
Revised: Following the GAN idea, we propose to learn a discriminator to distinguish between such real and fake pairs.

Original: To be able to fool the discriminator, the generator then has to achieve three goals.
Revised: To be able to fool the discriminator, the generator then has to achieve three goals.

Original: (i)
Revised: (i)

Original: As in regular GAN, each sample generated by G needs to look realistic (ii) Moreover, because real pairs are composed of two views of the same object, the generator should generate pairs of the same object.
Revised: As in regular GAN, each sample generated by G needs to look realistic (ii) Moreover, because real pairs are composed of two views of the same object, the generator should generate pairs of the same object.

Original: Since the two sampled view factors v 1 and v 2 are different, the only way this can be achieved is by encoding the invariant features into the content vector c .
Revised: Since the two sampled view factors v 1 and v 2 are different, the only way this can be achieved is by encoding the invariant features into the content vector c .

Original: (iii)
Revised: (iii)

Original: Finally, it is expected that the discriminator should easily discriminates between pair of samples corresponding to the same object under different views from a pair of samples corresponding to a same object under the same view.
Revised: Finally, it is expected that the discriminator should easily discriminates between pair of samples corresponding to the same object under different views from a pair of samples corresponding to a same object under the same view.

Original: Because the pair shares the same content factor c , this should force the generator to use the view factors v 1 and v 2 to produce diversity in the generated pair.
Revised: Because the pair shares the same content factor c , this should force the generator to use the view factors v 1 and v 2 to produce diversity in the generated pair.

Original: The Generative Multiview Model’s (GMV) architecutre is detailed in Figure 2.
Revised: The Generative Multiview Model’s (GMV) architecutre is detailed in Figure 2.

Original: It is learned by optimizing the following adversarial loss function:
Revised: It is learned by optimizing the following adversarial loss function:

Original: [Equation]
Revised: [Equation]

Original: ) , G( c , v 2 )))] (3) where l ( x ) stands for the label of x (e.g. a particular person in a face dataset).
Revised: ) , G( c , v 2 )))] (3) where l ( x ) stands for the label of x (e.g. a particular person in a face dataset).

Original: Note that, since the proposed model can be seen as a particular GAN architecture over pairs of inputs, the global minimum of the learning criterion is obtained when the model is able to sample pairs of views over a similar object.
Revised: Note that, since the proposed model can be seen as a particular GAN architecture over pairs of inputs, the global minimum of the learning criterion is obtained when the model is able to sample pairs of views over a similar object.

Original: Using the Model at inference: As discussed above the training of the discriminator on pairs of samples introduce useful constraints on how the content and the view information are used to generate samples.
Revised: Using the Model at inference: As discussed above the training of the discriminator on pairs of samples introduce useful constraints on how the content and the view information are used to generate samples.

Original: Once the model is learned we are left with a generator G that generates single samples by ﬁrst sampling c and v following p c and p v , then by computing G( c , v ) .
Revised: Once the model is learned we are left with a generator G that generates single samples by ﬁrst sampling c and v following p c and p v , then by computing G( c , v ) .

Original: By freezing c or v , one may then generate samples corresponding to multiple views of any particular content, or corresponding to many contents under a particular view.
Revised: By freezing c or v , one may then generate samples corresponding to multiple views of any particular content, or corresponding to many contents under a particular view.

Original: One can also make interpolations between two given views over a particular content, or between two contents using a particular view (See examples in Figure 4).
Revised: One can also make interpolations between two given views over a particular content, or between two contents using a particular view (See examples in Figure 4).

Original:  C ONDITIONAL G ENERATIVE M ODEL (C-GMV)
Revised:  C ONDITIONAL G ENERATIVE M ODEL (C-GMV)

Original: The GMV model allows one to sample objects with different views, but it is not able to change the view of a given object that would be provided as an input to the model.
Revised: The GMV model allows one to sample objects with different views, but it is not able to change the view of a given object that would be provided as an input to the model.

Original: This however, might be of interest in particular applications like image editing.
Revised: This however, might be of interest in particular applications like image editing.

Original: This section aims at extending our generative model the ability to extract the content factor from any given input and to use this extracted content in order to generate new views of the corresponding object.
Revised: This section aims at extending our generative model the ability to extract the content factor from any given input and to use this extracted content in order to generate new views of the corresponding object.

Original: To achieve such a goal, we must add to our generative model an encoder function denoted E : X → R C that will map any input in X to the content space R C
Revised: To achieve such a goal, we must add to our generative model an encoder function denoted E : X → R C that will map any input in X to the content space R C

Original: (see Figure 3).
Revised: (see Figure 3).

Original: To do so we take inspiration from the CGAN model (Section 2.2).
Revised: To do so we take inspiration from the CGAN model (Section 2.2).

Original: We will encode an input sample x in the content space using an encoder function, noted E (implemented again as a neural network).
Revised: We will encode an input sample x in the content space using an encoder function, noted E (implemented again as a neural network).

Original: This encoder serves to generate a content vector c = E(x) that will be combined with a randomly sampled view v ∼ p v to generate an artiﬁcial example.
Revised: This encoder serves to generate a content vector c = E(x) that will be combined with a randomly sampled view v ∼ p v to generate an artiﬁcial example.

Original: The artiﬁcial sample is then combined with the original input x to form a negative pair.
Revised: The artiﬁcial sample is then combined with the original input x to form a negative pair.

Original: This is illustrated in the extreme right part of Figure 3 which exactly corresponds to a CGAN architecture.
Revised: This is illustrated in the extreme right part of Figure 3 which exactly corresponds to a CGAN architecture.

Original: Yet CGAN has severe weaknesses and are known to easily miss modes of the underlying distribution.
Revised: Yet CGAN has severe weaknesses and are known to easily miss modes of the underlying distribution.

Original: The generator enters in a state where it ignores the noisy component v (see results in Figure 7).
Revised: The generator enters in a state where it ignores the noisy component v (see results in Figure 7).

Original: To overcome this phenomenon, we use the same idea as in GMV.
Revised: To overcome this phenomenon, we use the same idea as in GMV.

Original: We build negative pairs (G( c , v 1 ) , G( c , v 2 )) by randomly sampling two views v 1 and v 2 that we combine to a unique content c .
Revised: We build negative pairs (G( c , v 1 ) , G( c , v 2 )) by randomly sampling two views v 1 and v 2 that we combine to a unique content c .

Original: This time however, c is not sampled according to a noise distribution but it is computed from a sample x using the encoder E , i.e. c =
Revised: This time however, c is not sampled according to a noise distribution but it is computed from a sample x using the encoder E , i.e. c =

Original: E( x ) .
Revised: E( x ) .

Original: By doing so, we preserve the ability of our approach to generating pairs with view diversity.
Revised: By doing so, we preserve the ability of our approach to generating pairs with view diversity.

Original: Since this diversity can only be captured by taking into account the two different view vectors provided to the model ( v 1 and v 2 ), this will encourage G( c , v ) to generate samples containing both the content information c , and the view v .
Revised: Since this diversity can only be captured by taking into account the two different view vectors provided to the model ( v 1 and v 2 ), this will encourage G( c , v ) to generate samples containing both the content information c , and the view v .

Original: As it was done for the GMV model, positive pairs are sampled from the training set and correspond to two views of a given object.
Revised: As it was done for the GMV model, positive pairs are sampled from the training set and correspond to two views of a given object.

Original: In this setting, the resulting adversarial loss function can be written as:
Revised: In this setting, the resulting adversarial loss function can be written as:

Original: [Equation]
Revised: [Equation]

Original: At inference time, as we did with the GMV model, we are interested in getting the encoder E and the generator G .
Revised: At inference time, as we did with the GMV model, we are interested in getting the encoder E and the generator G .

Original: These models may be used for generating new views of any object which is observed as an input sample x by computing its content vector E( x ) , then sampling v ∼ p v and ﬁnally by computing the output G(E( x ) , v ) .
Revised: These models may be used for generating new views of any object which is observed as an input sample x by computing its content vector E( x ) , then sampling v ∼ p v and ﬁnally by computing the output G(E( x ) , v ) .

Original:  E XPERIMENTAL RESULTS 5. E XPERIMENTAL P ROTOCOL
Revised:  E XPERIMENTAL RESULTS 5. E XPERIMENTAL P ROTOCOL

Original: Datasets:
Revised: Datasets:

Original: In order to evaluate the quality of our two models, we have performed experiments over four image datasets of various domains.
Revised: In order to evaluate the quality of our two models, we have performed experiments over four image datasets of various domains.

Original: The statistics of the dataset are given in Table 1.
Revised: The statistics of the dataset are given in Table 1.

Original: Note that when supervision is available on the views (like CelebA for example where images are labeled with attributes) we do not use it for learning our models.
Revised: Note that when supervision is available on the views (like CelebA for example where images are labeled with attributes) we do not use it for learning our models.

Original: The only supervision that we use is if two samples correspond or not to the same object.
Revised: The only supervision that we use is if two samples correspond or not to the same object.

Original: Model Architecture
Revised: Model Architecture

Original: We have used the same architectures for every dataset.
Revised: We have used the same architectures for every dataset.

Original: The images were rescaled to 3 × 64 × 64 tensors.
Revised: The images were rescaled to 3 × 64 × 64 tensors.

Original: The generator G and the discriminator D follow that of the DCGAN
Revised: The generator G and the discriminator D follow that of the DCGAN

Original: [Table]
Revised: [Table]

Original: implementation proposed in Radford et al.
Revised: implementation proposed in Radford et al.

Original: For the conditional model, the encoder E is similar to D except for the ﬁrst layer that includes a batch normalization and the last layer that doesn’t have a non-linearity.
Revised: For the conditional model, the encoder E is similar to D except for the ﬁrst layer that includes a batch normalization and the last layer that doesn’t have a non-linearity.

Original: Following the article, an implementation of our algorithms is freely available 1 .
Revised: Following the article, an implementation of our algorithms is freely available 1 .

Original: Learning has been made using classical GAN learning techniques: we used Adam optimizer Kingma & Ba (2014) with batches of size 128.
Revised: Learning has been made using classical GAN learning techniques: we used Adam optimizer (Kingma & Ba (2014)) with batches of size 128.

Original: Following standard practice, learning rate in the GMV experiments are set to 1 · 10 − 3 of G and 2 · 10 − 4 for D .
Revised: Following standard practice, learning rate in the GMV experiments are set to 1 · 10 − 3 of G and 2 · 10 − 4 for D .

Original: For the C-GMV experiments, learning rates are set to 5 · 10 − 5 .
Revised: For the C-GMV experiments, learning rates are set to 5 · 10 − 5 .

Original: The adversarial objectives are optimized by alternating gradient descent over the generator/encoder, and over the discriminator.
Revised: The adversarial objectives are optimized by alternating gradient descent over the generator/encoder, and over the discriminator.

Original: Baselines:
Revised: Baselines:

Original: We compareartiﬁcial samples generated by our models and by recent state-of-the-art techniques.
Revised: We compare our proposal with recent state-of-the-art techniques.

Original: However, most existing methods are learned on datasets with view labeling.
Revised: However, most existing methods are learned on datasets with view labeling.

Original: To fairly compare with alternative models we have built baselines working in the same conditions as our models. 
Revised: To fairly compare with alternative models we have built baselines working in the same conditions as our models. In addition we compare our models with the model from Mathieu et al.

Revised: We report results gained with two implementations, the ﬁrst one based on the implementation provided by the authors 2 (denoted Mathieu et al.

Revised: (2016)), and the second one (denoted Mathieu et al.

Revised: (2016) (DCGAN) ) that implements the same model using architectures inspired from DCGAN Radford et al.

Revised: (2015), which is more stable and that we have carefully tuned to allow a fair comparison with our approach.

Original: For pure multi-view generative setting, we compared our generative model (GMV) with standard
Revised: For pure multi-view generative setting, we compared our generative model (GMV) with standard

Original: GANs that are learned to approximate the joint generation of multiple samples: DCGANx2 is learned to output pairs of views over the same object, DCGANx4 is trained on quadruplet, and
Revised: GANs that are learned to approximate the joint generation of multiple samples: DCGANx2 is learned to output pairs of views over the same object, DCGANx4 is trained on quadruplet, and

Original: DCGANx8 on eight different views.
Revised: DCGANx8 on eight different views.

Original: To be more detailed, for instance the generator of a DCGANx model takes as input a sampled noise vector and outputs 2 images, while its discriminator is learned to distinguish these pairs of images as negative samples from positive samples which are built as for the GMV model, i.e. pairs of samples in the dataset that corresponds to the same object.
Revised: To be more detailed, for instance the generator of a DCGANx model takes as input a sampled noise vector and outputs 2 images, while its discriminator is learned to distinguish these pairs of images as negative samples from positive samples which are built as for the GMV model, i.e. pairs of samples in the dataset that corresponds to the same object.

Original: The main difference with GMV is that the above GAN-based methods do not explicitly distinguish content and view factors as it is done in GMV.
Revised: The main difference with GMV is that the above GAN-based methods do not explicitly distinguish content and view factors as it is done in GMV.

Original: Likewise, for conditional generation, we compared our approach C-GMV with CGAN models that we brieﬂy introduced in Section 2.2.
Revised: Likewise, for conditional generation, we compared our approach C-GMV with CGAN models that we brieﬂy introduced in Section 2.2 and with the two variants of the model from Mathieu et al. (2016) that we just mentioned.

Original: 5. E XPERIMENTAL R ESULTS G ENERATING MULTIPLE CONTENTS AND VIEWS
Revised: 5. E XPERIMENTAL R ESULTS G ENERATING MULTIPLE CONTENTS AND VIEWS

Original: We ﬁrst evaluate the ability of our model to generate a large variety of object and views.
Revised: We ﬁrst evaluate the ability of our model to generate a large variety of object and views.

Original: Figureshow examples of generated images by our model and Figure 4 show images sampled by DCGANbased models (DCGANx2, DCGANx4, and DCGANx8) on 3DChairs and CelebA datasets (more results are provided in the Appendix).
Revised: Figureshow examples of generated images by our model and Figure 4 show images sampled by DCGANbased models (DCGANx2, DCGANx4, and DCGANx8) on 3DChairs and CelebA datasets (more results are provided in the Appendix).

Original: For GMV generated images, a row shows a number of samples that have been generated with the same sampled content factor c ∼ p c but with various sampled view factors v ∼ p v , while the same view factor v is used for all samples in a column.
Revised: For GMV generated images, a row shows a number of samples that have been generated with the same sampled content factor c ∼ p c but with various sampled view factors v ∼ p v , while the same view factor v is used for all samples in a column.

Original: Figure 5 shows additional results, using the same presentation, for the GMV model only on two other datasets.
Revised: Figure 5 shows additional results, using the same presentation, for the GMV model only on two other datasets.

Original: One sees on these ﬁgures that our approach allows to accurately generate the same views of multiple objects, or alternatively the multiple views of a single object.
Revised: One sees on these ﬁgures that our approach allows to accurately generate the same views of multiple objects, or alternatively the multiple views of a single object.

Original: The generated images are of good quality, and the diversity of the generated views is high, showing the ability of our model to capture the diversity of the training dataset in term of possible views.
Revised: The generated images are of good quality, and the diversity of the generated views is high, showing the ability of our model to capture the diversity of the training dataset in term of possible views.

Original: Figure 4 shows similar results for GAN-based models.
Revised: Figure 4 shows similar results for GAN-based models.

Original: For images generated by these models a row corresponds to the multiple images produced by the model for a given sampled noise vector.
Revised: For images generated by these models a row corresponds to the multiple images produced by the model for a given sampled noise vector.

Original: When comparing GMV generated images to those generated by GAN-based models, one can see that the quality of images produced by DCGANx2 is comparable to the ones we obtain with GMV showing our approach has the same ability than GAN to generate good outputs.
Revised: When comparing GMV generated images to those generated by GAN-based models, one can see that the quality of images produced by DCGANx2 is comparable to the ones we obtain with GMV showing our approach has the same ability than GAN to generate good outputs.

Original: But DCGANx2 is only able to generate two views of each object since it does not distinguish content and view factors.
Revised: But DCGANx2 is only able to generate two views of each object since it does not distinguish content and view factors.

Original: For the same reason, the images in the same column (for different objects) do not necessarily match the same view.
Revised: For the same reason, the images in the same column (for different objects) do not necessarily match the same view.

Original: While DCGANx4 or DCGANx8 could have the ability to generate more views for each object, the learning problem is more difﬁcult due to the very high-dimensionality of the observation space, and the visual qualities of the generation degrade.
Revised: While DCGANx4 or DCGANx8 could have the ability to generate more views for each object, the learning problem is more difﬁcult due to the very high-dimensionality of the observation space, and the visual qualities of the generation degrade.

Original: Figure 6 shows generated samples obtained by interpolation between two different view factors (left) or two content factors (right).
Revised: Figure 6 shows generated samples obtained by interpolation between two different view factors (left) or two content factors (right).

Original: It allows us to have a better idea of the underlying view/content structure captured by GMV.
Revised: It allows us to have a better idea of the underlying view/content structure captured by GMV.

Original: We can see that our approach is able to smoothly move from one content/view to another content/view while keeping the other factor constant.
Revised: We can see that our approach is able to smoothly move from one content/view to another content/view while keeping the other factor constant.

Original: This illustrates also that content and view factors are well independently handled by the generator
Revised: This illustrates also that content and view factors are well independently handled by the generator

Original: i.e changing the view does not modify the content and vice-et-versa.
Revised: i.e changing the view does not modify the content and vice-et-versa.

Original: [Table]

Original: G ENERATING M ULTIPLE V IEWS OF A G IVEN O BJECT
Revised: G ENERATING M ULTIPLE V IEWS OF A G IVEN O BJECT

Original: The second set of experiments evaluates the ability of C-GMV to capture a particular content from an input sample, and to use this content to generate multiple views of the same object.
Revised: The second set of experiments evaluates the ability of C-GMV to capture a particular content from an input sample, and to use this content to generate multiple views of the same object.

Original: Figureillustrates the diversity of view in samples generated by our model and compares our results with those obtained with the CGAN model.
Revised: Figureand 8 illustrate the diversity of view in samples generated by our model and compare our results with those obtained with the CGAN model and to models from Mathieu et al.

Original: For each row the input sample is shown in the left column.
Revised: For each row the input sample is shown in the left column.

Original: New views are generated from that input and shown on the right.
Revised: New views are generated from that input and shown on the right.

Original: Concerning the CGAN approach, the mode collapse phenomenon that we previously described clearly occurs: the model does not take into account the view factor and always generate similar samples without any view diversity. 
Revised: Concerning the CGAN approach, the mode collapse phenomenon that we previously described clearly occurs: the model does not take into account the view factor and always generate similar samples without any view diversity. The C-GMV model demonstrates here that it is able to extract the content information, and to generate a large variety of views of any object for which we have one image.

Original: The
Revised: [Table]

Original: C-GMV model demonstrates here thatit is able to extract the content information, and to generate a large variety of views of any object for which we have one image.
Revised: One can see that the quality of the CGMV images clearly outperforms the quality of CGAN images. Moreover, mode collapse of CGAN can be observed, making this model unable to generate diversity. Finally comparison with the closest work from Mathieu et al. (2016) show that images generated by the CGMV model have a better quality and are more diverse.

Original: To do so, we estimate the performance of simple classiﬁers operating on images, either true samples or generated.
Revised: To do so, we estimate the performance of simple classiﬁers operating on images, either true samples or generated.

Original: We consider two subsets of objects S 1 and S 2 with a null intersection.
Revised: We consider two subsets of objects S 1 and S 2 with a null intersection.

Original: We then train a C-GMV model on training samples from S 1 .
Revised: We then train a C-GMV model on training samples from S 1 .

Original: We then use this model on test samples from S 2 , yielding both generated images corresponding to objects in S 2 but with new views, and content vectors for these images.
Revised: We then use this model on test samples from S 2 , yielding both generated images corresponding to objects in S 2 but with new views, and content vectors for these images.

Original: We then evaluate the performance of two classiﬁers.
Revised: We then evaluate the performance of two classiﬁers.

Original: One is learned and tested on content vectors.
Revised: One is learned and tested on content vectors.

Original: The other one is trained on real images and tested on generated images.
Revised: The other one is trained on real images and tested on generated images.

Original: The latter is also evaluated on real images on the test set of S 2 for reference.
Revised: The latter is also evaluated on real images on the test set of S 2 for reference.

Original: Table 2 reports such classiﬁcation results obtained over two of the studied datasets.
Revised: Table 2 reports such classiﬁcation results obtained over two of the studied datasets.

Original: On these two datasets, one can see that the accuracy of the classiﬁers operating on true images and on content latent factors are close, showing that our model has actually captured the category information in
Revised: On these two datasets, one can see that the accuracy of the classiﬁers operating on true images and on content latent factors are close, showing that our model has actually captured the category information in

Original: E ( x ) as it is desired.
Revised: E ( x ) as it is desired.

Original: Moreover, although the accuracy of the classiﬁer learned on real images is lower when computed on generated samples, which is of course expected, the drop of performances seems reasonable and shows that C-GMV is able to well reconstruct the content information. 
Revised: Moreover, although the accuracy of the classiﬁer learned on real images is lower when computed on generated samples, which is of course expected, the drop of performances seems reasonable and shows that C-GMV is able to well reconstruct the content information. 5. E VALUATION OF THE Q UALITY OF THE G ENERATED S AMPLES The next sets of experiments aimed at evaluating the quality of the generated samples.

Revised: They have been made on the CelebA dataset and evaluate (i) the ability of the models to preserve the identity of

Original: R ELATED W ORK
Revised: [Table] [Table] a person in multiple generated views, (ii) to generate realistic samples, (iii) to preserve the diversity in the generated views and (iv) to capture the view distributions of the original dataset. I DENTITY P RESERVATION

Revised: We propose a set of experiments to measure the ability of our model to generated multiple views of the same image i.e to preserve the identity of the person in two generated images.

Revised: In order to evaluate if two images belong to the same person, we extracted features using VGG Face descriptor (Parkhi et al.), which has been proposed for person reidentiﬁcation task.

Revised: The proximity between the face descriptors computed from two images reﬂects the likeness of the persons in these images.

Revised: Tables 3 and 4 illustrate the AUC obtained by the different models.

Revised: This AUC corresponds to the probability that a positive pair (i.e a pair with two images of the same person) is associated with a lower distance than a negative pair.

Revised: This AUC has been computed using 15 000 positive pairs and 15 000 negative ones.

Revised: An AUC of 100% corresponds to a perfect model.

Revised: First, one can see that when pairs of images are sampled directly from the dataset (Table 3, ﬁrst column), our classiﬁer obtains an AUC of 93.3% (and not 100 %).

Revised: This is due to the imperfection of the identity matching model that we use.

Revised: This value can thus be seen as an upper bound of the expected performance of the different models.

Revised: When comparing generative models (Table 3, columns 2 to 5), the GMV model obtains an AUC of 78.1 % which is a quite high value that outperforms the other generative models (GANx2, GANx4, and GANx8) that are less able to generate images of the same person.

Revised: It means that 78.1 % of the pairs generated with the same content vector c (but random view vectors) have a lower distance than pairs generated with two randomly sampled content vectors cand c 2 .

Revised: When comparing conditional models, the pair of images is generated based on a given input image.

Revised: The AUC thus measures the ability of the models to generate new views while preserving the identity of the input person.

Revised: We compare the CGMV model with the two implementations of the model by

Revised: Mathieu et al.

Revised: (2016) that we mentioned previously.

Revised: One can see on Table 4 (ﬁrst line) that the Mathieu et al.

Revised: (2016) (DCGAN) model outperforms our proposed CGMV model, obtaining an AUC of 77.3 % while our approach obtains 67.2 %.

Revised: It thus means that Mathieu et al.

Revised: (2016) is better to generate a view of an input image while preserving the identity of the person.

Revised: When comparing two generated images, the AUC of Mathieu et al.

Revised: (2016) is 85.4% while CGMV obtains an AUC of 76.1%.

Revised: At ﬁrst glance, it gives the impression that Mathieu et al.

Revised: (2016) is better than our approach.

Revised: Yet, looking at the CGAN score, one can see that this last model obtains a 100% AUC (second line).

Revised: In fact, this score is due to the fact that the CGAN model is unable to generate diversity in the generated views, and thus generates two images that are exactly the same and that thus are easy to classify as belonging to the same person.

Revised: This means that we also have to evaluate the different models in term of output quality (evaluate if the generated images are

Revised: [Table]

Revised: of good quality)

Revised: , and in term of diversity (i.e the generated views are different, and well capture the dataset distribution).

Revised: This is the focus of the next sections.

Revised: Q UALITY AND D IVERSITY OF THE G ENERATED I MAGES

Revised: To deeply evaluate the quality of generative models we studied the distribution of generated samples by our models and by the baselines on the Celeba dataset.

Revised: The idea is that a good generative model should generate samples with a distribution on attributes that is close to the one in the training set (while it is not actually included in the objective criterion).

Revised: A ﬁrst measure of the quality of the generated samples is based on the blurry attribute of the CelebA dataset.

Revised: This attribute identiﬁes blurry images in the dataset i.e images that are less realistic.

Revised: We have used this attribute to train a blurry-detection classiﬁer that has been used to evaluate the quantity of blurry generated images (Table 5), the idea being that a good method would generate the same proportion of blurry samples than the proportion of blurry images in the dataset.

Revised: The results are illustrated in Table 5.

Revised: While the Blurry attribute has probability 0.05 according to the true empirical distribution (this probability being estimated as 0.08 using attribute classiﬁers), it goes up to more than 0.99 with CGAN and Mathieu et al.

Revised: and it is above 0.5 for all GANx models and about 0.

Revised: for Mathieu et al.

Revised: (2016) (DCGAN), while it remains limited at most at 0.4 for GMV and CGMV.

Revised: GMV and CGMV models clearly generate less blurry images than the GAN/CGAN/Mathieu et al.

Revised: (2016) models giving the intuition that the generated images are more realistic using our technique.

Revised: Even if the evaluation method is imperfect, it tends to assess that the CGMV and GMV methods are better able to generate interesting outputs.

Revised: We also studied the distribution of generated samples by our models and by our baselines onbinary attributes of the CelebA dataset.

Revised: To understand the behaviour of a generative model we generated a set of samples with it, from which we computed the distribution of generated samples with respect to each of the 40 attributes (see Table 8), and we ﬁnally computed a distance between this distribution and the true dataset distribution.

Revised: Note that, as the ground truth on attributes is not available for generated samples, we used attribute labels provided by classiﬁers, one for every attribute, that have been learned on the Celeba training set to infer the values of the 40 attributes of a sample.

Revised: The distribution of generated samples is then estimated by using these classiﬁers and data generated from 3800 samples corresponding either to the 3800 test samples with randomly sampled view vector (for conditional models such as CGMV and Mathieu) or to 3800 randomly generated samples for pure generative models (such as GMV).

Revised: The estimated true distribution is estimated by using the 40 classiﬁers on 3800 test samples, it is noted the Celeba distribution hereafter.

Revised: For information we also provide the true empirical distribution (abusively noted as Real distribution hereafter) computed from the available attribute labels of the test samples.

Revised: All estimated distributions are detailed in the supplementary material.

Revised: We observed that many rare attributes were completely ignored (i.e. no occurence in generated samples) by models such as Mathieu et al.

Revised: (2016), CGAN, GANx8 and that, on the contrary, common attributes were sometimes over generated by such models (e.g. Young).

Revised: Globally one sees strong differences between models with GMV and CGMV seeming to better capture the dataset distribution, which next statistics reveal better.

Revised: Next, we computed a score for each generative model that is based on these distributions.

Revised: For each model and for each attribute we computed the Bhattacharyya distance (with value in [0 , ∞ ] )

Revised: [Table]

Revised: [Table]

Revised: between the distribution yielded by a model and the estimated true distribution.

Revised: We report the sum over attributes of these distances in Table 6.

Revised: As may be seen the distance computed between the estimated true distribution and the ones yielded by models GMV and CGMV are very low, below 0.1, actually lower than the distance between the true empirical distribution and the estimated true distribution.

Revised: Moreover the distance computed for Mathieu et al.

Revised: (2016) and Mathieu et al.

Revised: (2016) (DCGAN) models, CGAN and all GANx models are clearly higher than those of our models.

Revised: To conclude on this set of experiments, the GMV and CGMV models seem to be the best trade-off between identity-preservation and diversity in the generated views.

Revised: While Mathieu et al.

Revised: and CGAN tend to have a better identity preservation ability, it is a the price of generating samples with a much lower diversity than our approach which is the most able to capture the dataset distribution.

Revised: The above results show clear differences between GMV and CGMV models and state for the art methods such as Mathieu et al.

Revised: (2016) and CGAN.

Revised: Yet it does not fully inform about the diversity of generated samples which is a key feature for such generative models, either purely generative or conditional.

Revised: We conducted the following experiment which aims at evaluating the diversity through the number of unique combinations of the 40 attributes that occur in a set of generated samples.

Revised: Again to fairly compare results we use our attribute classiﬁers learned on the training set to label sample images according to the 40 attributes.

Revised: Looking at the true distribution of the test samples there are about 1795 unique combinations amongst the 3800 test samples, i.e. a ratio of 47 % of unique combinations (which is actually 57 % on the 200k samples of the training set).

Revised: For conditional models we generated samples by computing the content vector of the 3800 test samples, then by randomly sampling 3800 view vectors and ﬁnally generating 3800 samples.

Revised: We then computed the attribute labels of these samples and count the number of unique combinations of the 40 attributes amongst the 3800 images.

Revised: For pure generative models we randomly sampled 3800 pairs of content and view vectors and computed the same statistic (note that results for pure generative and conditional models are not fully comparable since a number of test samples correspond to a same individual).

Revised: Table 7 compares the obtained results.

Revised: These results clearly show the behaviour of CGAN and Mathieu et al.

Revised: (2016) models which fully ignore a number of rare attributes (see also Table 8 in Appendix) hence yield very poor diversity.

Revised: While Mathieu et al.

Revised: (2016) (DCGAN) is better than the original

Revised: Mathieu et al.

Revised: (2016) model it does not reach the diversity obtained with either GMV and CGMV models which are close to the estimated true empiricial diversity.

Revised:  R ELATED W ORK

Revised: Learning generative models using three sets of latent variables to describe a pair of objects has been proposed a long time ago and is known as inter-battery factor analysis (IBFA) Tucker (1958);

Original: A ﬁrst family of related methods casts the problem as a domain transfer task.
Revised: Klami et al. Such methods are very related to Canonical Correlation Analysis (CCA) and have been used to deal with multiview data to infer one view from another one and/or to improve classiﬁcation systems. To do so non linear variants have been proposed such as Tang et al. Li et al.

Revised: Our approach is based on the same assumptions than these methods i.e each view is generated based on one common latent factor decribing the ”content” of the object and with its own ”view” latent factor responsible on the difference between two observations.

Revised: The main difference comes from the way the model is learned: while IBFA methods usually rely on regularization terms and on particular factorization functions to capture these factors (e.g. Damianou et al.

Revised: , our model is much simpler (allowing scaling to large datasets) and makes use of a discriminator function to capture common and speciﬁc informations based on a pair of observations.

Original:  Different views are considered as different domains, and the problem becomes to project any image from a source domain to a target domain.
Revised: Another family of related methods casts the problem as a domain transfer task. Different views are considered as different domains, and the problem becomes to project any image from a source domain to a target domain.

Original: Most of those approaches combine a prediction or auto-encoding loss ( (cid:96) 1 ) with an adversarial loss that is tasked to enforce a good visual quality, a technique used in
Revised: Most of those approaches combine a prediction or auto-encoding loss ( (cid:96) 1 ) with an adversarial loss that is tasked to enforce a good visual quality, a technique used in

Original: Isola et al.
Revised: Isola et al.

Original: The discriminators also serve to ensure that the produced output is in the correct domain, meaning that a discriminator must be learned for each domain.
Revised: The discriminators also serve to ensure that the produced output is in the correct domain, meaning that a discriminator must be learned for each domain.

Original: However, those methods are powerful as they can discover alignment between two unpaired datasets, as shown in Kim et al.
Revised: However, those methods are powerful as they can discover alignment between two unpaired datasets, as shown in Kim et al.

Original: (2017a)
Revised: (2017a)

Original: ; Zhu et al.
Revised: ; Zhu et al.

Original: ; Liu et al.
Revised: ; Liu et al.

Original: (2017) using a cycle consistent auto-encoding loss.
Revised: (2017) using a cycle consistent auto-encoding loss.

Original: Another family of models, perhaps closer to our work, considers the problem of editing images based on manipulating the attributes.
Revised: A third family of models, perhaps closer to our work, considers the problem of editing images based on manipulating the attributes.

Original: In this setting, most of the models consider a learning dataset where some factors of variation are labeled for each sample.
Revised: In this setting, most of the models consider a learning dataset where some factors of variation are labeled for each sample.

Original: Those labels can be used to disentangle between content and view.
Revised: Those labels can be used to disentangle between content and view.

Original: For example, the model proposed in Lample et al.
Revised: For example, the model proposed in Lample et al.

Original: (2017) trains an autoencoder simultaneously with a discriminator used to remove the labeled information at a latent level.
Revised: (2017) trains an autoencoder simultaneously with a discriminator used to remove the labeled information at a latent level.

Original: The model presented in Zhao et al.
Revised: The model presented in Zhao et al.

Original: (2017) uses a variational auto-encoder framework instead.
Revised: (2017) uses a variational auto-encoder framework instead.

Original: The attribute is given as a word in input and the disentanglement is ensured by a conditional discriminator at output level.
Revised: The attribute is given as a word in input and the disentanglement is ensured by a conditional discriminator at output level.

Original: The model proposed in Kim et al.
Revised: The model proposed in Kim et al.

Original: (2017b) revisits the cycle approach by learning to generate outputs images with a given set of attribute values, and then to go back to the initial image.
Revised: (2017b) revisits the cycle approach by learning to generate outputs images with a given set of attribute values, and then to go back to the initial image.

Original: Inboth those  approaches, domain transfer and attribute manipulation, while GAN is used to ensure visual quality, most approaches are not generative in the sense that one input in the source domain always produces the same output in the target domain.
Revised: In those last approaches, domain transfer and attribute manipulation, while GAN is used to ensure visual quality, most approaches are not generative in the sense that one input in the source domain always produces the same output in the target domain.

Original: Also, in these settings, one usually makes the assumption that the number of domains (or varying attributes) is very limited, as additional networks must be trained for each new domain.
Revised: Also, in these settings, one usually makes the assumption that the number of domains (or varying attributes) is very limited, as additional networks must be trained for each new domain.

Original: Our approach is original as we don’t use information on the views, but instead, we just use the fact that two training samples represent the same content.
Revised: Our approach is original as we don’t use information on the views, but instead, we just use the fact that two training samples represent the same content.

Original: This allows our approach to handle continuous view and content latent spaces, and thus to generate as many contents as needed and any number of views over these contents.
Revised: This allows our approach to handle continuous view and content latent spaces, and thus to generate as many contents as needed and any number of views over these contents.

Original: Other works have aimed at disentangling content and view/style without any supervision, i.e based on unlabelled samples.
Revised: Other works have aimed at disentangling content and view/style without any supervision, i.e based on unlabelled samples.

Original: In the Info-GAN model (Chen et al.
Revised: In the Info-GAN model (Chen et al.

Original: , a code is passed along a random noise to generate an image.
Revised: , a code is passed along a random noise to generate an image.

Original: An encoder is then used to retrieve the code so that the mutual information between the generated image and the code is maximized.
Revised: An encoder is then used to retrieve the code so that the mutual information between the generated image and the code is maximized.

Original: The beta-VAE model proposed in
Revised: The beta-VAE model proposed in

Original: Higgins et al.
Revised: Higgins et al.

Original: (2016) is based on the VAE model, where the dimensions of the latent representation are forced to be as independent as possible using a strong regularization.
Revised: (2016) is based on the VAE model, where the dimensions of the latent representation are forced to be as independent as possible using a strong regularization.

Original: The independent dimensions can then correspond to independent factors of the outputs.
Revised: The independent dimensions can then correspond to independent factors of the outputs.

Original: For these two models, the ability to disentangle content and view is not obvious since there is no supervision that can guide the learning.
Revised: For these two models, the ability to disentangle content and view is not obvious since there is no supervision that can guide the learning.

Original: More speciﬁcally, these models disentangle low-level visual attributes but struggle to grasp higher level concepts.
Revised: More speciﬁcally, these models disentangle low-level visual attributes but struggle to grasp higher level concepts.

Original: The work closest to ours is the model proposed in Mathieu et al.
Revised: The work closest to ours is the model proposed in Mathieu et al.

Original: (2016) in which they use an encoder to extract both view and content vectors from a datapoint and a decoder that combines both vectors to produce an output.
Revised: (2016) in which they use an encoder to extract both view and content vectors from a datapoint and a decoder that combines both vectors to produce an output.

Original: They use both a reconstruction loss and a discriminator that serves multiple purpose.
Revised: They use both a reconstruction loss and a discriminator that serves multiple purpose.

Original: However, this work is mostly centered on disentangling the factors, and their purely generative abilities are limited.
Revised: However, this work is mostly centered on disentangling the factors, and their purely generative abilities are limited.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: We have proposed a generative model operating on a disentangled latent space which may be learned from multiview data without any view supervision, allowing its application to many multiview dataset.
Revised: We have proposed a generative model operating on a disentangled latent space which may be learned from multiview data without any view supervision, allowing its application to many multiview dataset.

Original: Our model allows generating realistic data with a rich view diversity.
Revised: Our model allows generating realistic data with a rich view diversity.

Original: We also proposed a conditional version of this model which allows generating new views of an input image which may again be learned without view supervison.
Revised: We also proposed a conditional version of this model which allows generating new views of an input image which may again be learned without view supervison.

Original: Our experimental results show the quality of the produced outputs, and the ability of the model to capture content and view factors.
Revised: Our experimental results show the quality of the produced outputs, and the ability of the model to capture content and view factors.

Revised: They also illustrate the ability of GMV and CGMV to capture the diversity over the CelebA dataset and to generate more realistic samples than baseline models.

Original: In a near future, we plan to investigate the use of such an approach for data augmentation.
Revised: In a near future, we plan to investigate the use of such an approach for data augmentation.

Original: Indeed, when only a few training data are available, one elegant solution for learning a model could be to generate new views of the existing data in order to increase the size of the training set.
Revised: Indeed, when only a few training data are available, one elegant solution for learning a model could be to generate new views of the existing data in order to increase the size of the training set.

Original: This solution will be explored in both semi-supervised and one-shot/few-shot learning settings
Revised: This solution will be explored in both semi-supervised and one-shot/few-shot learning settings