Original: T RAINING G ENERATIVE A DVERSARIAL N ETWORKS FROM I NCOMPLETE O BSERVATIONS USING F ACTORISED D ISCRIMINATORS
Revised: T RAINING G ENERATIVE A DVERSARIAL N ETWORKS FROM I NCOMPLETE O BSERVATIONS USING F ACTORISED D ISCRIMINATORS

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Generative adversarial networks (GANs) have shown great success in applications such as image generation and inpainting.
Revised: Generative adversarial networks (GANs) have shown great success in applications such as image generation and inpainting.

Original: However, they typically require large datasets, which are often not available, especially in the context of prediction tasks such as image segmentation that require labels.
Revised: However, they typically require large datasets, which are often not available, especially in the context of prediction tasks such as image segmentation that require labels.

Original: Therefore, methods such as the CycleGAN use more easily available unlabelled data, but do not offer a way to leverage additional labelled data for improved performance.
Revised: Therefore, methods such as the CycleGAN use more easily available unlabelled data, but do not offer a way to leverage additional labelled data for improved performance.

Original: To address this shortcoming, we show how to factorise the joint data distribution into a set of lower-dimensional distributions along with their dependencies.
Revised: To address this shortcoming, we show how to factorise the joint data distribution into a set of lower-dimensional distributions along with their dependencies.

Original: This allows splitting the discriminator in a GAN into multiple “sub-discriminators” that can be independently trained from incomplete observations.
Revised: This allows splitting the discriminator in a GAN into multiple “sub-discriminators” that can be independently trained from incomplete observations.

Original: Their outputs can be combined to estimate the density ratio between the joint real and the generator distribution, which enables training generators as in the original GAN framework.
Revised: Their outputs can be combined to estimate the density ratio between the joint real and the generator distribution, which enables training generators as in the original GAN framework.

Original: We apply our method to image generation, image segmentation and audio source separation, and obtain improved performance over a standard GAN when additional incomplete training examples are available.
Revised: We apply our method to image generation, image segmentation and audio source separation, and obtain improved performance over a standard GAN when additional incomplete training examples are available.

Original: For the Cityscapes segmentation task in particular, our method also improves accuracy by an absolute 13 .
Revised: For the Cityscapes segmentation task in particular, our method also improves accuracy by an absolute 14 .

Original: 6% over
Revised: 9% over

Original: CycleGAN while using only 25 additional paired examples.
Revised: CycleGAN while using only 25 additional paired examples.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: In generative adversarial networks (GANs) (Goodfellow et al., 2014) a generator network is trained to produce samples from a given target distribution.
Revised: In generative adversarial networks (GANs) (Goodfellow et al., 2014) a generator network is trained to produce samples from a given target distribution.

Original: To achieve this, a discriminator network is employed to distinguish between “real” samples from the dataset and “fake” samples from the generator network.
Revised: To achieve this, a discriminator network is employed to distinguish between “real” samples from the dataset and “fake” samples from the generator network.

Original: The discriminator’s feedback is used by the generator to improve its output.
Revised: The discriminator’s feedback is used by the generator to improve its output.

Original: While
Revised: While

Original: GANs have become highly effective at synthesising realistic examples even for complex data such as natural images (Radford et al., 2015; Karras et al., 2018), they typically rely on large training datasets.
Revised: GANs have become highly effective at synthesising realistic examples even for complex data such as natural images (Radford et al., 2015; Karras et al., 2018), they typically rely on large training datasets.

Original: These are not available in many cases, especially for prediction tasks such as audio source separation (Stoller et al., 2018) or image-to-image translation (Zhu et al., 2017).
Revised: These are not available in many cases, especially for prediction tasks such as audio source separation (Stoller et al., 2018) or image-to-image translation (Zhu et al., 2017).

Original: Instead, one often encounters many incomplete observations, such as unpaired images in image-to-image translation, or isolated source recordings in source separation.
Revised: Instead, one often encounters many incomplete observations, such as unpaired images in image-to-image translation, or isolated source recordings in source separation.

Original: However, standard GANs cannot be trained with these observations.
Revised: However, standard GANs cannot be trained with these observations.

Original: Recent approaches that work with unpaired data can not make use of additional paired data (Zhu et al., 2017) or lead to computational overhead due to additional generators and discriminators that model the inverse of the mapping of interest (Almahairi et al., 2018; Gan et al., 2017).
Revised: Recent approaches that work with unpaired data can not make use of additional paired data (Zhu et al., 2017) or lead to computational overhead due to additional generators and discriminators that model the inverse of the mapping of interest (Almahairi et al., 2018; Gan et al., 2017).

Original: For training the generator, multiple losses are combined whose interactions are not clear and that do not guarantee that the generator converges to the desired distribution.
Revised: For training the generator, multiple losses are combined whose interactions are not clear and that do not guarantee that the generator converges to the desired distribution.

Original: In this paper, we adapt the standard GAN framework to enable training predictive models with both paired and unpaired data as well as generative models with incomplete observations.
Revised: In this paper, we adapt the standard GAN framework to enable training predictive models with both paired and unpaired data as well as generative models with incomplete observations.

Original: To achieve this, we split the discriminator into multiple “marginal” discriminators, each modelling a separate set of dimensions of the input.
Revised: To achieve this, we split the discriminator into multiple “marginal” discriminators, each modelling a separate set of dimensions of the input.

Original: As this modiﬁcation on its own would ignore any dependencies between these parts, we incorporate two additional “dependency discriminators”, each focusing only on inter-part relationships.
Revised: As this modiﬁcation on its own would ignore any dependencies between these parts, we incorporate two additional “dependency discriminators”, each focusing only on inter-part relationships.

Original: We show how the outputs from these marginal and dependency discriminators can be recombined and used to estimate the same density ratios as in the original
Revised: We show how the outputs from these marginal and dependency discriminators can be recombined and used to estimate the same density ratios as in the original

Original: GAN framework – which enables training any generator network in an unmodiﬁed form.
Revised: GAN framework – which enables training any generator network in an unmodiﬁed form.

Original: In contrast to previous GANs, our approach only requires full observations to train the smaller dependency discriminator and can leverage much bigger, simpler datasets to train the marginal discriminators, which enables the generator to model the marginal distributions more accurately.
Revised: In contrast to previous GANs, our approach only requires full observations to train the smaller dependency discriminator and can leverage much bigger, simpler datasets to train the marginal discriminators, which enables the generator to model the marginal distributions more accurately.

Original: Additionally, prior knowledge about the marginals and dependencies can be incorporated into the architecture of each discriminator.
Revised: Additionally, prior knowledge about the marginals and dependencies can be incorporated into the architecture of each discriminator.

Original: Deriving from ﬁrst principles, we obtain a consistent adversarial learning framework without the need for extra losses that rely on more assumptions or conﬂict with the GAN objective.
Revised: Deriving from ﬁrst principles, we obtain a consistent adversarial learning framework without the need for extra losses that rely on more assumptions or conﬂict with the GAN objective.

Original: In our experiments, we apply our approach (“FactorGAN”) 1 to two image generation tasks (Sections 4.1 and 4.2), image segmentation (Section 4.3) and audio source separation (Section 4.4), and observe improved performance in missing data scenarios compared to a GAN.
Revised: In our experiments, we apply our approach (“FactorGAN”) 1 to two image generation tasks (Sections 4.1 and 4.2), image segmentation (Section 4.3) and audio source separation (Section 4.4), and observe improved performance in missing data scenarios compared to a GAN.

Original: For image segmentation, we also compare to the CycleGAN (Zhu et al., 2017), which does not require images to be paired with their segmentation maps.
Revised: For image segmentation, we also compare to the CycleGAN (Zhu et al., 2017), which does not require images to be paired with their segmentation maps.

Original: By leveraging both paired and unpaired examples with a uniﬁed adversarial objective, we achieve a substantially higher segmentation accuracy even with onlypaired samples  and a comparatively simple network architecture.
Revised: By leveraging both paired and unpaired examples with a uniﬁed adversarial objective, we achieve a substantially higher segmentation accuracy even with onlypaired samples than GAN and CycleGAN models.

Original:  M ETHOD
Revised:  M ETHOD

Original: After a brief summary of GANs in Section 2.1, we introduce our method from a missing data perspective in Section 2.2, before extending it to conditional generation (Section 2.3) and the case of independent outputs (Section 2.4).
Revised: After a brief summary of GANs in Section 2.1, we introduce our method from a missing data perspective in Section 2.2, before extending it to conditional generation (Section 2.3) and the case of independent outputs (Section 2.4).

Original: 2. G ENERATIVE ADVERSARIAL NETWORKS
Revised: 2. G ENERATIVE ADVERSARIAL NETWORKS

Original: To model a probability distribution p x over x ∈ R d , we follow the standard GAN framework and introduce a generator model G φ : R n → R d that maps an n -dimensional input z ∼ p z to a d -dimensional sample G φ ( z ) , resulting in the generator distribution q x .
Revised: To model a probability distribution p x over x ∈ R d , we follow the standard GAN framework and introduce a generator model G φ : R n → R d that maps an n -dimensional input z ∼ p z to a d -dimensional sample G φ ( z ) , resulting in the generator distribution q x .

Original: To train G φ such that q x approximates the real data density p x , a discriminator D θ : R d → (0 , 1) is trained to estimate whether a given sample is real or generated:
Revised: To train G φ such that q x approximates the real data density p x , a discriminator D θ : R d → (0 , 1) is trained to estimate whether a given sample is real or generated:

Original: [Equation]
Revised: [Equation]

Original: In the non-parametric limit (Goodfellow et al., 2014), D θ ( x ) approaches ˜ D ( x ) :
Revised: In the non-parametric limit (Goodfellow et al., 2014), D θ ( x ) approaches ˜ D ( x ) :

Original: = p x ( x ) p x ( x )+ q x ( x ) at every point x .
Revised: [Equation] ( x ) p x ( x )+ q x ( x ) at every point x .

Original: The generator is updated based on the discriminator’s estimate of ˜ D ( x ) .
Revised: The generator is updated based on the discriminator’s estimate of ˜ D ( x ) .

Original: In this paper, we use the alternative loss function for G φ as proposed by Goodfellow et al.
Revised: In this paper, we use the alternative loss function for G φ as proposed by Goodfellow et al.

Original: [Equation]
Revised: [Equation]

Original: 2. A DAPTATION TO MISSING DATA
Revised: 2. A DAPTATION TO MISSING DATA

Original: In the following we consider the case that incomplete observations are available in addition to our regular dataset (i.e. simpler yet larger datasets).
Revised: In the following we consider the case that incomplete observations are available in addition to our regular dataset (i.e. simpler yet larger datasets).

Original: In particular, we partition the set of d input dimensions of x into K ( 2 ≤ K ≤ d ) non-overlapping subsets D 1 , . . .
Revised: In particular, we partition the set of d input dimensions of x into K ( 2 ≤ K ≤ d ) non-overlapping subsets D 1 , . . .

Original: , D K .
Revised: , D K .

Original: For each i ∈ { 1 , . . .
Revised: For each i ∈ { 1 , . . .

Original: , K } , an incomplete (“marginal”)
Revised: , K } , an incomplete (“marginal”)

Original: observation x i can be drawn from p ix , which is obtained from p x after marginalising out all dimensions not in D i .
Revised: observation x i can be drawn from p ix , which is obtained from p x after marginalising out all dimensions not in D i .

Original: Analogously, q ix denotes the i -th marginal distribution of the generator G φ .
Revised: Analogously, q ix denotes the i -th marginal distribution of the generator G φ .

Original: Next, we extend the existing GAN framework such we can employ the additional incomplete observations.
Revised: Next, we extend the existing GAN framework such we can employ the additional incomplete observations.

Original: In this context, a main hurdle is that a standard GAN discriminator is trained with samples from the full joint p x .
Revised: In this context, a main hurdle is that a standard GAN discriminator is trained with samples from the full joint p x .

Original: To eliminate this restriction, we note that ˜ D ( x ) can be mapped to a “joint density ratio” p x ( x ) q x ( x ) by applying the bijective function h : [0 , 1) → R + , h
Revised: To eliminate this restriction, we note that ˜ D ( x ) can be mapped to a “joint density ratio” p x ( x ) q x ( x ) by applying the bijective function h : [0 , 1) → R + , h

Original: ( a )
Revised: ( a )

Original: = − aa − 1 .
Revised: = − aa − 1 .

Original: For our approach, we exploit that this joint density ratio can be factorised into a product of density ratios:
Revised: For our approach, we exploit that this joint density ratio can be factorised into a product of density ratios:

Original: [Equation]
Revised: [Equation]

Original: i = pix
Revised: p

Original: [Equation]
Revised: [Equation]

Original: x
Revised: x

Original: [Equation]
Revised: [Equation]

Original: q ix ( x i ) with
Revised: q ix ( x i ) with

Revised: [Equation]

Revised: p

Original: [Equation]
Revised: [Equation]

Revised: x i

Original:  Each “marginal density ratio” p ix ( x i ) q ix ( x i ) captures the generator’s output quality for one marginal variable x i , while the c P and c Q terms describe the dependency structure between marginal variables in the real and generated distribution, respectively.
Revised: [Equation] c Q ( x [Equation] Each “marginal density ratio” p i x ( x i ) q ix ( x i ) captures the generator’s output quality for one marginal variable x i , while the c P and c Q terms describe the dependency structure between marginal variables in the real and generated distribution, respectively.

Original: Note that our theoretical considerations assume that the densities p x and q x are non-zero everywhere.
Revised: Note that our theoretical considerations assume that the densities p x and q x are non-zero everywhere.

Original: While this might not be fulﬁlled in practice, our implementation does not directly compute density ratios and instead relies on the same assumptions as Goodfellow et al.
Revised: While this might not be fulﬁlled in practice, our implementation does not directly compute density ratios and instead relies on the same assumptions as Goodfellow et al.

Original: We can estimate each density ratio independently by training a “sub-discriminator” network, and combine their outputs to estimate ˜ D ( x ) , as shown below.
Revised: We can estimate each density ratio independently by training a “sub-discriminator” network, and combine their outputs to estimate ˜ D ( x ) , as shown below.

Original: Estimating the marginal density ratios: To estimate p ix ( x i ) q ix ( x i ) for each i ∈ { 1 , . .
Revised: Estimating the marginal density ratios: To estimate p ix ( x i ) q ix ( x i ) for each i ∈ { 1 , . .

Original: , K } , we train a “marginal discriminator network” D θ i : R |D i | → (0 , 1) with parameters θ i to determine whether a marginal sample x i is real or generated following the GAN discriminator loss in Equation (1) 2 .
Revised: , K } , we train a “marginal discriminator network” D θ i : R |D i | → (0 , 1) with parameters θ i to determine whether a marginal sample x i is real or generated following the GAN discriminator loss in Equation (1) 2 .

Original: This allows making use of the additional incomplete observations.
Revised: This allows making use of the additional incomplete observations.

Original: In the non-parametric limit, D θ i ( x i ) will approach ˜ D
Revised: In the non-parametric limit, D θ i ( x i ) will approach ˜ D

Original: i ( x i ) :
Revised: i ( x i ) :

Original: = p ix ( x i ) p ix ( x i )+ q ix ( x i ) , so that we can use h ( D θ i ( x i )) as an estimate of p ix ( x i ) q ix ( x i ) .
Revised: = p ix ( x i ) p ix ( x i )+ q ix ( x i ) , so that we can use h ( D θ i ( x i )) as an estimate of p ix ( x i ) q ix ( x i ) .

Original: Estimation of c P ( x ) and c Q ( x ) : Note that c P and c Q are also density ratios, this time containing a distribution over x in both the numerator and denominator – the main difference being that in the latter the individual parts x i are independent from each other.
Revised: Estimation of c P ( x ) and c Q ( x ) : Note that c P and c Q are also density ratios, this time containing a distribution over x in both the numerator and denominator – the main difference being that in the latter the individual parts x i are independent from each other.

Original: To approximate the ratio c P , we can apply the same principles as above and train a “p-dependency discriminator” D Pθ P : R d → (0 , 1) to distinguish samples from the two distributions, i.e. to discriminate real joint samples from samples where the individual parts are real but were drawn independently of each other (i.e. the individual parts might not originate from the same real joint sample).
Revised: To approximate the ratio c P , we can apply the same principles as above and train a “p-dependency discriminator” D Pθ P : R d → (0 , 1) to distinguish samples from the two distributions, i.e. to discriminate real joint samples from samples where the individual parts are real but were drawn independently of each other (i.e. the individual parts might not originate from the same real joint sample).

Original: Again, in the non-parametric limit, its response approaches ˜ D P ( x ) :=
Revised: Again, in the non-parametric limit, its response approaches ˜ D P ( x ) :

Original: [Equation] x i ) and thus c P can be approximated via h ◦ D Pθ P .
Revised: = p x ( x ) p x ( x )+ (cid:81) Ki =1 p ix ( x i ) and thus c P can be approximated via h ◦ D Pθ P .

Original: Analogously, the c Q term is estimated with a “q-dependency discriminator” D Qθ Q – here, we compare joint generator samples with samples where the individual parts were shufﬂed across several generated samples (to implement the independence assumption).
Revised: Analogously, the c Q term is estimated with a “q-dependency discriminator” D Qθ Q – here, we compare joint generator samples with samples where the individual parts were shufﬂed across several generated samples (to implement the independence assumption).

Original: Joint discriminator sample complexity:
Revised: Joint discriminator sample complexity:

Original: In contrast to c Q , where the generator provides an inﬁnite number of samples, estimating c P without overﬁtting to the limited number of joint training samples can be challenging.
Revised: In contrast to c Q , where the generator provides an inﬁnite number of samples, estimating c P without overﬁtting to the limited number of joint training samples can be challenging.

Original: While standard GANs suffer from the same difﬁculty, our factorisation into specialised sub-units allows for additional opportunities to improve the sample complexity.
Revised: While standard GANs suffer from the same difﬁculty, our factorisation into specialised sub-units allows for additional opportunities to improve the sample complexity.

Original: In particular, we can design the architecture of the p-dependency discriminator to incorporate prior knowledge about the dependency structure 3 .
Revised: In particular, we can design the architecture of the p-dependency discriminator to incorporate prior knowledge about the dependency structure 3 .

Original: Combining the discriminators: As the marginal and the p- and q-dependency sub-discriminators provide estimates of their respective density ratios, we can multiply them and apply h − 1 to obtain the desired ratio ˜ D ( x ) , following Equation (3).
Revised: Combining the discriminators: As the marginal and the p- and q-dependency sub-discriminators provide estimates of their respective density ratios, we can multiply them and apply h − 1 to obtain the desired ratio ˜ D ( x ) , following Equation (3).

Original: This can be implemented in a simple and stable fashion using a linear combination of pre-activation sub-discriminator outputs followed by a sigmoid (see Section A.4 for details and proof).
Revised: This can be implemented in a simple and stable fashion using a linear combination of pre-activation sub-discriminator outputs followed by a sigmoid (see Section A.4 for details and proof).

Original: The time for a generator update step grows linearly with the number of marginals K , assuming the time to update each of the K marginal discriminators remains constant.
Revised: The time for a generator update step grows linearly with the number of marginals K , assuming the time to update each of the K marginal discriminators remains constant.

Original: 2. A DAPTATION TO CONDITIONAL GENERATION
Revised: 2. A DAPTATION TO CONDITIONAL GENERATION

Original: Conditional generation, such as image segmentation or inpainting, can be performed with GANs by using a generator G φ that maps a conditional input x 1 and noise to an output x 2 , resulting in an output probability q φ ( x 2 | x 1 ) .
Revised: Conditional generation, such as image segmentation or inpainting, can be performed with GANs by using a generator G φ that maps a conditional input x 1 and noise to an output x 2 , resulting in an output probability q φ ( x 2 | x 1 ) .

Original: When viewing x 1 and x 2 as parts of a joint variable x := ( x 1 , x 2 ) with distribution p x , we can also frame the above task as matching p x to the joint generator distribution q x ( x ) :
Revised: When viewing x 1 and x 2 as parts of a joint variable x := ( x 1 , x 2 ) with distribution p x , we can also frame the above task as matching p x to the joint generator distribution q x ( x ) :

Original: = p 1 x ( x 1 ) q φ ( x 2 | x 1 ) .
Revised: = p 1 x ( x 1 ) q φ ( x 2 | x 1 ) .

Original: In a standard conditional GAN, the discriminator is asked to distinguish between joint samples from p x and q x , which requires paired samples from p x and is inefﬁcient as the inputs x 1 are the same in both p x and q x .
Revised: In a standard conditional GAN, the discriminator is asked to distinguish between joint samples from p x and q x , which requires paired samples from p x and is inefﬁcient as the inputs x 1 are the same in both p x and q x .

Original: In contrast, applying our factorisation principle from Equation (3) to x 1 and x 2 (for the special case K = 2 ) yields
Revised: In contrast, applying our factorisation principle from Equation (3) to x 1 and x 2 (for the special case K = 2 ) yields

Original: [Equation]
Revised: [Equation]

Original: suggesting the use of a p- and a q-dependency discriminator to model the input-output relationship, and a marginal discriminator over x 2 that matches aggregate generator predictions from q 2 x to real 2 Samples are drawn from p ix and q ix instead of p x and q x , respectively.
Revised: suggesting the use of a p- and a q-dependency discriminator to model the input-output relationship, and a marginal discriminator over x 2 that matches aggregate generator predictions from q 2 x to real 2 Samples are drawn from p ix and q ix instead of p x and q x , respectively.

Original: If only certain features of a marginal variable inﬂuence the dependencies, we can limit the input to the p-dependency discriminator to these features instead of the full marginal sample to prevent overﬁtting.
Revised: If only certain features of a marginal variable inﬂuence the dependencies, we can limit the input to the p-dependency discriminator to these features instead of the full marginal sample to prevent overﬁtting.

Original: output examples from p 2 x .
Revised: output examples from p 2 x .

Original: Note that we do not need a marginal discriminator for x 1 , which increases computational efﬁciency.
Revised: Note that we do not need a marginal discriminator for x 1 , which increases computational efﬁciency.

Original: This adaptation can also involve additionally partitioning x 2 into multiple partial observations as shown in Equation 3.
Revised: This adaptation can also involve additionally partitioning x 2 into multiple partial observations as shown in Equation 3.

Original: 2. A DAPTION TO INDEPENDENT MARGINALS
Revised: 2. A DAPTION TO INDEPENDENT MARGINALS

Original: In case the marginals can be assumed to be completely independent, one can remove the p-dependency discriminator from our framework, since c P ( x ) = 1 for all inputs x .
Revised: In case the marginals can be assumed to be completely independent, one can remove the p-dependency discriminator from our framework, since c P ( x ) = 1 for all inputs x .

Original: This approach can be useful in the conditional setting, when each output is related to the input but their marginals are independent from each other.
Revised: This approach can be useful in the conditional setting, when each output is related to the input but their marginals are independent from each other.

Original: In this setting, our method is related to adversarial ICA (Brakel & Bengio, 2017).
Revised: In this setting, our method is related to adversarial ICA (Brakel & Bengio, 2017).

Original: Note that the q-dependency discriminator still needs to be trained on the full generator outputs if the generator should not introduce unwanted dependencies between the marginals.
Revised: Note that the q-dependency discriminator still needs to be trained on the full generator outputs if the generator should not introduce unwanted dependencies between the marginals.

Original: 2. F URTHER EXTENSIONS
Revised: 2. F URTHER EXTENSIONS

Original: There are many more ways of partitioning the joint distribution into marginals.
Revised: There are many more ways of partitioning the joint distribution into marginals.

Original: We discuss two additional variants ( Hierarchical and auto-regressive FactorGANs ) of our approach in Section A.3.
Revised: We discuss two additional variants ( Hierarchical and auto-regressive FactorGANs ) of our approach in Section A.3.

Original:  R ELATED WORK
Revised:  R ELATED WORK

Original: For conditional generation,the “CycleGAN” (Zhu et al., 2017) exploits unpaired samples by assuming a one-to-one mapping between the domains and using bidirectional generators (along with Gan et al.
Revised: For conditional generation, “CycleGAN” (Zhu et al., 2017) exploits unpaired samples by assuming a one-to-one mapping between the domains and using bidirectional generators (along with Gan et al.

Original: (2017)), while FactorGAN makes no such assumptions and instead uses paired examples to learn the dependency structure.
Revised: (2017)), while FactorGAN makes no such assumptions and instead uses paired examples to learn the dependency structure.

Original: The “Augmented CycleGAN” (Almahairi et al., 2018) learns from paired examples with an additional reconstruction-based loss, but introduces hyper-parameters for weighting the different loss terms.
Revised: Almahairi et al. (2018) and Tripathy et al. (2018) learn from paired examples with an additional reconstruction-based loss, but use a sum of many different loss terms which have to be balanced by additional hyper-parameters.

Revised: Additionally, it can not be applied to generation tasks with missing data or prediction tasks with multiple outputs.

Original: Brakel & Bengio (2017) perform independent component analysis in an adversarial fashion using a discriminator to identify correlations.
Revised: Brakel & Bengio (2017) perform independent component analysis in an adversarial fashion using a discriminator to identify correlations.

Original: Similarly to our q-dependency discriminator, the separator outputs are enforced to be independent, but our method is fully adversarial and can model arbitrary dependencies with the p-dependency discriminator.
Revised: Similarly to our q-dependency discriminator, the separator outputs are enforced to be independent, but our method is fully adversarial and can model arbitrary dependencies with the p-dependency discriminator.

Original: GANs were also used for source separation, butsource dependencies were either ignored (Zhang et al., 2017) or modelled with an additional L2 loss (Stoller et al., 2018) that supports only deterministic separators.
Revised: GANs were also used for source separation, but dependencies were either ignored (Zhang et al., 2017) or modelled with an additional L2 loss (Stoller et al., 2018) that supports only deterministic separators.

Original: Pu et al.
Revised: Pu et al.

Original: (2018) use GANs for joint distribution modelling by training a generator for each possible factorisation of the joint distribution, but this requires K !
Revised: (2018) use GANs for joint distribution modelling by training a generator for each possible factorisation of the joint distribution, but this requires K !

Original: generators for K marginals, whereas we assume either all parts or exactly one part of the variable of interest is observed to avoid functional redundancies between the different networks.
Revised: generators for K marginals, whereas we assume either all parts or exactly one part of the variable of interest is observed to avoid functional redundancies between the different networks.

Original: Karaletsos (2016) propose adversarial inference on local factors of a high-dimensional joint distribution and factorise both generator and discriminator based on independence assumptions given by a Bayesian network, whereas we keep a joint sample generator and model all dependencies.
Revised: Karaletsos (2016) propose adversarial inference on local factors of a high-dimensional joint distribution and factorise both generator and discriminator based on independence assumptions given by a Bayesian network, whereas we keep a joint sample generator and model all dependencies.

Original: Finally, Yoon et al.
Revised: Finally, Yoon et al.

Original: (2018) randomly mask the inputs to a GAN generator so it learns to impute missing values, whereas our generator aims to learn a transformation where inputs are fully observed.
Revised: (2018) randomly mask the inputs to a GAN generator so it learns to impute missing values, whereas our generator aims to learn a transformation where inputs are fully observed.

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: To validate our method, we compare our FactorGAN with the regular GAN approach, both for unsupervised generation as well as supervised prediction tasks.
Revised: To validate our method, we compare our FactorGAN with the regular GAN approach, both for unsupervised generation as well as supervised prediction tasks.

Original: For the latter, we also compare to the
Revised: For the latter, we also compare to the

Original: CycleGAN (Zhu et al., 2017) as an unsupervised baseline.
Revised: CycleGAN (Zhu et al., 2017) as an unsupervised baseline.

Original: To investigate whether FactorGAN makes efﬁcient use of all observations, we vary the proportion of the training samples available for joint sampling (paired), while using the rest to sample from the marginals (unpaired).
Revised: To investigate whether FactorGAN makes efﬁcient use of all observations, we vary the proportion of the training samples available for joint sampling (paired), while using the rest to sample from the marginals (unpaired).

Original: We train all models using a single NVIDIA GTX 1080 GPU.
Revised: We train all models using a single NVIDIA GTX 1080 GPU.

Original: Training procedure For stable training, we employ spectral normalisation (Miyato et al., 2018) on each discriminator network to ensure they satisfy a Lipschitz condition.
Revised: Training procedure For stable training, we employ spectral normalisation (Miyato et al., 2018) on each discriminator network to ensure they satisfy a Lipschitz condition.

Original: Since the overall output used for training the generator is simply a linear combination of the individual discriminators (see
Revised: Since the overall output used for training the generator is simply a linear combination of the individual discriminators (see

Original: Section A.4), the generator gradients are also constrained in magnitude accordingly.
Revised: Section A.4), the generator gradients are also constrained in magnitude accordingly.

Original: Unless otherwise noted, we use an Adam optimiser with learning rate 10 − 4 and a batch size of 25 for training all models.
Revised: Unless otherwise noted, we use an Adam optimiser with learning rate 10 − 4 and a batch size of 25 for training all models.

Original: We perform two discriminator updates after each generator update.
Revised: We perform two discriminator updates after each generator update.

Original: [Figure]
Revised: [Figure]

Original: 4. P AIRED MNIST
Revised: 4. P AIRED MNIST

Original: Our ﬁrst experiment will involve “Paired MNIST”, a synthetic dataset of low complexity whose dependencies between marginals can be easily controlled.
Revised: Our ﬁrst experiment will involve “Paired MNIST”, a synthetic dataset of low complexity whose dependencies between marginals can be easily controlled.

Original: More precisely, we generate a paired version of the original MNIST dataset 4 by creating samples that contain a pair of vertically stacked digit images.
Revised: More precisely, we generate a paired version of the original MNIST dataset 4 by creating samples that contain a pair of vertically stacked digit images.

Original: With a probability of λ , the lower digit chosen during random generation is the same as the upper one, and different otherwise.
Revised: With a probability of λ , the lower digit chosen during random generation is the same as the upper one, and different otherwise.

Original: For FactorGAN, we model the distributions of upper and lower digits as individual marginal distributions ( K = 2 ).
Revised: For FactorGAN, we model the distributions of upper and lower digits as individual marginal distributions ( K = 2 ).

Original: Experimental setup We compare the normal GAN with our FactorGAN, also including a variant without p-dependency discriminator that assumes marginals to be independent (“FactorGAN-no-cp”).
Revised: Experimental setup We compare the normal GAN with our FactorGAN, also including a variant without p-dependency discriminator that assumes marginals to be independent (“FactorGAN-no-cp”).

Original: We conduct the experiment with λ = 0 .
Revised: We conduct the experiment with λ = 0 .

Original: 1 and λ = 0 .
Revised: 1 and λ = 0 .

Original: 9 and also vary the amount of training samples available in paired form, while keeping the others as marginal samples only usable by FactorGAN.
Revised: 9 and also vary the amount of training samples available in paired form, while keeping the others as marginal samples only usable by FactorGAN.

Original: For both generators and discriminators, we used simple multi-layer perceptrons (Tables 1 and 2).
Revised: For both generators and discriminators, we used simple multi-layer perceptrons (Tables 1 and 2).

Original: To evaluate the quality of generated digits, we adopt the “Frecht Inception Distance” (FID) as metric (Heusel et al., 2017).
Revised: To evaluate the quality of generated digits, we adopt the “Frecht Inception Distance” (FID) as metric (Heusel et al., 2017).

Original: It is based on estimating the distance between the distributions of hidden layer activations of a pre-trained Imagenet object detection model for real and fake examples.
Revised: It is based on estimating the distance between the distributions of hidden layer activations of a pre-trained Imagenet object detection model for real and fake examples.

Original: To adapt the metric to MNIST data, we pre-train a classiﬁer to predict MNIST digits (see Table 3) on the training set for 4 epochs, obtaining a test accuracy of 94% .
Revised: To adapt the metric to MNIST data, we pre-train a classiﬁer to predict MNIST digits (see Table 3) on the training set for 4 epochs, obtaining a test accuracy of 94% .

Original: We input the top and bottom digits in each sample separately to the classiﬁer and collect the activations from the last hidden layer (FC1) to compute FIDs for the top and bottom digits, respectively.
Revised: We input the top and bottom digits in each sample separately to the classiﬁer and collect the activations from the last hidden layer (FC1) to compute FIDs for the top and bottom digits, respectively.

Original: We use the average of both FIDs to measure the overall output quality of the marginals (lower value is better).
Revised: We use the average of both FIDs to measure the overall output quality of the marginals (lower value is better).

Original: Since the only dependencies in the data are digit correlations controlled by λ , we can evaluate how well FactorGAN models these dependencies.
Revised: Since the only dependencies in the data are digit correlations controlled by λ , we can evaluate how well FactorGAN models these dependencies.

Original: We compute p D ( D t , D b ) as the probability for a real sample to have digit D t ∈ { 0 , . . .
Revised: We compute p D ( D t , D b ) as the probability for a real sample to have digit D t ∈ { 0 , . . .

Original: , 9 } at the top and digit D b ∈ { 0 , . . .
Revised: , 9 } at the top and digit D b ∈ { 0 , . . .

Original: , 9 } at the bottom, along with marginal probabilities p tD ( D t ) and p bD ( D b ) (and analogously q D ( D t , D b ) for generated data).
Revised: , 9 } at the bottom, along with marginal probabilities p tD ( D t ) and p bD ( D b ) (and analogously q D ( D t , D b ) for generated data).

Original: Since we do not have ground truth digit labels for the generated samples, we instead use the class predicted by the pre-trained classiﬁer.
Revised: Since we do not have ground truth digit labels for the generated samples, we instead use the class predicted by the pre-trained classiﬁer.

Original: We encode the dependency as a ratio between a joint and the product of its marginals, where the ratios for real and generated data are ideally the same.
Revised: We encode the dependency as a ratio between a joint and the product of its marginals, where the ratios for real and generated data are ideally the same.

Original: Therefore, we take their absolute difference for all digit combinations as evaluation metric (lower is better):[Equation]
Revised: Therefore, we take their absolute difference for all digit combinations as evaluation metric (lower is better):

Original: [Equation]
Revised: [Equation]

Original: Note that the metric computes how well dependencies in the real data are modelled by a generator, but not whether it introduces any additional unwanted dependencies such as top and bottom digits sharing stroke thickness, and thus presents only a necessary condition for a good generator.
Revised: Note that the metric computes how well dependencies in the real data are modelled by a generator, but not whether it introduces any additional unwanted dependencies such as top and bottom digits sharing stroke thickness, and thus presents only a necessary condition for a good generator.

Original: Results
Revised: Results

Original: The results of our experiment are shown in Figure 1.
Revised: The results of our experiment are shown in Figure 1.

Original: Since FactorGAN-no-cp trains on all samples independently of the number of paired observations, both FID and d dep are constant.As
Revised: Since FactorGAN-no-cp trains on all samples independently of the number of paired observations, both FID and d dep are constant.

Original: [Figure] expected, FactorGAN-no-cp delivers good digit quality, and performs well for λ = 0 .
Revised: As expected, FactorGAN-no-cp delivers good digit quality, and performs well for λ = 0 .

Original: 1 (as it assumes independence) and badly for λ = 0 .
Revised: 1 (as it assumes independence) and badly for λ = 0 .

Original: 9 with regards to dependency modelling.
Revised: 9 with regards to dependency modelling.

Original:  FactorGAN outperforms GAN with small numbers of paired samples in terms of FID by exploiting the additional unpaired samples, although this gap closes as both models eventually have access to the same amount of data.
Revised: [Figure] FactorGAN outperforms GAN with small numbers of paired samples in terms of FID by exploiting the additional unpaired samples, although this gap closes as both models eventually have access to the same amount of data.

Original: FactorGAN also consistently improves in modelling the digit dependencies with an increasing number of paired observations.
Revised: FactorGAN also consistently improves in modelling the digit dependencies with an increasing number of paired observations.

Original: For λ = 0 .
Revised: For λ = 0 .

Original: 1 , this also applies to the normal
Revised: 1 , this also applies to the normal

Original: GAN, although its performance is much worse for small sample sizes as it introduces unwanted digit dependencies.
Revised: GAN, although its performance is much worse for small sample sizes as it introduces unwanted digit dependencies.

Original: Additionally, its performance appears unstable for λ = 0 .
Revised: Additionally, its performance appears unstable for λ = 0 .

Original: 9 , where it achieves the best results for a small number of paired examples.
Revised: 9 , where it achieves the best results for a small number of paired examples.

Original: Further improvements in this setting could be gained by incorporating prior knowledge about the nature of these dependencies into the p-dependency discriminator to increase its sample efﬁciency, but this is left for future work.
Revised: Further improvements in this setting could be gained by incorporating prior knowledge about the nature of these dependencies into the p-dependency discriminator to increase its sample efﬁciency, but this is left for future work.

Original: 4. I MAGE PAIR GENERATION
Revised: 4. I MAGE PAIR GENERATION

Original: In this section, we use GAN and FactorGAN for generating pairs of images in an unsupervised way to evaluate how well FactorGAN models more complex data distributions.
Revised: In this section, we use GAN and FactorGAN for generating pairs of images in an unsupervised way to evaluate how well FactorGAN models more complex data distributions.

Original: Datasets We use the “Cityscapes” dataset (Cordts et al., 2016) and the “Edges2Shoes” dataset (Isola et al., 2016).
Revised: Datasets We use the “Cityscapes” dataset (Cordts et al., 2016) and the “Edges2Shoes” dataset (Isola et al., 2016).

Original: To keep the outputs in a continuous domain, we treat the segmentation maps in the
Revised: To keep the outputs in a continuous domain, we treat the segmentation maps in the

Original: Cityscapes dataset as RGB images, instead of a set of discrete categorical labels.
Revised: Cityscapes dataset as RGB images, instead of a set of discrete categorical labels.

Original: Each input and output image is downsampled to 64 × 64 pixels as a preprocessing step to reduce computational complexity and to ensure stable GAN training.
Revised: Each input and output image is downsampled to 64 × 64 pixels as a preprocessing step to reduce computational complexity and to ensure stable GAN training.

Original: Experimental setup We deﬁne the distributions of input as well as output images as marginal distributions.
Revised: Experimental setup We deﬁne the distributions of input as well as output images as marginal distributions.

Original: Therefore, FactorGAN uses two marginal discriminators and a p- and q-dependency discriminator.
Revised: Therefore, FactorGAN uses two marginal discriminators and a p- and q-dependency discriminator.

Original: All discriminators employ a convolutional architecture shown in Table 5 with W =and H = 6 .
Revised: All discriminators employ a convolutional architecture shown in Table 5 with W =and H = 6 .

Original: To control for the impact of discriminator size, we also train a GAN with twice the number of ﬁlters in each discriminator layer to match its size with the combined size of the
Revised: To control for the impact of discriminator size, we also train a GAN with twice the number of ﬁlters in each discriminator layer to match its size with the combined size of the

Original: FactorGAN discriminators.
Revised: FactorGAN discriminators.

Original: The same convolutional generator shown in Table 4 is used for GAN and FactorGAN.
Revised: The same convolutional generator shown in Table 4 is used for GAN and FactorGAN.

Original: Each image pair is concatenated along the channel dimension to form one sample, so that C = 6 for the Cityscapes and C = 4 for the Edges2Shoes dataset (since edge maps are greyscale).
Revised: Each image pair is concatenated along the channel dimension to form one sample, so that C = 6 for the Cityscapes and C = 4 for the Edges2Shoes dataset (since edge maps are greyscale).

Original: We make either 100 , 1000 , or all training samples available in paired form, to investigate whether FactorGAN can improve upon GAN by exploiting the remaining unpaired samples or match its quality if there are none.
Revised: We make either 100 , 1000 , or all training samples available in paired form, to investigate whether FactorGAN can improve upon GAN by exploiting the remaining unpaired samples or match its quality if there are none.

Original: For evaluation, we randomly assign 80% of validation data to a “test-train” and the rest to a “test-test” partition.
Revised: For evaluation, we randomly assign 80% of validation data to a “test-train” and the rest to a “test-test” partition.

Original: We train an LSGAN discriminator (Mao et al., 2017) with the architecture shown in Table(but half the ﬁlters in each layer) on the test-train partition for 40 epochs to distinguish real from generated samples, before measuring its loss on the test set.
Revised: We train an LSGAN discriminator (Mao et al., 2017) with the architecture shown in Table(but half the ﬁlters in each layer) on the test-train partition for 40 epochs to distinguish real from generated samples, before measuring its loss on the test set.

Original: We continuously sample from the generator during training and testing instead of using a ﬁxed set of samples to better approximate the true generator distribution.
Revised: We continuously sample from the generator during training and testing instead of using a ﬁxed set of samples to better approximate the true generator distribution.

Original: As evaluation metric, we use the average test loss over 10 training runs, which was shown to correlate with subjective ratings of visual quality (Im et al., 2018) and also with our own quality judgements throughout this study.
Revised: As evaluation metric, we use the average test loss over 10 training runs, which was shown to correlate with subjective ratings of visual quality (Im et al., 2018) and also with our own quality judgements throughout this study.

Original: A larger value indicates better performance, as we use a ﬂipped sign compared to Im et al.
Revised: A larger value indicates better performance, as we use a ﬂipped sign compared to Im et al.

Original: While the quantitative results appear indicative of output quality, accurate GAN evaluation is still an open problem and so we encourage the reader to judge generated examples given in Section A.5.[Figure]
Revised: While the quantitative results appear indicative of output quality, accurate GAN evaluation is still an open problem and so we encourage the reader to judge generated examples given in Section A.5.

Original: [Figure] Results Our FactorGAN achieves better or similar output quality compared to the GAN baseline in all cases, as seen in Figure 2.
Revised: Results Our FactorGAN achieves better or similar output quality compared to the GAN baseline in all cases, as seen in Figure 2.

Original: For the Edges2Shoes dataset, the performance gains are most pronounced for small numbers of paired samples.
Revised: For the Edges2Shoes dataset, the performance gains are most pronounced for small numbers of paired samples.

Original: On the more complex Cityscapes dataset, FactorGAN outperforms GAN by a large margin independent of training set size, even when the discriminators  are closely matched in size.
Revised: On the more complex Cityscapes dataset, FactorGAN outperforms GAN by a large margin independent of training set size, even when the discriminators [Figure] [Figure] are closely matched in size.

Original: This suggests that FactorGAN converges with fewer training iterations for G φ , although the exact cause is unclear and should be investigated in future work.
Revised: This suggests that FactorGAN converges with fewer training iterations for G φ , although the exact cause is unclear and should be investigated in future work.

Original: We show some generated examples in Figure 3.
Revised: We show some generated examples in Figure 3.

Original: Due to the small number of available paired samples, we observe a strong mode collapse of the GAN in Figure 3a, while FactorGAN provides high-ﬁdelity, diverse outputs, as shown in Figure 3b.
Revised: Due to the small number of available paired samples, we observe a strong mode collapse of the GAN in Figure 3a, while FactorGAN provides high-ﬁdelity, diverse outputs, as shown in Figure 3b.

Original: Similar observations can be made for the Cityscapes dataset when using 100 paired samples (see Section A.5.2).
Revised: Similar observations can be made for the Cityscapes dataset when using 100 paired samples (see Section A.5.2).

Original: 4. I MAGE SEGMENTATION
Revised: 4. I MAGE SEGMENTATION

Original: Our approach extends to the case of conditional generation (see Section 2.3), so we tackle a complex and important image segmentation task on the Cityscapes dataset, where we ask the generator to predict a segmentation map for a city scene (instead of generating both from scratch as in Section 4.2).
Revised: Our approach extends to the case of conditional generation (see Section 2.3), so we tackle a complex and important image segmentation task on the Cityscapes dataset, where we ask the generator to predict a segmentation map for a city scene (instead of generating both from scratch as in Section 4.2).

Original: Experimental setup We downsample the scenes and segmentation maps to 128 × 128 pixels and use a U-Net architecture (Ronneberger et al., 2015) (shown in Table 6 with W = 7 and C = 3 ) as segmentation model.
Revised: Experimental setup We downsample the scenes and segmentation maps to 128 × 128 pixels and use a U-Net architecture (Ronneberger et al., 2015) (shown in Table 6 with W = 7 and C = 3 ) as segmentation model.

Original: For FactorGAN, we use one marginal discriminator to match the distribution of real and fake segmentation maps to ensure realistic predictions, which enables training with isolated city scenes and segmentation maps.
Revised: For FactorGAN, we use one marginal discriminator to match the distribution of real and fake segmentation maps to ensure realistic predictions, which enables training with isolated city scenes and segmentation maps.

Original: To ensure the correct predictions for each city scene, a p- and a q-dependency discriminator learns the input-output relationship using joint samples, both employing a convolutional architecture shown in Table 5.
Revised: To ensure the correct predictions for each city scene, a p- and a q-dependency discriminator learns the input-output relationship using joint samples, both employing a convolutional architecture shown in Table 5.

Original: Note that as in Section 4.2, we output segmentation maps in thecontinuous RGB space instead of performing classiﬁcation.
Revised: Note that as in Section 4.2, we output segmentation maps in the RGB space instead of performing classiﬁcation.

Original: In addition to the MSE in the RGB space, we compute the widely used pixel-wise classiﬁcation accuracy (Cordts et al., 2016) by assigning each output pixel to the class whose colour has the lowest Euclidean distance in RGB space.
Revised: In addition to the MSE in the RGB space, we compute the widely used pixel-wise classiﬁcation accuracy (Cordts et al., 2016) by assigning each output pixel to the class whose colour has the lowest Euclidean distance in RGB space.

Revised: Using the same experimental setup (including network architectures), we also implement the CycleGAN (Zhu et al., 2017) as an unsupervised baseline.

Revised: For the CycleGAN objective, the same GAN losses as shown in (1) and (2) are used 5 .

Original: Results
Revised: Results

Original: The results in Figure 4 demonstrate that our approach can exploit additional unpaired samples to deliver better MSE and accuracy than a GAN and less noisy outputs as seen in Figure 5.
Revised: The results in Figure 4 demonstrate that our approach can exploit additional unpaired samples to deliver better MSE and accuracy than a GAN and less noisy outputs as seen in Figure 5.

Original: While the CycleGAN reaches 58% accuracy (Zhu et al., 2017) treating all samples as unpaired, FactorGAN offers an increase to 71 . 
Revised: When using only 25 paired samples, FactorGAN reaches 71 . 6% accuracy, outperforming both GAN and CycleGAN by an absolute 17 . 7% and 14 .

Original: 6% accuracy when only 25 samples are paired.
Revised: 9% , respectively. Interestingly, CycleGAN performs better than GAN only in this setting, and increasingly falls behind both GAN and FactorGAN with a growing number of paired samples.

Original: Although differences in experimental setup might also affect this difference, it is probably under-estimated since we use a comparatively simple network architecture and training procedure.
Revised: These ﬁndings suggest that FactorGAN can efﬁciently learn the dependency structure from few paired samples with more accuracy than a CycleGAN that is limited by its simplistic cycle consistency assumption. [Figure]

Original: [Figure]
Revised: [Figure]

Original: A UDIO SOURCE SEPARATION
Revised: A UDIO SOURCE SEPARATION

Original: We apply our method to audio source separation as another conditional generation task to investigate whether it transfers acrossapplication domains.
Revised: We apply our method to audio source separation as another conditional generation task to investigate whether it transfers across domains.

Original: Speciﬁcally, we separate music signals into singing voice and accompaniment, as detailed in Section A.2.
Revised: Speciﬁcally, we separate music signals into singing voice and accompaniment, as detailed in Section A.2.

Original: Similarly to image segmentation in Section 4.3, we ﬁnd that FactorGAN outperforms the normal GAN regarding separation quality, suggesting that our factorisation isindeed useful across problem domains.
Revised: As in Section 4.3, we ﬁnd that FactorGAN provides better separtion than GAN, suggesting that our factorisation is useful across problem domains.

Original:  D ISCUSSION
Revised:  D ISCUSSION

Original: We ﬁnd that FactorGAN outperforms GAN across all experiments when additional incomplete samples are available, especially when they are abundant in comparison to the number of joint samples.
Revised: We ﬁnd that FactorGAN outperforms GAN across all experiments when additional incomplete samples are available, especially when they are abundant in comparison to the number of joint samples.

Original: When using only joint observations, FactorGAN should be expected to match the GAN in quality, and it does so quite closely in most of our experiments.
Revised: When using only joint observations, FactorGAN should be expected to match the GAN in quality, and it does so quite closely in most of our experiments.

Original: Surprisingly, it outperforms
Revised: Surprisingly, it outperforms

Original: GAN in some scenarios such as image segmentation even with matched discriminator sizes – a phenomenon we do not fully understand yet and should be investigated in the future.
Revised: GAN in some scenarios such as image segmentation even with matched discriminator sizes – a phenomenon we do not fully understand yet and should be investigated in the future.

Original: For image segmentation, FactorGAN substantially improves segmentation accuracy compared to the fully unsupervised CycleGAN model even when only using 25 paired examples (and simpler network architectures than in Zhu et al. , indicating that it can efﬁciently exploit the pairing information.
Revised: For image segmentation, FactorGAN substantially improves segmentation accuracy compared to the fully unsupervised CycleGAN model even when only using 25 paired examples, indicating that it can efﬁciently exploit the pairing information.

Original: Since the p-dependency discriminator does not rely on generator samples that change during training, it could be pre-trained to reduce computation time, but this led to sudden training instabilities in our experiments.
Revised: Since the p-dependency discriminator does not rely on generator samples that change during training, it could be pre-trained to reduce computation time, but this led to sudden training instabilities in our experiments.

Original: We suspect that this is due to a mismatch between training and testing conditions for the p-dependency discriminator since it is trained on real but evaluated on fake data, and neural networks can yield overly conﬁdent predictions outside the support of the training set (Gal & Ghahramani, 2016).
Revised: We suspect that this is due to a mismatch between training and testing conditions for the p-dependency discriminator since it is trained on real but evaluated on fake data, and neural networks can yield overly conﬁdent predictions outside the support of the training set (Gal & Ghahramani, 2016).

Original: Therefore, we expect classiﬁers with better uncertainty calibration to alleviate this issue.
Revised: Therefore, we expect classiﬁers with better uncertainty calibration to alleviate this issue.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: In this paper, we demonstrated how a joint distribution can be factorised into a set of marginals and dependencies, giving rise to the FactorGAN – a GAN in which the discriminator is split into parts that can be independently trained with incomplete observations.
Revised: In this paper, we demonstrated how a joint distribution can be factorised into a set of marginals and dependencies, giving rise to the FactorGAN – a GAN in which the discriminator is split into parts that can be independently trained with incomplete observations.

Original: For both generation and conditional prediction tasks in multiple domains, we ﬁnd that FactorGAN outperforms the standard GAN when additional incomplete observations are available.
Revised: For both generation and conditional prediction tasks in multiple domains, we ﬁnd that FactorGAN outperforms the standard GAN when additional incomplete observations are available.

Original: For Cityscapes scene segmentation in particular,
Revised: For Cityscapes scene segmentation in particular,

Original: FactorGAN achieves a much higher accuracy than the fully unsupervised CycleGAN, while requiring only 25 of all examples to be annotated.
Revised: FactorGAN achieves a much higher accuracy than the supervised GAN as well as the unsupervised CycleGAN, while requiring only 25 of all examples to be annotated.

Original: Factorising discriminators enables incorporating more prior knowledge into the design of neural architectures in GANs, which could improve empirical results in applied domains.
Revised: Factorising discriminators enables incorporating more prior knowledge into the design of neural architectures in GANs, which could improve empirical results in applied domains.

Original: The presented factorisation is generally applicable independent of model choice, so it can be readily integrated into many existing GAN-based approaches.
Revised: The presented factorisation is generally applicable independent of model choice, so it can be readily integrated into many existing GAN-based approaches.

Original: Since the joint density can be factorised in different ways, multiple extensions are conceivable depending on the particular application (as shown in Section A.3).
Revised: Since the joint density can be factorised in different ways, multiple extensions are conceivable depending on the particular application (as shown in Section A.3).

Original: This paper derives FactorGAN from the original GAN proposed by Goodfellow et al.
Revised: This paper derives FactorGAN from the original GAN proposed by Goodfellow et al.

Original: (2014) by exploiting the probabilistic view of the optimal discriminator.
Revised: (2014) by exploiting the probabilistic view of the optimal discriminator.

Original: Adapting the FactorGAN to alternative
Revised: Adapting the FactorGAN to alternative

Original: GAN objectives (such as the Wasserstein GAN (Arjovsky et al., 2017)) might be possible as well.
Revised: GAN objectives (such as the Wasserstein GAN (Arjovsky et al., 2017)) might be possible as well.

Revised: Instead of relying on additional techniques such as spectral normalisation to ensure training stability, which our theory does not explicitly incorporate, this would enable the use of an inherently more stable GAN variant with the same theoretical guarantees.

Original: A. A UDIO 
Revised: A. A UDIO SOURCE SEPARATION EXPERIMENT

Original: [Figure]

Original: For our audio source separation experiment, our generator G φ takes a music spectrogram m along with noise z and maps it to an estimate of the accompaniment and vocal spectra a and v , implicitly deﬁning an output probability q φ ( a , v | m ) .
Revised: For our audio source separation experiment, our generator G φ takes a music spectrogram m along with noise z and maps it to an estimate of the accompaniment and vocal spectra a and v , implicitly deﬁning an output probability q φ ( a , v | m ) .

Original: We deﬁne the joint real and generated distributions that should be matched as p ( m , a , v ) and q ( m , a , v ) =
Revised: We deﬁne the joint real and generated distributions that should be matched as p ( m , a , v ) and q ( m , a , v ) =

Original: q φ ( a , v | m ) p ( m ) .
Revised: q φ ( a , v | m ) p ( m ) .

Original: Since the source signals in our dataset are simply added in the time-domain to produce the mixture, this approximately applies to the spectrogram as well, so we assume that p ( m | a , v ) =
Revised: Since the source signals in our dataset are simply added in the time-domain to produce the mixture, this approximately applies to the spectrogram as well, so we assume that p ( m | a , v ) =

Original: δ ( m − a − v ) .
Revised: δ ( m − a − v ) .

Original: We can constrain our generator
Revised: We can constrain our generator

Original: G φ to make predictions that always satisfy this condition, thereby taking care of the input-output relationship manually, similarly to Sønderby et al.
Revised: G φ to make predictions that always satisfy this condition, thereby taking care of the input-output relationship manually, similarly to Sønderby et al.

Original: Instead of predicting the sources directly, a mask b with values in the range [0 , 1] is computed, and the accompaniment and vocals are estimated as b (cid:12) m and ( b − 1) (cid:12) m , respectively.
Revised: Instead of predicting the sources directly, a mask b with values in the range [0 , 1] is computed, and the accompaniment and vocals are estimated as b (cid:12) m and ( b − 1) (cid:12) m , respectively.

Original: As a result, q ( m | a , v ) = p ( m | a , v ) , so we can simplify the joint density ratio to
Revised: As a result, q ( m | a , v ) = p ( m | a , v ) , so we can simplify the joint density ratio to

Original: [Equation]
Revised: [Equation]

Original: meaning that the discriminator(s) in the GAN and the FactorGAN only require ( a , v ) pairs, but not the mixture m as additional input, as the correct input-output relationship is already incorporated into the generator.
Revised: meaning that the discriminator(s) in the GAN and the FactorGAN only require ( a , v ) pairs, but not the mixture m as additional input, as the correct input-output relationship is already incorporated into the generator.

Original: Furthermore, the last equality suggests a FactorGAN application with one marginal discriminator for each source along with dependency discriminators to model source dependencies.
Revised: Furthermore, the last equality suggests a FactorGAN application with one marginal discriminator for each source along with dependency discriminators to model source dependencies.

Original: Dataset We use MUSDB (Raﬁi et al., 2017) as multi-track dataset for our experiment, featuring 100 songs for training and 50 songs for testing.
Revised: Dataset We use MUSDB (Raﬁi et al., 2017) as multi-track dataset for our experiment, featuring 100 songs for training and 50 songs for testing.

Original: Each song is downsampled to 22 .
Revised: Each song is downsampled to 22 .

Original: 05 KHz before spectrogram magnitudes are computed, using an STFT with a 512 -sample window and a 256 -sample hop 5 .
Revised: 05 KHz before spectrogram magnitudes are computed, using an STFT with a 512 -sample window and a 256 -sample hop 6 .

Original: Snippets with 128 timeframes each are created by cropping each song’s full spectrogram at regular intervals of 64 timeframes.
Revised: Snippets with 128 timeframes each are created by cropping each song’s full spectrogram at regular intervals of 64 timeframes.

Original: Thus, the generator only separates snippets m ∈ R 256 × 128 ≥and outputs predictions of the same shape, however this does not change the derivation presented in
Revised: Thus, the generator only separates snippets m ∈ R [Table] and outputs predictions of the same shape, however this does not change the derivation presented in

Original: Equation (6), and longer inputs at test time can be processed by partitioning them into snippets and concatenating the model predictions.
Revised: Equation (6), and longer inputs at test time can be processed by partitioning them into snippets and concatenating the model predictions.

Original: Experimental setup For our generator, we use the U-Net architecture detailed in Table 6 with W = 8 and C = 1 .
Revised: Experimental setup For our generator, we use the U-Net architecture detailed in Table 6 with W = 8 and C = 1 .

Original: We use the convolutional discriminator described in Table 5 with W = 8 , H =and C = 1 .
Revised: We use the convolutional discriminator described in Table 5 with W = 8 , H =and C = 1 .

Original: The source dependency discriminators take two sources as input via concatenation along the channel dimension, so they use C = 2 .
Revised: The source dependency discriminators take two sources as input via concatenation along the channel dimension, so they use C = 2 .

Original: In each experiment, we vary the number of training songs whose snippets are available for paired training between 10 , 20 and 50 and compare between GAN and FactorGAN.
Revised: In each experiment, we vary the number of training songs whose snippets are available for paired training between 10 , 20 and 50 and compare between GAN and FactorGAN.

Original: The spectrograms predicted on the test set are converted to audio with the inverse STFT by reusing the phase from the mixture, and then evaluated using the signal-to-distortion ratio (SDR), a well-established evaluation metric for source separation (Vincent et al., 2006).
Revised: The spectrograms predicted on the test set are converted to audio with the inverse STFT by reusing the phase from the mixture, and then evaluated using the signal-to-distortion ratio (SDR), a well-established evaluation metric for source separation (Vincent et al., 2006).

Original: Results Figure 6 shows our separation results.
Revised: Results Figure 6 shows our separation results.

Original: Compared to a GAN, the separation performance is signiﬁcantly higher using FactorGAN.
Revised: Compared to a GAN, the separation performance is signiﬁcantly higher using FactorGAN.

Original: As expected, FactorGAN improves slightly with more paired examples, which is not the case for the GAN – here we ﬁnd that the vocal output becomes too quiet when increasing the number of songs for training, possibly a sign of mode collapse.
Revised: As expected, FactorGAN improves slightly with more paired examples, which is not the case for the GAN – here we ﬁnd that the vocal output becomes too quiet when increasing the number of songs for training, possibly a sign of mode collapse.

Original: Similarly to the results seen in the image pair generation experiments, we suspect that the FactorGAN discriminator might approximate the joint density
Revised: Similarly to the results seen in the image pair generation experiments, we suspect that the FactorGAN discriminator might approximate the joint density

Original: ˜ D ( x ) more closely than the GAN discriminator due to its use of multiple discriminators, although the reasons for this are not yet understood.
Revised: ˜ D ( x ) more closely than the GAN discriminator due to its use of multiple discriminators, although the reasons for this are not yet understood.

Original: A. P OSSIBLE EXTENSIONS
Revised: A. P OSSIBLE EXTENSIONS

Original: We can decompose the joint density ratio p x ( x ) q x ( x ) in other ways than shown in Equation 3 in the paper.
Revised: We can decompose the joint density ratio p x ( x ) q x ( x ) in other ways than shown in Equation 3 in the paper.

Original: In the following, we discuss two additional possibilities.
Revised: In the following, we discuss two additional possibilities.

Original: A.3. H IERARCHICAL F ACTOR GAN
Revised: A.3. H IERARCHICAL F ACTOR GAN

Original: The decomposition of the joint density ratio could be applied recursively, splitting the obtained marginals further into “sub-marginals” and their dependencies, which could be repeated multiple times.
Revised: The decomposition of the joint density ratio could be applied recursively, splitting the obtained marginals further into “sub-marginals” and their dependencies, which could be repeated multiple times.

Original: In addition to training with incomplete observations where only a single part is given, this also allows making use of samples where only sub-parts of these parts are given and is thus more ﬂexible than a single factorisation as used in the standard FactorGAN.
Revised: In addition to training with incomplete observations where only a single part is given, this also allows making use of samples where only sub-parts of these parts are given and is thus more ﬂexible than a single factorisation as used in the standard FactorGAN.

Original: As a demonstration, we split each marginal x i further into a group of J i marginals, J i ≤ |D i | , and their dependencies, without further recursion for simplicity:
Revised: As a demonstration, we split each marginal x i further into a group of J i marginals, J i ≤ |D i | , and their dependencies, without further recursion for simplicity:

Original: [Equation]
Revised: [Equation]

Original: c iP and c iQ are dependency terms analogously to c P and c Q , but only deﬁned on marginal variable x i , whose J “sub-marginals” are denoted by x i, 1 , . . .
Revised: c iP and c iQ are dependency terms analogously to c P and c Q , but only deﬁned on marginal variable x i , whose J “sub-marginals” are denoted by x i, 1 , . . .

Original: , x i,J .
Revised: , x i,J .

Original: Such a hierarchical decomposition might also be beneﬁcial if the data is known to be generated from a hierarchical process.
Revised: Such a hierarchical decomposition might also be beneﬁcial if the data is known to be generated from a hierarchical process.

Original: We leave the empirical exploration of this concept to future work.
Revised: We leave the empirical exploration of this concept to future work.

Original: A.3. A UTOREGRESSIVE F ACTOR GAN
Revised: A.3. A UTOREGRESSIVE F ACTOR GAN

Original: For a multi-dimensional variable x = [ x 1 , x 2 , . . .
Revised: For a multi-dimensional variable x = [ x 1 , x 2 , . . .

Original: , x T ] composed of T elements arranged in a sequence, such as time series data, the joint density ratio can also be decomposed in a causal, auto-regressive fashion:
Revised: , x T ] composed of T elements arranged in a sequence, such as time series data, the joint density ratio can also be decomposed in a causal, auto-regressive fashion:

Original: [Equation]
Revised: [Equation]

Original: Note that c P is deﬁned here as
Revised: Note that c P is deﬁned here as

Original: [Equation]
Revised: [Equation]

Revised: [Equation]

Original: Q analogously using q x ).
Revised: Q analogously using q x ).

Original: Equation (8) suggests an auto-regressive version of FactorGAN in which the generator output quality at each time-step i is evaluated using a marginal discriminator that estimates p x ( x i ) q x ( x i ) combined with dependency discriminators that model the dependency between the current and all past time-steps.
Revised: Equation (8) suggests an auto-regressive version of FactorGAN in which the generator output quality at each time-step i is evaluated using a marginal discriminator that estimates p x ( x i ) q x ( x i ) combined with dependency discriminators that model the dependency between the current and all past time-steps.

Original: The ﬁnal product formulation in Equation (9) reveals a close similarity to auto-regressive models and suggests a modiﬁcation of the normal GAN with an auto-regressive discriminator that rates an input at each time-step given the previous ones.
Revised: The ﬁnal product formulation in Equation (9) reveals a close similarity to auto-regressive models and suggests a modiﬁcation of the normal GAN with an auto-regressive discriminator that rates an input at each time-step given the previous ones.

Original: Using a derivation analogous to the one shown in
Revised: Using a derivation analogous to the one shown in

Original: Section A.4, this implies taking the unnormalised discriminator outputs at each time-step, summing them, and applying a sigmoid non-linearity to obtain the overall estimate of the probability ˜ D ( x ) .
Revised: Section A.4, this implies taking the unnormalised discriminator outputs at each time-step, summing them, and applying a sigmoid non-linearity to obtain the overall estimate of the probability ˜ D ( x ) .

Original: A similar implementation was used before in Mogren (2016), attempting to stabilise GAN training with recurrent neural networks as discriminators, but for the ﬁrst time, we provide a rigorous theoretical justiﬁcation for this practice here.
Revised: A similar implementation was used before in Mogren (2016), attempting to stabilise GAN training with recurrent neural networks as discriminators, but for the ﬁrst time, we provide a rigorous theoretical justiﬁcation for this practice here.

Original: A. D ISCRIMINATOR COMBINATION
Revised: A. D ISCRIMINATOR COMBINATION

Original: Deﬁnition A.1.
Revised: Deﬁnition A.1.

Original: Sigmoid discriminator output.
Revised: Sigmoid discriminator output.

Original: Let D
Revised: Let D

Original: θ i ( x i ) :
Revised: θ i ( x i ) :

Original: = σ ( d θ i ( x i )) , d θ i : R |D i | → R for all i ∈ { 1 , . . .
Revised: = σ ( d θ i ( x i )) , d θ i : R |D i | → R for all i ∈ { 1 , . . .

Original: , K }
Revised: , K }

Original: , analogously deﬁne D Pθ P ( x ) and D Qθ Q ( x ) .
Revised: , analogously deﬁne D Pθ P ( x ) and D Qθ Q ( x ) .

Original: Deﬁnition A.2.
Revised: Deﬁnition A.2.

Original: Combined discriminator.
Revised: Combined discriminator.

Original: Let D C ( x ) :
Revised: Let D C ( x ) :

Original: = σ ( d Pθ P ( x ) −
Revised: = σ ( d Pθ P ( x ) −

Original: d Qθ Q ( x ) + (cid:80) Ki
Revised: d Qθ Q ( x ) + (cid:80) Ki

Original: =

Revised: =

Original: d θ i ( x i )) be the output of the combined discriminator that is used for training G φ using Equation 2.
Revised: d θ i ( x i )) be the output of the combined discriminator that is used for training G φ using Equation 2.

Original: Theorem 1.
Revised: Theorem 1.

Original: Combined discriminator approximates ˜ D ( x ) .
Revised: Combined discriminator approximates ˜ D ( x ) .

Original: Under deﬁnitions A.1 and A.2 and assuming optimally trained sub-discriminators, D C ( x ) =
Revised: Under deﬁnitions A.1 and A.2 and assuming optimally trained sub-discriminators, D C ( x ) =

Original: ˜ D ( x ) =
Revised: ˜ D ( x ) =

Original: p x [Equation] x
Revised: p x ( x

Original: [Equation]
Revised: [Equation]

Original: p x ( x )+ q x ( x ) .
Revised: p x ( x )+ q x ( x ) .

Original: Proof.
Revised: Proof.

Original: Proof of Theorem 1 using Deﬁnitions A.1 and A.2:
Revised: Proof of Theorem 1 using Deﬁnitions A.1 and A.2:

Original: [Equation]
Revised: [Equation]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]