Original: MICRO GAN: P ROMOTING V ARIETY THROUGH M ICROBATCH D ISCRIMINATION
Revised: MICRO GAN: P ROMOTING V ARIETY THROUGH M ICROBATCH D ISCRIMINATION

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: We propose to tackle the mode collapse problem in generative adversarial networks (GANs) by using multiple discriminators and assigning a different portion of each minibatch, called microbatch, to each discriminator.
Revised: We propose to tackle the mode collapse problem in generative adversarial networks (GANs) by using multiple discriminators and assigning a different portion of each minibatch, called microbatch, to each discriminator.

Original: We gradually change each discriminator’s task from distinguishing between real and fake samples to discriminating samples coming from inside or outside its assigned microbatch by using a diversity parameter α .
Revised: We gradually change each discriminator’s task from distinguishing between real and fake samples to discriminating samples coming from inside or outside its assigned microbatch by using a diversity parameter α .

Original: The generator is then forced to promote variety in each minibatch to make the microbatch discrimination harder to achieve by each discriminator.
Revised: The generator is then forced to promote variety in each minibatch to make the microbatch discrimination harder to achieve by each discriminator.

Original: Thus, all models in our framework beneﬁt from having variety in the generated set to reduce their respective losses.
Revised: Thus, all models in our framework beneﬁt from having variety in the generated set to reduce their respective losses.

Original: We show evidence that our solution promotes sample diversity since early training stages on multiple datasets.
Revised: We show evidence that our solution promotes sample diversity since early training stages on multiple datasets.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Generative adversarial networks (Goodfellow et al.
Revised: Generative adversarial networks (Goodfellow et al.

Original: (2014)), or GANs, consist of a framework describing the interaction between two different models - one generator (G) and one discriminator (D) - that are trained together.
Revised: (2014)), or GANs, consist of a framework describing the interaction between two different models - one generator (G) and one discriminator (D) - that are trained together.

Original: While G tries to learn the real data distribution by generating realistic looking samples that are able to fool D , D tries to do a better job at distinguishing between real and the fake samples produced by G .
Revised: While G tries to learn the real data distribution by generating realistic looking samples that are able to fool D , D tries to do a better job at distinguishing between real and the fake samples produced by G .

Original: Although showing very promising results across various domains (Edwards and Storkey (2015);
Revised: Although showing very promising results across various domains (Edwards and Storkey (2015);

Original: Ho and Ermon (2016); Yu et al.
Revised: Ho and Ermon (2016); Yu et al.

Original: Kim et al.
Revised: Kim et al.

Original: (2017a);
Revised: (2017a);

Original: Yang et al.
Revised: Yang et al.

Original: (2017); Donahue et al.
Revised: (2017); Donahue et al.

Original: , GANs have also been continually associated with instability in training, more speciﬁcally mode collapse (Kim et al. (2017b); Arjovsky and Bottou (2017);
Revised: , GANs have also been continually associated with instability in training, more speciﬁcally mode collapse (Kim et al. (2017b); Arjovsky and Bottou (2017);

Original: Mescheder et al.
Revised: Mescheder et al.

Original: ; Che et al. (2016); Arjovsky et al.
Revised: ; Che et al. (2016); Arjovsky et al.

Original: This behavior is observed when
Revised: This behavior is observed when

Original: G is able to fool D by only generating samples from the same data mode, leading to very similar looking generated samples.
Revised: G is able to fool D by only generating samples from the same data mode, leading to very similar looking generated samples.

Original: This suggests that G did not succeed in learning the full data distribution but, instead, only a small part of it.
Revised: This suggests that G did not succeed in learning the full data distribution but, instead, only a small part of it.

Original: This is the main problem we are trying to solve with this work.
Revised: This is the main problem we are trying to solve with this work.

Original: The proposed solution is to use multiple discriminators and assign each D a different portion of the real and fake minibatches, i.e., microbatch.
Revised: The proposed solution is to use multiple discriminators and assign each D a different portion of the real and fake minibatches, i.e., microbatch.

Original: Then, we update each D ’s task to discriminate between samples coming from its assigned fake microbatch and samples from the microbatches assigned to the other discriminators, together with the real samples.
Revised: Then, we update each D ’s task to discriminate between samples coming from its assigned fake microbatch and samples from the microbatches assigned to the other discriminators, together with the real samples.

Original: We call this microbatch discrimination.
Revised: We call this microbatch discrimination.

Original: Throughout training, we gradually change from the originally proposed real and fake discrimination by Goodfellow et al.
Revised: Throughout training, we gradually change from the originally proposed real and fake discrimination by Goodfellow et al.

Original: (2014) to the introduced microbatch discrimination by the use of an additional diversity parameter α that ultimately controls the diversity in the overall minibatch.
Revised: (2014) to the introduced microbatch discrimination by the use of an additional diversity parameter α that ultimately controls the diversity in the overall minibatch.

Original: The main idea of this work is to force G to reduce its loss by inducing variety in the generated set, complicating each D ’s task on separating the samples in its microbatch from the rest.
Revised: The main idea of this work is to force G to reduce its loss by inducing variety in the generated set, complicating each D ’s task on separating the samples in its microbatch from the rest.

Original: Even though only producing very similar images would also complicate the desired discrimination, it would not beneﬁt any of the models.
Revised: Even though only producing very similar images would also complicate the desired discrimination, it would not beneﬁt any of the models.

Original: This is due to the attribution of distinct probabilities by each D to samples from and outside its microbatch being required to minimize G and D ’s losses.
Revised: This is due to the attribution of distinct probabilities by each D to samples from and outside its microbatch being required to minimize G and D ’s losses.

Original: Hence, all models in the proposed framework, called microGAN, beneﬁt directly from diversity in the generated set.
Revised: Hence, all models in the proposed framework, called microGAN, beneﬁt directly from diversity in the generated set.

Original: Our main contributions can be stated as follows: (i) proposal of a novel multi-adversarial GANs framework 1 (Section 3) that mitigates the inherent mode collapse problem in GANs; (ii) empirical evidence on multiple datasets showing the success of our approach in promoting sample variety since early stages of training (Section 4) (iii) Competitiveness against other previously proposed methods on multiple datasets and evaluation metrics (Section 5).
Revised: Our main contributions can be stated as follows: (i) proposal of a novel multi-adversarial GANs framework 1 (Section 3) that mitigates the inherent mode collapse problem in GANs; (ii) empirical evidence on multiple datasets showing the success of our approach in promoting sample variety since early stages of training (Section 4) (iii) Competitiveness against other previously proposed methods on multiple datasets and evaluation metrics (Section 5).

Original: 1. R ELATED W ORK
Revised: 1. R ELATED W ORK

Original: To further tackle training instability, previous works  optimized GANs training by changing the overall models’ objectives, either by using discrepancy measurements (Li et al.
Revised: Previous works have optimized GANs training by changing the overall models’ objectives, either by using discrepancy measurements (Li et al.

Original: ; Sutherland et al.
Revised: ; Sutherland et al.

Original: (2016)) or different divergence functions (Nowozin et al.
Revised: (2016)) or different divergence functions (Nowozin et al.

Original: ; Uehara et al.
Revised: ; Uehara et al.

Original: to approximate the real data distribution.
Revised: to approximate the real data distribution.

Original: Moreover, Zhao et al. ; Berthelot et al.
Revised: Moreover, Zhao et al. (2016); Berthelot et al.

Original: ; Unterthiner et al.
Revised: ; Unterthiner et al.

Original: (2017) proposed to use energy-driven objective functions to encourage sample variety, Mroueh et al.
Revised: (2017) proposed to use energy-driven objective functions to encourage sample variety, Mroueh et al.

Original: (2017) tried to match the mean and covariance of the real data distribution, and Metz et al.
Revised: (2017) tried to match the mean and covariance of the real data, and Metz et al.

Original: (2016) used an unrolled optimization of D to train G .
Revised: (2016) used an unrolled optimization of D to train G .

Original: Che et al.
Revised: Che et al.

Original: (2016); Warde-Farley and Bengio (2016);
Revised: (2016); Warde-Farley and Bengio (2016);

Original: Wang et al.
Revised: Wang et al.

Original: ; Berthelot et al.
Revised: Berthelot et al.

Original:  penalized missing modes by using an extra autoenconder in the framework.
Revised: (2017) penalized missing modes by using an extra autoenconder in the framework.

Original: Salimans et al.
Revised: Salimans et al.

Original: (2016) performed minibatch discrimination by forcing D to condition its output on the similarity between the samples in the minibatch.
Revised: (2016) performed minibatch discrimination by forcing D to condition its output on the similarity between the samples in the minibatch.

Original: Ghosh et al. ; Tolstikhin et al.
Revised: Springenberg (2015) increased D ’s robustness by maximizing the mutual information between inputs and corresponding labels, while Lin et al.

Original: ; Hoang et al. (2018) proposed to augment the number ofgenerators in the framework.
Revised: (2017) forced D to make decisions over multiple samples of the same class, instead of independently.

Original: Regarding using multiple discriminators, Neyshabur et al.
Revised: Regarding using multiple discriminators, Neyshabur et al.

Original: (2017) extended the framework to several discriminators with each focusing in a low-dimensional projection of the data, set a priori.
Revised: (2017) extended the framework to several discriminators with each focusing in a low-dimensional projection of the data, set a priori.

Original: Durugkar et al.
Revised: Durugkar et al.

Original: (2016) proposed GMAN, consisting of an ensemble of discriminators that could be accessed by the single generator according to different levels of difﬁculty.
Revised: (2016) proposed GMAN, consisting of an ensemble of discriminators that could be accessed by the single generator according to different levels of difﬁculty.

Original: Nguyen et al.
Revised: Nguyen et al.

Original: (2017) introduced D2GAN, introducing a single generator dual discriminator architecture where one discriminator rewards samples coming from the true data distribution whilst the other rewards samples coming from the generator, forcing the generator to continuously change its output.
Revised: (2017) introduced D2GAN, introducing a single generator dual discriminator architecture where one discriminator rewards samples coming from the true data distribution whilst the other rewards samples coming from the generator, forcing the generator to continuously change its output.

Original: Mordido et al.
Revised: Mordido et al.

Original: (2018) applied adversarial dropout by omitting the feedback of a given Dwith a given probability at the end of each batch, leading to a more generalized G .
Revised: (2018) applied adversarial dropout by omitting the feedback of a given D at the end of each batch.

Original:  G ENERATIVE A DVERSARIAL N ETWORKS
Revised:  G ENERATIVE A DVERSARIAL N ETWORKS

Original: The original GANs framework (Goodfellow et al.
Revised: The original GANs framework (Goodfellow et al.

Original: (2014)) consists of two models: a generator ( G ) and a discriminator ( D ).
Revised: (2014)) consists of two models: a generator ( G ) and a discriminator ( D ).

Original: Both models are assigned different tasks: whilst G tries to capture the real data distribution p r , D learns how to distinguish real from fake samples.
Revised: Both models are assigned different tasks: whilst G tries to capture the real data distribution p r , D learns how to distinguish real from fake samples.

Original: G maps a noise vector z , retrieved from a noise distribution p z , to a realistic looking sample belonging to the data space.
Revised: G maps a noise vector z , retrieved from a noise distribution p z , to a realistic looking sample belonging to the data space.

Original: D maps a sample to a probability p , representing the likeliness of that given sample coming from p r rather than from p g .
Revised: D maps a sample to a probability p , representing the likeliness of that given sample coming from p r rather than from p g .

Original: The two models are trained together and play the following minimax game:
Revised: The two models are trained together and play the following minimax game:

Original: [Equation]
Revised: [Equation]

Original: where D maximizes the probability of assigning samples to the correct distribution and G minimizes the probability of its samples being considered from the fake data distribution.
Revised: where D maximizes the probability of assigning samples to the correct distribution and G minimizes the probability of its samples being considered from the fake data distribution.

Original: Alternatively, one can also train G to maximize the probability of its output being considered from the real data distribution, i.e., log D ( G ( z )) .
Revised: Alternatively, one can also train G to maximize the probability of its output being considered from the real data distribution, i.e., log D ( G ( z )) .

Original: Even though this changes the type of the game, by being no longer minimax, it avoids the saturation of the gradient signals at the beginning of training (Goodfellow et al.
Revised: Even though this changes the type of the game, by being no longer minimax, it avoids the saturation of the gradient signals at the beginning of training (Goodfellow et al.

Original: (2014)), where G only receives continuously negative feedback, making training more stable in practice.
Revised: (2014)), where G only receives continuously negative feedback, making training more stable in practice.

Original: However, since we employ multiple discriminators in the proposed framework, it is less likely that G does not receive any positive feedback from the whole adversarial ensemble (Durugkar et al.
Revised: However, since we employ multiple discriminators in the proposed framework, it is less likely that G does not receive any positive feedback from the whole adversarial ensemble (Durugkar et al.

Original: Therefore, we make use of the original value function in this work.
Revised: Therefore, we make use of the original value function in this work.

Original:  MICRO GAN
Revised:  MICRO GAN

Original: In this work, we propose a novel generative multi-adversarial framework named microGAN, where we start by splitting each minibatch into several microbatches and assigning a unique one to each D .
Revised: In this work, we propose a novel generative multi-adversarial framework named microGAN, where we start by splitting each minibatch into several microbatches and assigning a unique one to each D .

Original: The key aspect of this work is the usage of microbatch discrimination, where we change the original discrimination task of distinguishing between real and fake samples, as proposed in Goodfellow et al.
Revised: The key aspect of this work is the usage of microbatch discrimination, where we change the original discrimination task of distinguishing between real and fake samples, as proposed in Goodfellow et al.

Original: (2014), to each D distinguishing between samples coming or not from its fake microbatch.
Revised: (2014), to each D distinguishing between samples coming or not from its fake microbatch.

Original: This change is performed in a gradual fashion, using an additional diversity parameter α .
Revised: This change is performed in a gradual fashion, using an additional diversity parameter α .

Original: Thus, each
Revised: Thus, each

Original: D ’s output gradually changes from the probability of a given sample being real to the probability of a given sample not belonging to its fake microbatch.
Revised: D ’s output gradually changes from the probability of a given sample being real to the probability of a given sample not belonging to its fake microbatch.

Original: Moreover, since each D is trained with different fake and real samples, we encourage them to focus on different data properties.
Revised: Moreover, since each D is trained with different fake and real samples, we encourage them to focus on different data properties.

Original: Figureillustrates the proposed framework.
Revised: Figureillustrates the proposed framework.

Original: The proposed microbatch-level discrimination task leads to G making such discrimination harder for each D to lower its loss.
Revised: The proposed microbatch-level discrimination task leads to G making such discrimination harder for each D to lower its loss.

Original: Hence, G is forced to induce variety on the overall minibatch, making it a substantially harder task for each D to be able to separate its subset of fake samples in the diverse minibatch.
Revised: Hence, G is forced to induce variety on the overall minibatch, making it a substantially harder task for each D to be able to separate its subset of fake samples in the diverse minibatch.

Original: Note that producing very similar samples across the whole minibatch would also make such discrimination difﬁcult by making the whole minibatch the same.
Revised: Note that producing very similar samples across the whole minibatch would also make such discrimination difﬁcult by making the whole minibatch the same.

Original: However, G also beneﬁts from each
Revised: However, G also beneﬁts from each

Original: D assigning distinct probabilities to samples from inside and outside its designed microbatch to lower its loss, making the generation of different samples in the minibatch a necessary requirement to obtain different outputs from D .
Revised: D assigning distinct probabilities to samples from inside and outside its designed microbatch to lower its loss, making the generation of different samples in the minibatch a necessary requirement to obtain different outputs from D .

Original: Hence, all models in our framework beneﬁt directly from sample variety in the generated set.
Revised: Hence, all models in our framework beneﬁt directly from sample variety in the generated set.

Original: In the microGAN scenario with a positive diversity parameter α , each D assigns low probabilities to fake samples from its microbatch and high probabilities to fake samples from the rest of the microbatches as well as samples from the real data distribution.
Revised: In the microGAN scenario with a positive diversity parameter α , each D assigns low probabilities to fake samples from its microbatch and high probabilities to fake samples from the rest of the microbatches as well as samples from the real data distribution.

Original: Hence, fake samples in the rest of the minibatch, i.e., not coming from its assigned microbatch, shall be given distinct output probabilities by each D .
Revised: Hence, fake samples in the rest of the minibatch, i.e., not coming from its assigned microbatch, shall be given distinct output probabilities by each D .

Original: On the other hand, G minimizes the probability given by each D to the samples outside its microbatch and maximizes the probability given to the fake samples assigned to that speciﬁc D .
Revised: On the other hand, G minimizes the probability given by each D to the samples outside its microbatch and maximizes the probability given to the fake samples assigned to that speciﬁc D .

Original: The value function of our minimax game is as follows:
Revised: The value function of our minimax game is as follows:

Original: [Equation]
Revised: [Equation]

Original: where K represents the number of total discriminators in the set.
Revised: where K represents the number of total discriminators in the set.

Original: p r Dk represents real samples from
Revised: p r Dk represents real samples from

Original: D k ’s real microbatch, p z GDk indicates fake samples from D k ’s fake microbatch, and p z GD \{ z GDk } relates to the rest of the fake samples in the minibatch but not in p
Revised: D k ’s real microbatch, p z GDk indicates fake samples from D k ’s fake microbatch, and p z GD \{ z GDk } relates to the rest of the fake samples in the minibatch but not in p

Original: z GDk .
Revised: z GDk .

Original: α represents the diversity parameter responsible for penalizing the incorrect discrimination of fake samples coming from p z GD \{ z GDk } by each D k .
Revised: α represents the diversity parameter responsible for penalizing the incorrect discrimination of fake samples coming from p z GD \{ z GDk } by each D k .

Original: Note that α = 0 would represent the original GANs objective for each D in the set.
Revised: Note that α = 0 would represent the original GANs objective for each D in the set.

Original: Appropriate α values and its effects on the whole minibatch diversity are discussed next.
Revised: Appropriate α values and its effects on the whole minibatch diversity are discussed next.

Original: 3. D IVERSITY P ARAMETER α
Revised: 3. D IVERSITY P ARAMETER α

Original: We control the weight of the microbatch discrimination in the models’ losses by introducing an additional diversity parameter α .
Revised: We control the weight of the microbatch discrimination in the models’ losses by introducing an additional diversity parameter α .

Original: Lower α values lead to G signiﬁcantly lowering its loss by generating realistic looking samples on each microbatch without taking much consideration on the variety of the overall minibatch.
Revised: Lower α values lead to G signiﬁcantly lowering its loss by generating realistic looking samples on each microbatch without taking much consideration on the variety of the overall minibatch.

Original: On the other hand, higher α values induce a stronger effect on G ’s loss if each
Revised: On the other hand, higher α values induce a stronger effect on G ’s loss if each

Original: D is able to discriminate between samples inside and outside its microbatch.
Revised: D is able to discriminate between samples inside and outside its microbatch.

Original: However, high values of α might compromise the realistic properties of the produced samples, since too much weight is given to the last part of Eq. 2, being sufﬁcient to effectively minimize G ’s loss.
Revised: However, high values of α might compromise the realistic properties of the produced samples, since too much weight is given to the last part of Eq. 2, being sufﬁcient to effectively minimize G ’s loss.

Original: Thus, using α >represents an additional way of ensuring data variety within the minibatch produced by G at each iteration.
Revised: Thus, using α >represents an additional way of ensuring data variety within the minibatch produced by G at each iteration.

Original: An overview of different possible α settings follows below.
Revised: An overview of different possible α settings follows below.

Original: 3.1. S TATIC α
Revised: 3.1. S TATIC α

Original: First, we statically set α to values between 0 and 1 throughout the whole training.
Revised: First, we statically set α to values between 0 and 1 throughout the whole training.

Original: For the evaluation of the effects of each α value, we used a toy experiment of a 2D mixture of 8 Gaussian distributions (representing 8 data modes) ﬁrstly presented by Metz et al.
Revised: For the evaluation of the effects of each α value, we used a toy experiment of a 2D mixture of 8 Gaussian distributions (representing 8 data modes) ﬁrstly presented by Metz et al.

Original: (2016), and further adopted by Nguyen et al.
Revised: (2016), and further adopted by Nguyen et al.

Original: We used 8 discriminators for all the experiments.
Revised: We used 8 discriminators for all the experiments.

Original: Results are shown in Figure 2.
Revised: Results are shown in Figure 2.

Original: When setting α = 0 , G mode collapses on a speciﬁc mode, showing the importance of using positive α values to mitigate mode collapse.
Revised: When setting α = 0 , G mode collapses on a speciﬁc mode, showing the importance of using positive α values to mitigate mode collapse.

Original: When setting 0 .
Revised: When setting 0 .

Original: 1 ≤ α ≤ 0 .
Revised: 1 ≤ α ≤ 0 .

Original: 5 , G is able to capture all data modes during training.
Revised: 5 , G is able to capture all data modes during training.

Original: However, learning problems in the early stages are observed, with G only focusing on promoting variety in the generated samples.
Revised: However, learning problems in the early stages are observed, with G only focusing on promoting variety in the generated samples.

Original: For higher α values ( α ≥ 0 . 6 ), G was unable to produce any realistic looking samples throughout the whole training, focusing solely on sample diversity to lower its loss, suggesting the dominance of the last part of Eq.
Revised: For higher α values ( α ≥ 0 . 6 ), G was unable to produce any realistic looking samples throughout the whole training, focusing solely on sample diversity to lower its loss, suggesting the dominance of the last part of Eq.

Original: Hence, a mild, dynamic, manipulation of α values seems to be necessary for a successful training of G , ultimately meaning both realistic and diverse samples from an early training stage.
Revised: Hence, a mild, dynamic, manipulation of α values seems to be necessary for a successful training of G , ultimately meaning both realistic and diverse samples from an early training stage.

Original: 3.1. S ELF - LEARNED α
Revised: 3.1. S ELF - LEARNED α

Original: We dynamically set α over time by adding it as a parameter of G and letting it self-learn its values to lower its loss.
Revised: We dynamically set α over time by adding it as a parameter of G and letting it self-learn its values to lower its loss.

Original: However, we observed that G takes advantage of being able to reduce its loss by increasing α at a large rate, focusing simply on promoting diversity in the generated samples without much realism, similarly to what was observed when using α = 0 .
Revised: However, we observed that G takes advantage of being able to reduce its loss by increasing α at a large rate, focusing simply on promoting diversity in the generated samples without much realism, similarly to what was observed when using α = 0 .

Original: 6 in the toy experiment (Figure 2).
Revised: 6 in the toy experiment (Figure 2).

Original: Hence, we suggest several properties that α should have so that diversity does not compromise the veracity of the generated samples.
Revised: Hence, we suggest several properties that α should have so that diversity does not compromise the veracity of the generated samples.

Original: First, α should be upper bounded so that the last part of Eq. 2 (responsible for sample diversity) does not overpower the ﬁrst part (responsible for sample realism), ultimately not compromising the feedback given to G to also be able to generate realistic samples.
Revised: First, α should be upper bounded so that the last part of Eq. 2 (responsible for sample diversity) does not overpower the ﬁrst part (responsible for sample realism), ultimately not compromising the feedback given to G to also be able to generate realistic samples.

Original: Second, α ’s growth should saturate over time, meaning that continuously increasing at large rates α is no longer an option to substantially decrease G ’s loss over time.
Revised: Second, α ’s growth should saturate over time, meaning that continuously increasing at large rates α is no longer an option to substantially decrease G ’s loss over time.

Original: Lastly, to tackle the problem in learning of early to mid stages, we suggest that α should grow in a controlled fashion, so focus can also be given in the realistic aspect of the samples since the beginning of training.
Revised: Lastly, to tackle the problem in learning of early to mid stages, we suggest that α should grow in a controlled fashion, so focus can also be given in the realistic aspect of the samples since the beginning of training.

Original: Thus, we propose to make α a function of β , where α ( β ) ∈ [0 , 1[ , and let G regulate β instead of directly learning α .
Revised: Thus, we propose to make α a function of β , where α ( β ) ∈ [0 , 1[ , and let G regulate β instead of directly learning α .

Original: We evaluated regulating α over three different functions that have the desired properties:
Revised: We evaluated regulating α over three different functions that have the desired properties:

Original: [Equation]
Revised: [Equation]

Original: with β sigm , β soft , and β tanh representing the initial values of β when training begins for the respective functions.
Revised: with β sigm , β soft , and β tanh representing the initial values of β when training begins for the respective functions.

Original: For all the experiments of this paper, we set β tanh =
Revised: For all the experiments of this paper, we set β tanh =

Original: β soft = 0 , to obtain a positive codomain, and β sigm = − 1 .
Revised: β soft = 0 , to obtain a positive codomain, and β sigm = − 1 .

Original: 8 , since we achieved better empirical results by starting β with this value (for further discussion about the effects of using different β sigm on α sigm ( β ) ’s growth please see the Appendix).
Revised: 8 , since we achieved better empirical results by starting β with this value (for further discussion about the effects of using different β sigm on α sigm ( β ) ’s growth please see the Appendix).

Original: Note that learning α without any constraints can be characterized as using the identity function ( α ( β )
Revised: Note that learning α without any constraints can be characterized as using the identity function ( α ( β )

Original: =
Revised: =

Original: α ident ( β ) = β ).
Revised: α ident ( β ) = β ).

Original: Thus, each used function promotes a different α growth over time.
Revised: Thus, each used function promotes a different α growth over time.

Original: To ease presentation, we neglect to write β ’s dependence for the rest of the manuscript and use only the function names to described each α setting: α sigm , α soft , α tanh , and α ident .
Revised: To ease presentation, we neglect to write β ’s dependence for the rest of the manuscript and use only the function names to described each α setting: α sigm , α soft , α tanh , and α ident .

Original: [Equation]
Revised: [Equation]

Original: [Figure]
Revised: [Figure]

Original: the toy dataset.
Revised: the toy dataset.

Original: The generated samples are shown in (a).
Revised: The generated samples are shown in (a).

Original: The evolution of α on each function is presented in (b).
Revised: The evolution of α on each function is presented in (b).

Original: Results on the toy dataset using the different proposed α functions are shown in Figure 3.
Revised: Results on the toy dataset using the different proposed α functions are shown in Figure 3.

Original: The beneﬁts of increasing α in a milder fashion, as performed when using α sigm , are observed especially early on training, with G being concerned with the realism of the generated samples.
Revised: The beneﬁts of increasing α in a milder fashion, as performed when using α sigm , are observed especially early on training, with G being concerned with the realism of the generated samples.

Original: On the other hand, when using α tanh and α soft , the network takes longer to focus on the data realism (10K steps) since it is able to reduce its loss signiﬁcantly by simply promoting variety due to the steeper growth of α in the earlier stages on both functions.
Revised: On the other hand, when using α tanh and α soft , the network takes longer to focus on the data realism (10K steps) since it is able to reduce its loss signiﬁcantly by simply promoting variety due to the steeper growth of α in the earlier stages on both functions.

Original: Nevertheless, as the functions gradually saturate, all α settings manage to eventually capture the real data distribution while still keeping the diversity in the generated samples.
Revised: Nevertheless, as the functions gradually saturate, all α settings manage to eventually capture the real data distribution while still keeping the diversity in the generated samples.

Original: In conclusion, one can summarize microGAN’s training using these variations of selflearned α as the following: in the ﬁrst iterations, G increases α to reduce its loss, expanding its output.
Revised: In conclusion, one can summarize microGAN’s training using these variations of selflearned α as the following: in the ﬁrst iterations, G increases α to reduce its loss, expanding its output.

Original: As α starts to saturate and each
Revised: As α starts to saturate and each

Original: D learns how to distinguish between real and fake samples, G is forced to lower its loss by creating both realistic and diverse samples.
Revised: D learns how to distinguish between real and fake samples, G is forced to lower its loss by creating both realistic and diverse samples.

Original:  E XPERIMENTAL R ESULTS
Revised:  E XPERIMENTAL R ESULTS

Original: We validated the effects of using different α functions on MNIST (LeCun and Cortes (2010)),
Revised: We validated the effects of using different α functions on MNIST (LeCun and Cortes (2010)),

Original: CIFAR-10 (Krizhevsky (2009)), and cropped CelebA (Liu et al.
Revised: CIFAR-10 (Krizhevsky (2009)), and cropped CelebA (Liu et al.

Original: To quantitatively evaluate such effects, we used the Fr ´ echet Inception Distance (Heusel et al.
Revised: To quantitatively evaluate such effects, we used the Fr ´ echet Inception Distance (Heusel et al.

Original: (2017)), or FID, since it has been shown to be sensitive to image quality as well as mode collapse (Lucic et al.
Revised: (2017)), or FID, since it has been shown to be sensitive to image quality as well as mode collapse (Lucic et al.

Original: (2017)), with the returned distance increasing notably when modes are missing from the generated data.
Revised: (2017)), with the returned distance increasing notably when modes are missing from the generated data.

Original: We used several variations of the standard FID for a thorough study of α ’s effects in training, as well as the inﬂuence of using a different number of discriminators in our framework.
Revised: We used several variations of the standard FID for a thorough study of α ’s effects in training, as well as the inﬂuence of using a different number of discriminators in our framework.

Original: 4. I NTRA FID
Revised: 4. I NTRA FID

Original: To measure the variety of samples of the generated set, we propose to calculate the FID between two subsets of 10K randomly picked fake samples generated at the end of every thousand iterations.
Revised: To measure the variety of samples of the generated set, we propose to calculate the FID between two subsets of 10K randomly picked fake samples generated at the end of every thousand iterations.

Original: We call this metric Intra FID.
Revised: We call this metric Intra FID.

Original: Important to note that Intra FID only measures the diversity in the generated set, not its realism.
Revised: Important to note that Intra FID only measures the diversity in the generated set, not its realism.

Original: Hence, higher values indicate more diversity within the generated samples while lower values might indicate mode collapse in the generated set.
Revised: Hence, higher values indicate more diversity within the generated samples while lower values might indicate mode collapse in the generated set.

Original: The relation between
Revised: The relation between

Original: Intra FID and progressive values of α is shown in Figure 4.
Revised: Intra FID and progressive values of α is shown in Figure 4.

Original: We observe a strong correlation between α ’s growth and variety in the set, especially in beginning to mid-training.
Revised: We observe a strong correlation between α ’s growth and variety in the set, especially in beginning to mid-training.

Original: Later on, as α saturates, the variety is kept (represented by the stability of the Intra
Revised: Later on, as α saturates, the variety is kept (represented by the stability of the Intra

Original: FID).
Revised: FID).

Original: It is further visible that α sigm , α soft , and α tanh converge to similar Intra FID on all datasets.
Revised: It is further visible that α sigm , α soft , and α tanh converge to similar Intra FID on all datasets.

Original: Important to note, that, to ease the visualization, the graphs only represent 0 ≤ α ≤ 1 , with α ident ’s values naturally surpassing 1 as time progresses.
Revised: Important to note, that, to ease the visualization, the graphs only represent 0 ≤ α ≤ 1 , with α ident ’s values naturally surpassing 1 as time progresses.

Original: 4. C UMULATIVE I NTRA FID
Revised: 4. C UMULATIVE I NTRA FID

Original: To analyze the sample variety over time, we summed the Intra FID values obtained from every thousand iterations.
Revised: To analyze the sample variety over time, we summed the Intra FID values obtained from every thousand iterations.

Original: Hence, higher values indicate that the model was able to promote more variety in the set across time.
Revised: Hence, higher values indicate that the model was able to promote more variety in the set across time.

Original: Results are shown in Figure 5, where we observe that using more discriminators leads to more variety across all datasets and α functions.
Revised: Results are shown in Figure 5, where we observe that using more discriminators leads to more variety across all datasets and α functions.

Original: Moreover, using α = 0 leads to lower variety compared to using positive α values, with α sigm , α soft , and α tanh obtaining similar values throughout the different datasets.
Revised: Moreover, using α = 0 leads to lower variety compared to using positive α values, with α sigm , α soft , and α tanh obtaining similar values throughout the different datasets.

Original: Even though α ident promotes the highest variety, the generated samples lack realism, as previously witnessed in the toy experiment and further discussed next.
Revised: Even though α ident promotes the highest variety, the generated samples lack realism, as previously witnessed in the toy experiment and further discussed next.

Original: 4. M EAN AND M INIMUM FID
Revised: 4. M EAN AND M INIMUM FID

Original: To analyze both the realism and variety of the generated samples, we used the standard FID calculated between 10K fake samples and the real training data.
Revised: To analyze both the realism and variety of the generated samples, we used the standard FID calculated between 10K fake samples and the real training data.

Original: Lower values should indicate both diversity and high-quality samples.
Revised: Lower values should indicate both diversity and high-quality samples.

Original: The Mean FID and Minimum FID across 50K iterations are presented in Table 1 for each dataset.
Revised: The Mean FID and Minimum FID across 50K iterations are presented in Table 1 for each dataset.

Original: We observe that the best values, both in terms of mean and minimum, are obtained when using a higher number of discriminators, i.e., 5 or 10, and α tanh , α soft , and α sigm .
Revised: We observe that the best values, both in terms of mean and minimum, are obtained when using a higher number of discriminators, i.e., 5 or 10, and α tanh , α soft , and α sigm .

Original: Moreover, the high distances obtained when using α ident conﬁrm the lack of realism of the generated samples, highlighting the importance of constraining α by the properties previously stated in Section 3.
Revised: Moreover, the high distances obtained when using α ident conﬁrm the lack of realism of the generated samples, highlighting the importance of constraining α by the properties previously stated in Section 3.

Original: [Table]
Revised: [Table]

Original: 4. G ENERATED SAMPLES
Revised: 4. G ENERATED SAMPLES

Original: The generated samples on each dataset using 1 and 10 discriminators with different α are presented in Figure 6.
Revised: The generated samples on each dataset using 1 and 10 discriminators with different α are presented in Figure 6.

Original: For an objective assessment of the variety by the end of each iteration, the Intra FID is also provided.
Revised: For an objective assessment of the variety by the end of each iteration, the Intra FID is also provided.

Original: We observe the superiority of the generated samples, both in terms of realism and variety, when using α sigm , α soft , and α tanh on all datasets.
Revised: We observe the superiority of the generated samples, both in terms of realism and variety, when using α sigm , α soft , and α tanh on all datasets.

Original: However, α tanh seems to show a delayed ability in generating realistic samples, possibly due to the increase of α at a steeper fashion.
Revised: However, α tanh seems to show a delayed ability in generating realistic samples, possibly due to the increase of α at a steeper fashion.

Original: The inability of generating realistic samples when using α ident is also clearly detected on all datasets, as previously discussed.
Revised: The inability of generating realistic samples when using α ident is also clearly detected on all datasets, as previously discussed.

Original: More importantly, the high variety on the generated set, observed by the high Intra FID, is witnessed since very early iterations when using α sigm , α soft , and α tanh .
Revised: More importantly, the high variety on the generated set, observed by the high Intra FID, is witnessed since very early iterations when using α sigm , α soft , and α tanh .

Original: The observed mitigation of mode collapse is carried out throughout the whole training.
Revised: The observed mitigation of mode collapse is carried out throughout the whole training.

Original: When using standard GANs, we notice severe mode collapse, especially early on training.
Revised: When using standard GANs, we notice severe mode collapse, especially early on training.

Original: When using 10 discriminators and α set to 0, we notice a slight variation in the generated set, yet, this is only detected after a decent number of iterations, when each D has seen enough samples to guide its judgment to a speciﬁc data mode due to the usage of different microbatch for each D , delaying sample variety substantially.
Revised: When using 10 discriminators and α set to 0, we notice a slight variation in the generated set, yet, this is only detected after a decent number of iterations, when each D has seen enough samples to guide its judgment to a speciﬁc data mode due to the usage of different microbatch for each D , delaying sample variety substantially.

Original: Thus, using positive α values is shown to be a necessary measure to stimulate variety since the beginning and until the end of training.
Revised: Thus, using positive α values is shown to be a necessary measure to stimulate variety since the beginning and until the end of training.

Original:  M ETHOD C OMPARISON
Revised:  M ETHOD C OMPARISON

Original: We proceeded to compare different settings of microGAN to other existing methods on 3 different datasets: CIFAR-10, STL-10 (Coates et al.
Revised: We proceeded to compare different settings of microGAN to other existing methods on 3 different datasets: CIFAR-10, STL-10 (Coates et al.

Original: (2011)), and ImageNet (Deng et al.
Revised: (2011)), and ImageNet (Deng et al.

Original: We downsampled the images of the last two datasets down to 32x32 pixels.
Revised: We downsampled the images of the last two datasets down to 32x32 pixels.

Original: We used Inception Score (Salimans et al.
Revised: We used Inception Score (Salimans et al.

Original: or IS (higher is better) as the ﬁrst quantitative metric.
Revised: or IS (higher is better) as the ﬁrst quantitative metric.

Original: Even though IS has been shown to be less correlated with human judgment than FID, most previous works only report results on this metric, making it a useful measure for model comparisons.
Revised: Even though IS has been shown to be less correlated with human judgment than FID, most previous works only report results on this metric, making it a useful measure for model comparisons.

Original: Out of fairness to the single discriminator methods that we compare our method against, we used only 2 discriminators in our experiments.
Revised: Out of fairness to the single discriminator methods that we compare our method against, we used only 2 discriminators in our experiments.

Original: The architectures and training settings used for all the experiments can be found in the
Revised: The architectures and training settings used for all the experiments can be found in the

Original: Appendix.
Revised: Appendix.

Original: The comparison results are shown in Table 2.
Revised: The comparison results are shown in Table 2.

Original: We point special attention to the underlined method representing standard GANs, since it was the only method executed with our own implementation and identical training settings as microGAN.
Revised: We point special attention to the underlined method representing standard GANs, since it was the only method executed with our own implementation and identical training settings as microGAN.

Original: Thus, this represents the only method directly comparable to ours.
Revised: Thus, this represents the only method directly comparable to ours.

Original: We notice a fair improvement of IS on all the tested datasets, observing an increase up to around 15% for CIFAR-10, 7% for STL-10, and 5% for ImageNet.
Revised: We notice a fair improvement of IS on all the tested datasets, observing an increase up to around 15% for CIFAR-10, 7% for STL-10, and 5% for ImageNet.

Original: This indicates the success of our approach on improving the standard GANs framework on multiple datasets with different sizes and challenges.
Revised: This indicates the success of our approach on improving the standard GANs framework on multiple datasets with different sizes and challenges.

Original: [Table]
Revised: [Table]

Original: On CIFAR-10, microGAN achieves competitive results, signiﬁcantly outperforming GMAN with 5 discriminators while using a similar architecture.
Revised: On CIFAR-10, microGAN achieves competitive results, signiﬁcantly outperforming GMAN with 5 discriminators while using a similar architecture.

Original: We argue that the use of more powerful architectures in the higher ranked methods plays a big role in their end score, especially for DCGAN.
Revised: We argue that the use of more powerful architectures in the higher ranked methods plays a big role in their end score, especially for DCGAN.

Original: Nonetheless, we acknowledge that using different objectives for each D (as proposed in D2GAN) seems to be beneﬁcial in a multi-discriminator setting, representing a good path to follow in the future.
Revised: Nonetheless, we acknowledge that using different objectives for each D (as proposed in D2GAN) seems to be beneﬁcial in a multi-discriminator setting, representing a good path to follow in the future.

Original: Moreover, we observe that using extra autoencoders (DFM) or classiﬁers (MGAN) in the framework can help to achieve a better performance in the end.
Revised: Moreover, we observe that using extra autoencoders (DFM) or classiﬁers (MGAN) in the framework can help to achieve a better performance in the end.

Original: However, we note that MGAN makes use of a 10 generator framework, on top of an extra classiﬁer, to achieve the presented results.
Revised: However, we note that MGAN makes use of a 10 generator framework, on top of an extra classiﬁer, to achieve the presented results.

Original: Furthermore, the generated samples presented in their paper (Hoang et al.
Revised: Furthermore, the generated samples presented in their paper (Hoang et al.

Original: (2017)) indicate signs of partial mode collapse, which is not reﬂected in its high IS
Revised: (2017)) indicate signs of partial mode collapse, which is not reﬂected in its high IS.

Original: [Table]
Revised: [Table]

Original: [Table] We further compared our best FID with  the reported methods in Lucic et al.
Revised: We further compared our best FID with a subset of the reported methods in Lucic et al.

Original: (2017) on CIFAR-10and CelebA (Table 3).
Revised: (2017), namely GANs, both with the original and modiﬁed objective, LSGAN, and DRAGAN on CIFAR-10 (Table 3).

Original: Important to note that their results were obtained by picking the best FID at every 5’th epoch, after a large-scale hyperparameter search on bigger architectures.
Revised: These methods were chosen since they represent interesting variants of standard GANs. We extended each method to an ensemble of discriminators, for a fair comparison to our multiple discriminator approach.

Original: On the other hand, we only varied α and report the FID at the end of training, while using the modest architecture of the last experiment. Nevertheless, we managed to achieve competitive results on both datasets.
Revised: We used the same architecture of the last experiment for all methods. We observe that all variants of microGAN outperform the rest of the compared methods under controlled and equal experiments.

Original: A subset of the generated samples produced by the different variations of microGAN reported in 
Revised: A subset of the generated samples produced by the different variations of microGAN reported in Table 2 are shown in

Original: Table 2 are shown in Figure 7, where we observe high variety and realism across all generated sets.
Revised: Figure 7, where we observe high variety and realism across all generated sets.

Original: Extended results are provided in the Appendix.
Revised: Extended results are provided in the Appendix

Original:  C ONCLUSIONS
Revised:  C ONCLUSIONS

Original: In this work, we present a novel framework, named microGAN, where each D performs microbatch discrimination, differentiating between samples within and outside its fake microbatch.
Revised: In this work, we present a novel framework, named microGAN, where each D performs microbatch discrimination, differentiating between samples within and outside its fake microbatch.

Original: This behavior is enforced by the diversity parameter α , that is indirectly self-learned by G .
Revised: This behavior is enforced by the diversity parameter α , that is indirectly self-learned by G .

Original: In the ﬁrst iterations, G increases α to lower its loss, expanding its output.
Revised: In the ﬁrst iterations, G increases α to lower its loss, expanding its output.

Original: Then, as α gradually saturates and each D learns how to better distinguish between real and fake samples, G is forced to fool each D by promoting realism in its output, while keeping the diversity in the generated set.
Revised: Then, as α gradually saturates and each D learns how to better distinguish between real and fake samples, G is forced to fool each D by promoting realism in its output, while keeping the diversity in the generated set.

Original: We show evidence that our solution produces realistic and diverse samples on multiple datasets of different sizes and nature, ultimately mitigating mode collapse.
Revised: We show evidence that our solution produces realistic and diverse samples on multiple datasets of different sizes and nature, ultimately mitigating mode collapse.

Original: B T HEORETICAL D ISCUSSION
Revised: B T HEORETICAL D ISCUSSION

Original: To better understand how our approach differs from the original GANs in promoting variety in the generated set, we used a simpliﬁed version of the minimax game where we freeze each D k and train
Revised: To better understand how our approach differs from the original GANs in promoting variety in the generated set, we used a simpliﬁed version of the minimax game where we freeze each D k and train

Original: G until convergence.
Revised: G until convergence.

Original: In the most extreme case, we say that we have mode collapse when:
Revised: In the most extreme case, we say that we have mode collapse when:

Original: For all z (cid:48) ∼ p
Revised: For all z (cid:48) ∼ p

Original: g
Revised: g

Original: ( z ) , G ( z
Revised: ( z ) , G ( z

Original: [Equation]
Revised: [Equation]

Original: Theorem 1.
Revised: Theorem 1.

Original: In original GANs, mode collapse fully minimizes G ’s loss when we train G exhaustively without updating D .
Revised: In original GANs, mode collapse fully minimizes G ’s loss when we train G exhaustively without updating D .

Original: Proof.
Revised: Proof.

Original: The optimal x ∗ is the one that maximizes D ’s output, where:
Revised: The optimal x ∗ is the one that maximizes D ’s output, where:

Original: [Equation]
Revised: [Equation]

Original: Thus, assuming G would eventually learn how to produce x ∗ , mode collapse on x ∗ would fully minimize its loss, making x ∗ independent of z .
Revised: Thus, assuming G would eventually learn how to produce x ∗ , mode collapse on x ∗ would fully minimize its loss, making x ∗ independent of z .

Original: Theorem 2.
Revised: Theorem 2.

Original: In microGAN, assuming α > 0 , x ∼ p g must be dependent of z for G to fully minimize its loss, eliminating mode collapse when we train G exhaustively without updating any D k .
Revised: In microGAN, assuming α > 0 , x ∼ p g must be dependent of z for G to fully minimize its loss, eliminating mode collapse when we train G exhaustively without updating any D k .

Original: Proof.
Revised: Proof.

Original: The value function between G and each D k can be expressed as follows:
Revised: The value function between G and each D k can be expressed as follows:

Original: [Equation]
Revised: [Equation]

Original: To fully minimize its loss in relation to D k , G must ﬁnd: x
Revised: To fully minimize its loss in relation to D k , G must ﬁnd: x

Original: [Equation]
Revised: [Equation]

Original: and
Revised: = argmax [Equation] D

Original: [Equation]
Revised: [Equation]

Original: x
Revised: x

Revised: [Equation]

Original:  D 
Revised: and x [Equation] = argmin x D k

Original: [Equation]
Revised: [Equation]

Revised: x

Revised: [Equation]

Original: which implies
Revised: which implies

Original: [Equation]
Revised: [Equation]

Original: Thus, generating different outputs for different z is a requirement to fully minimize G ’s loss regarding each D k .
Revised: Thus, generating different outputs for different z is a requirement to fully minimize G ’s loss regarding each D k .

Original: Since we sum all V ( D k , G ) to calculate G ’s ﬁnal loss, this also applies to overall adversarial set, concluding the proof.
Revised: Since we sum all V ( D k , G ) to calculate G ’s ﬁnal loss, this also applies to overall adversarial set, concluding the proof.

Original: C T RAINING SETTINGS
Revised: C T RAINING SETTINGS

Original: The architectural and training settings used in Sections 3, 4, and 5 are presented in Tables 4, 5, and 6, respectively.
Revised: The architectural and training settings used in Sections 3, 4, and 5 are presented in Tables 4, 5, and 6, respectively.

Original: For the FID comparison on CIFAR-10 and CelebA in Section 5, we used the same architectures as Table 6 but with a batch size of 64 on both datasets, and ran for 78K iterations on CIFAR-10 and 125K iterations on CelebA.
Revised: For the FID comparison on CIFAR-10 and CelebA in Section 5, we used the same architectures as Table 6 but with a batch size of 64 on both datasets, and ran for 78K iterations on CIFAR-10 and 125K iterations on CelebA.

Original: D S IGMOID INITIAL VALUE
Revised: D S IGMOID INITIAL VALUE

Original: In Figure 8, we show and discuss the effects of using different β sigm on α sigm on the toy dataset, giving more insights regarding the choice of β sigm = − 1 .
Revised: In Figure 8, we show and discuss the effects of using different β sigm on α sigm on the toy dataset, giving more insights regarding the choice of β sigm = − 1 .

Original: 8 mentioned in Section 3.
Revised: 8 mentioned in Section 3.

Original: [Figure]
Revised: [Figure]

Revised: E T OY DATASET COMPARISONS

Revised: Figure 9 shows how different methods compare using the above mentioned toy dataset.

Original: E
Revised: We compared microGAN’s results (K = 8, α sigm ) to the standard GAN (Goodfellow et al. (2014)), UnrollledGAN (Metz et al. (2016)), D2GAN (Nguyen et al. (2017)), and MGAN (Hoang et al.

Revised: We observe bigger sample diversity with our method, while still approximating the real data distribution.

Original:  E XTENDED R ESULTS
Revised: F E XTENDED R ESULTS

Original: Additional results for CIFAR-10, STL-10, and ImageNet are presented bellow.
Revised: Additional results for CIFAR-10, STL-10, and ImageNet are presented bellow.

Original: F EATURE MAPS
Revised: F EATURE MAPS

Original: N ONLINEARITY
Revised: N ONLINEARITY

Original: [Figure]
Revised: [Table]

Original: z ) : z [Figure] Normal (0 , I )
Revised: CONNECTED

Original: [Figure]
Revised: [Table]

Original: [Figure] [Figure] L INEAR [Figure]
Revised: ULLY CONNECTED

Original: [Figure]
Revised: [Table]

Original: S OFTPLUS
Revised: ULLY CONNECTED

Original: [Figure] [Table] [Figure]
Revised: L INEAR D

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: x )

Original: [Table]
Revised: [Table]

Original: [Figure]

Revised: ULLY CONNECTED

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Figure] [Figure] [Figure]
Revised: OFTPLUS

Original: [Figure]
Revised: [Table]

Original: E

Original: [Figure] [Figure] [Figure]
Revised: [Table] [Table]

Original: [Figure]
Revised: [Figure]