Revised: C OM

Original:  C O GAN S : C OLLABORATIVE G ENERATIVE A DVERSARIAL N ETWORKS
Revised: M UT E-GAN: C OMPTETITIVE M ULTIPLE E FFICIENT

Revised: G

Revised: ENERATIVE A

Revised: DVERSARIAL

Revised: N ETWORKS

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: In complex creative scenarios, co-creativity by multiple agents offer great advantages.
Revised: In complex creative scenarios, co-creativity by multiple agents offers great advantages.

Original: Each agent has a specific skill set and a set of abilities, which is sometimes not enough to perform ageneral, large and complex task single-handed.
Revised: Each agent has a specific skill set and a set of abilities, which is generally not enough to perform a large and complex task single-handed.

Original: These kinds of tasks benefit substantially from collaboration.
Revised: These kinds of tasks benefit substantially from collaboration.

Original: In deep learning applications, data generation is an example of such a complex, potentially multi-modal task.
Revised: In deep learning applications, data generation is an example of such a complex, potentially multi-modal task.

Original: Previous Generative Adversarial Networks (GANs) focused on using a single generator to generate multi-modal datasets, which is sometimes known to face issues such as mode-collapse and failure to converge.
Revised: Previous Generative Adversarial Networks (GANs) focused on using a single generator to generate multi-modal datasets, which is sometimes known to face issues such as mode-collapse and failure to converge.

Revised: Single generators also have to be very large so that they can generalize complex datasets, so this method can easily run into memory constraints.

Original: The  multi-generator based works such as MGAN, MMGAN, MADGAN and AdaGAN either require training a classifier online, the use of complex mixture models or sequentially adding generators, which is computationally complex.
Revised: The current multi-generator based works such as MGAN, MMGAN, MADGAN and AdaGAN either require training a classifier online, the use of complex mixture models or sequentially adding generators, which is computationally complex.

Original: In this work, we present a simple, novel approach of training collaborative GANs (CoGAN), with multiple generators and a single critic/discriminator, without introducing external complexities such as a classifier model.
Revised: In this work, we present a simple, novel approach of training competitive multiple efficient GANs (ComMutE-GANs), with multiple generators and a single critic/discriminator, without introducing external complexities such as a classifier model.

Original: We show that this method of workload division meets the state-of-the-art quality metrics, and makes GAN training robust.
Revised: We introduce a new component to the generator loss during GAN training, based on the Total Variation Distance (TVD). Our method offers a robust, stable, memory efficient and easily parallelizable architecture.

Original: We present a proof-of-concept on the MNIST dataset, which has 10 modes of data.
Revised: We present a proof-of-concept on the MNIST dataset, which has 10 modes of data.

Original: The individual generators learn to generate different digits from the distribution, and together learn to generate the whole distribution.
Revised: The individual generators learn to generate different digits from the distribution, and together learn to generate the whole distribution.

Original: We introduce a new component to the generator loss during GAN training, based on the Total Variation Distance (TVD) and show that it significantly improves stability during training and performance over state-of-the-art single generator GANs.
Revised: We compare ComMutE-GANs with larger single-generator GANs and show its memory efficiency and increased accuracy.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: The 7 C’s of creativity as stated by Lubart (2017) describe the main aspects of the creativity:

Original: Creators (person-centered characteristics), Creating (the creative process), Collaborations (co-creating),

Original: Contexts (environmental conditions), Creations (the nature of creative work), Consumption (the adoption of creative products) and Curricula (the development and enhancement of creativity).

Original: With respect to human beings, ”Creators” refer to any and all who engage in creative thinking.
Revised: With respect to human beings, ”Creators” refer to any and all who engage in creative thinking.

Original: When people learn about new topics, they create cognitive structures that allow them to understand the topics; they generate concepts that are new to them, although possibly already very well known to others.
Revised: When people learn about new topics, they create cognitive structures that allow them to understand the topics; they generate concepts that are new to them, although possibly already very well known to others.

Original: This is creativity at a strictly intra-personal level.
Revised: This is creativity at a strictly intra-personal level.

Original: When working in a social setting, such as a company or a classroom, one has to broaden this horizon to include ”Co-creativity”.
Revised: When working in a social setting, such as a company or a classroom, one has to broaden this horizon to include ”Co-creativity”.

Original: ”Creativity through collaboration” summarizes the definition of co-creativity as defined by Lubart (2017).
Revised: ”Creativity through collaboration” summarizes the definition of co-creativity as defined by Lubart (2017).

Original: Often, collaborators have different or complementary skills that enable them to frequently produce shared creations that they could not or would not produce on their own (Lubart & Thornhill-Miller, 2020).
Revised: Often, collaborators have different or complementary skills that enable them to frequently produce shared creations that they could not or would not produce on their own (Lubart & Thornhill-Miller, 2020).

Original: AI-aided co-creation has also been proven to improve general well-being (Yu et al., 2021).
Revised: AI-aided co-creation has also been proven to improve general well-being (Yu et al., 2021).

Original: Generative Adversarial Nets (GANs) are implicit generative models where one or more generators play a zero-sum game with a discriminator to recreate and potentially expand a chosen dataset.
Revised: Generative Adversarial Nets (GANs) are implicit generative models where one or more generators play a zero-sum game with a discriminator to recreate and potentially expand a chosen dataset.

Original: According to the definition established above, the generator models in modern day GANs such as those described in the works of  Goodfellow et al.
Revised: According to the definition established above, the generator models in modern day GANs such as those described in the works of Karras et al. Sauer et al. ; Karras et al. Goodfellow et al.

Original: ; Radford et al.
Revised: ; Radford et al.

Original: Arjovsky et al.
Revised: Arjovsky et al.

Original: Gulrajani et al.
Revised: Gulrajani et al.

Original: (2017) exhibit creativity on an intra-personal level.
Revised: (2017) exhibit creativity on an intra-personal level.

Original: Accordingly, generative networks have been applied in many creative applications such as painting (Ganin et al., 2018; Mellor et al., 2019; Parikh & Zitnick, 2020), doodling (Ha & Eck, 2017; Cao et al., 2019) and extending fictional languages (Zacharias et al., 2022).
Revised: Accordingly, generative networks have been applied in many creative applications such as painting (Ganin et al., 2018; Mellor et al., 2019; Parikh & Zitnick, 2020), doodling (Ha & Eck, 2017; Cao et al., 2019) and extending fictional languages (Zacharias et al., 2022).

Original: Most noticeable, in all the applications listed above, a single generative agent was applied to perform a complex task rather than breaking it down into smaller, 
Revised: Most noticeable, in all the applications listed above, a single, large, generative agent was applied to perform a complex task rather than breaking it down into smaller, more easily manageable sub-tasks.

Revised: This approach, although effective, is upper-bounded by memory constraints.

Original:  Other implementations of GANs  such as MGAN (Hoang et al., 2017), MMGAN (Pandeva & Schubert, 2019), MADGAN (Ghosh et al., 2018) and AdaGAN (Tolstikhin et al., 2017) try to rectify the missing co-creativity functionality of GANs by using a mixture of multiple generators, modeling the input latent space as a mixture model, requiring the discriminator to classify as well and sequentially train and add generators to the mixture, respectively.
Revised: Inspiration from co-creativity research aims to resolve these constraints. Other implementations of GANs that use collaboration, such as MGAN (Hoang et al., 2017), MMGAN (Pandeva & Schubert, 2019), MADGAN (Ghosh et al., 2018) and AdaGAN (Tolstikhin et al., 2017) try to rectify the missing co-creativity functionality of GANs by using a mixture of multiple generators, modeling the input latent space as a mixture model, requiring the discriminator to classify as well and sequentially train and add generators to the mixture, respectively.

Original: MGAN and MADGAN require the discriminator to have classification capabilities and both implementations force the generators to generate different modes.
Revised: MGAN and MADGAN require the discriminator to have classification capabilities and both implementations force the generators to generate different modes.

Original: The AdaGAN implementation poses a problem of computational complexity because of the sequential training nature.
Revised: The AdaGAN implementation poses a problem of computational complexity because of the sequential training nature.

Original: MMGAN on the other hand focuses on using mixtures at the input latent space level rather than on separating generators.
Revised: MMGAN on the other hand focuses on using mixtures at the input latent space level rather than on separating generators.

Original: Our work stands to provide an easier approach to co-creativity than the ones presented above.
Revised: Our work stands to provide an easier approach to co-creativity than the ones presented above.

Original: It does not require the online training of a classifier, we do use a pre-trained MNIST classifier to augment the generator loss with the Total Variation Distance (TVD) to enforce mode separation during training, but mode separation maybe enforced using other methods, not requiring this pre-trained classifier at all.
Revised: It does not require the online training of a classifier, we do use a pre-trained MNIST classifier to augment the generator loss with the Total Variation Distance (TVD) to enforce mode separation during training, but mode separation maybe enforced using other methods, not requiring this pre-trained classifier at all.

Original: We also show that this is easily and efficiently scalable to more than two generators, which can all be trained in parallel without making complex transformations to the latent space.
Revised: We also show that this is easily and efficiently scalable to more than two generators, which can all be trained in parallel without making complex transformations to the latent space.

Original: Analogous to human behaviour in a social setting, collaboration among two or more such generators allows each individual generator to focus on and specialize in a specific sub-task, making the training process more stable,  the generated images clearer and the distribution of the generated images closer to the actual distribution of the chosen MNIST dataset.
Revised: Analogous to human behaviour in a social setting, collaboration among two or more such generators allows each individual generator to focus on and specialize in a specific sub-task, making the training process more stable, more efficient (memory-wise), the generated images clearer and the distribution of the generated images closer to the actual distribution of the chosen MNIST dataset.

Original:  B ACKGROUND 2. GAN S 
Revised:  B ACKGROUND 2. GAN S , WGAN S AND WGAN-GP

Original: Generative Adversarial Nets (GANs) (Goodfellow et al., 2014)were first presented as a novel, gametheory based approach towards data generation. These type of networks consist of two Artificial Neural Networks (ANNs), where the generator ( G ( z ) )  takes innoise, that is randomly sampled from a possibly multi-dimensional distribution, and using its weight parameters, transforms it into another distribution.
Revised: Generative Adversarial Nets (GANs) (Goodfellow et al., 2014) consist of two neural networks: a generator ( G ( z ) ) that takes in randomly sampled noise, and generates fake images; a discriminator ( D ( x )), that takes in batches of real and fake data points and outputs a 0 for fake images or 1 for real ones.

Original: The noise z ∼ P z is transformed into ˜ x through successive linear transformations followed by the application of non-linear activation functions. The second model, termed as the discriminator ( D ( x )), takes as an input batches of real and fake data points and outputs a 0 or 1 based on whether it computes the data point to be from the generator’s distribution ( P g ) or the actual data distribution ( P r ), respectively. The goal of the generator is to minimize the Jensen-Shannon divergence (JSD) between P r and P g . The objective functionthat is being solved under this setup is given by
Revised: They minimize the well-known GAN objective function

Original: [Equation]
Revised: [Equation]

Original: Where ˜ x = G ( z ) , z ∼ P z , P r is the data distribution, P g is the generator distribution and G and D are the generator function and discriminator function, respectively.
Revised: Where ˜ x = G ( z ) , z ∼ P z , P r is the data distribution, P g is the generator distribution and G and D are the generator function and discriminator function, respectively.

Original: This particular approach is unstable during training and suffers from commonly known problems such as failure to converge, mode collapse and vanishing gradients (Arjovsky & Bottou, 2017).

Original: According to Arjovsky et al.

Original: (2017), the distance metric (JSD), which this method attempts to minimize, suffers from potential discontinuities with respect to the generator’s parameters, leading to training difficulty.

Original: A variation of this method is the Deep Convolution GAN (DCGAN)
Revised: A variation of this method is the Deep Convolution GAN (DCGAN)

Original: (Radford et al., 2015), that uses
Revised: (Radford et al., 2015), that uses

Original: Convolutional Neural Networks (CNNs), to improve image data generation significantly, but ends up suffering from the same problems in training, as specified above.
Revised: Convolutional Neural Networks (CNNs), to improve image data generation significantly.

Original: Wasserstein GAN (WGAN) (Arjovsky et al., 2017) presents an improvement over vanilla GANs by presenting and rectifying problems with the  GAN objective function.
Revised: Wasserstein GAN (WGAN) (Arjovsky et al., 2017) improves vanilla GANs by presenting and rectifying problems with the original GAN objective function.

Original: The claim is that the original objective function which the GAN generator aims to minimize (the JSD) can at times be discontinuous with respect to the generator’s parameters, which induces instability during the training.

Original: Two notable changes were made, to rectify the training instabilities: changing the distance metric to Wasserstein distance or Earth-Mover distance and changing the discriminator network to a critic network, that instead of outputting a 0 or a 1 , outputs a linear activation of the output layer.

Original: These changes required the addition of the constraint that the weights of the critic network must satisfy the

Original: Lipschitz constraints. These constraints were enforced using weight clipping in the original paper, i.e. w ∈ [ − 0 . l . The objective function which the WGAN solves can be given by 
Revised: In WGANs, the Wasserstein distance between real data and generated data is minimized by optimizing over the objective

Original: [Equation]
Revised: [Equation]

Original: which is obtained by using the Kantorovich-Rubinstein duality (Villani, 2009).

Original: Here, D is the space of 1-Lipschitz functions, P g is the generator distribution, P r is the data distribution and ˜ x

Original: = G ( z ) , z ∼ P z .

Original: In that case, under an optimal critic, minimizing the value function with respect to the generator parameters minimizes W ( P r , P g ) , W ( ., . )

Original: denoting the Wasserstein distance.

Original: The WGAN-GP method (Gulrajani et al., 2017) is an improvement over this method which uses their novel gradient penalty loss, instead of the former weight clipping to implicitly enforce the Lipschitz constraints.
Revised: The WGAN-GP method (Gulrajani et al., 2017) is an improvement over this method which uses their novel gradient penalty loss, instead of the former weight clipping to implicitly enforce the Lipschitz constraints.

Original: The new objective function solved by WGAN-GP is given by
Revised: The new objective function solved by WGAN-GP is given by

Original: [Equation]
Revised: [Equation]

Original: where the first two terms are the same as equation (2) and the third one is the gradient penalty (GP) term.
Revised: where the first two terms are the same as equation (2) and the third one is the gradient penalty (GP) term.

Original: Here, ˆ x = ϵ x + (1 − ϵ )
Revised: Here, ˆ x = ϵ x + (1 − ϵ )

Original: ˜ x and ϵ ∼ U [0 , 1] .
Revised: ˜ x and ϵ ∼ U [0 , 1] .

Original: P ˆ x is the probability distribution associated with ˆ x .λ
Revised: P ˆ x is the probability distribution associated with ˆ x .

Original: = 10 is the default gradient penalty coefficient as in the original paper.
Revised: Modern GANs such as Karras et al. Sauer et al.

Original: The training stability improvement because of using Wasserstein distance paired with the GP to implicitly apply the Lipschitz constraint shows a substantial improvement over vanilla GANs. The
Revised: ; Karras et al. (2020) that provide state-of-the-art performance on large-scale image synthesis improve these basic methods by making their models larger and training procedures more complex.

Original: WGAN-GP method is very closely related to the implementation presented in this contribution.
Revised: Our method explores another direction, namely, more instances of compact generators competing with each other.

Original: When we refer to convergence , it will be equivalent to saying that the discriminator loss goes to 0 .
Revised: The idea takes inspiration from social dynamics and co-creativity research.

Original: 2. GAN S FOR MULTI - MODAL DATASETS
Revised: 2. GAN S FOR MULTI - MODAL DATASETS

Original: In past works, there have been attempts with varied scope, to learn multi-modal datasets such as multi-class generation.
Revised: In past works, there have been attempts with varied scope, to learn multi-modal datasets such as multi-class generation.

Original: The most notable are the MMGAN by Pandeva & Schubert (2019), MGAN by Hoang et al.
Revised: The most notable are the MMGAN by Pandeva & Schubert (2019), MGAN by Hoang et al.

Original: (2017), MAD-GAN by Ghosh et al.
Revised: (2017), MAD-GAN by Ghosh et al.

Original: (2018) and AdaGAN by Tolstikhin et al.
Revised: (2018) and AdaGAN by Tolstikhin et al.

Original: MMGAN approaches multi-modal generation task by considering the latent space z as a Gaussian mixture model consisting multiple modes.
Revised: MMGAN approaches multi-modal generation task by considering the latent space z as a Gaussian mixture model consisting multiple modes.

Original: Initially, a Gaussian distribution is sampled from a cluster of Gaussian distributions ( Cat ( K, 1 / K )), where the dataset has K ”clusters”.
Revised: Initially, a Gaussian distribution is sampled from a cluster of Gaussian distributions ( Cat ( K, 1 / K )), where the dataset has K ”clusters”.

Original: Then, z is sampled from this distribution and transformed by the generator to create images.
Revised: Then, z is sampled from this distribution and transformed by the generator to create images.

Original: This method uses an extra encoder (E) model to predict the cluster of data objects.
Revised: This method uses an extra encoder (E) model to predict the cluster of data objects.

Original: The MGAN method (Hoang et al., 2017) implements multiple generators that generate K different batches.
Revised: The MGAN method (Hoang et al., 2017) implements multiple generators that generate K different batches.

Original: One of these batches is sampled randomly using a predefined distribution π .
Revised: One of these batches is sampled randomly using a predefined distribution π .

Original: The generated batch is fed into a ”discriminator + classifier” network with shared weights in all layers except the output layer.
Revised: The generated batch is fed into a ”discriminator + classifier” network with shared weights in all layers except the output layer.

Original: The goal of the classifier is to predict which generator generated the images, and the discriminator has the same functionality as the vanilla GAN method.
Revised: The goal of the classifier is to predict which generator generated the images, and the discriminator has the same functionality as the vanilla GAN method.

Original: MAD-GAN (Ghosh et al., 2018) is a method which combines the classifier and discriminator functionalities into a single model.
Revised: MAD-GAN (Ghosh et al., 2018) is a method which combines the classifier and discriminator functionalities into a single model.

Original: All of the previously mentioned methods require that an extra model be trained with the original
Revised: All of the previously mentioned methods require that an extra model be trained with the original

Original: GAN or multi-generator setup to force separation of modes among the generators.
Revised: GAN or multi-generator setup to force separation of modes among the generators.

Original: Unlike these methods, AdaGAN introduces a sequential ”training+adding generators” approach.
Revised: Unlike these methods, AdaGAN introduces a sequential ”training+adding generators” approach.

Original: This approach is computationally inefficient as shown by Hoang et al.
Revised: This approach is computationally inefficient as shown by Hoang et al.

Original: Although the results suggest that these methods work well, they require adding an extra step in the training process.
Revised: Although the results suggest that these methods work well, they require adding an extra step in the training process.

Original: The method presented in this contribution manages to achieve promising results without the need of such additions.
Revised: The method presented in this contribution manages to achieve promising results without the need of such additions.

Original: 2. GAN EVALUATION METRICS
Revised: 2. GAN EVALUATION METRICS

Original: The most commonly used metrics in the GAN community to evaluate the generated images is the
Revised: The most commonly used metrics in the GAN community to evaluate the generated images is the

Original: Inception Score (IS) as introduced in Salimans et al. (2016).
Revised: Inception Score (IS) as introduced in Salimans et al. (2016).

Original: This method is an objective metric which evaluates two things: Image quality and image diversity.

Original: It uses the pre-trained InceptionV3 model (Szegedy et al., 2014) to predict the class probabilities. This is actually the conditional probability of class prediction given the image p ( y | x ) .
Revised: It uses the pre-trained InceptionV model (Szegedy et al., 2014) to predict the class probabilities, the conditional probability of class prediction given the image p ( y | x ) .

Original: The marginal distribution p ( y ) is calculated using the batch-wise average of p ( y | x ) .
Revised: The marginal distribution p ( y ) is calculated using the batch-wise average of p ( y | x ) .

Original: Finally the KL-Divergence is calculated using
Revised: Finally the KL-Divergence is calculated using

Original: [Equation]
Revised: [Equation]

Original: This is performed over multiple batches and the final score is reported as the mean and standard deviation of all the batches.
Revised: This is performed over multiple batches and the final score is reported as the mean and standard deviation of all the batches.

Original: Over the past few years, IS has started to be considered an unreliable metric for evaluation, especially for datasets that are not ImageNet (Barratt & Sharma, 2018).
Revised: Over the past few years, IS has started to be considered an unreliable metric for evaluation, especially for datasets that are not ImageNet (Barratt & Sharma, 2018).

Original: A better evaluation metric that has become more popular is the Frechet Inception Distance (FID) (Heusel et al., 2017).
Revised: A better evaluation metric that has become more popular is the Frechet Inception Distance (FID) (Heusel et al., 2017).

Original: This metric uses the InceptionV3 model without the final classification layer.
Revised: This metric uses the InceptionV3 model without the final classification layer.

Original: The latent distribution generated by the
Revised: The latent distribution generated by the

Original: InceptionV3 network for real and fake images is compared.
Revised: InceptionV3 network for real and fake images is compared.

Original: Specifically, the FID metric is given by
Revised: Specifically, the FID metric is given by

Original: [Equation]
Revised: [Equation]

Original: where µ 1 and µ 2 are means of the latent representation of fake and real data, respectively and C 1 and
Revised: where µ 1 and µ 2 are means of the latent representation of fake and real data, respectively and C 1 and

Original: C 2 are their covariance matrices.
Revised: C 2 are their covariance matrices.

Original: This metric was proposed as an improvement over the IS.
Revised: This metric was proposed as an improvement over the IS.

Original: However, it suffers from some of the same issues as the IS because of the fact that InceptionV3 is pre-trained on a 1000 class dataset of RGB images.
Revised: However, it suffers from some of the same issues as the IS because of the fact that InceptionV3 is pre-trained on a 1000 class dataset of RGB images.

Original: For our particular use case with MNIST (a grayscale dataset with only 10 classes), we used the tensorflow-gan library’s built in method mnist frechet distance function.
Revised: For our particular use case with MNIST (a grayscale dataset with only 10 classes), we used the tensorflow-gan library’s built in method mnist frechet distance function.

Original: This function uses a pre-trained MNIST classifier, rather than the less relevant InceptionV3, to calculate the
Revised: This function uses a pre-trained MNIST classifier, rather than the less relevant InceptionV3, to calculate the

Original: Frechet distance.
Revised: Frechet distance.

Original:  M ETHODS 3. D ATASET
Revised:  M ETHODS 3. D ATASET

Original: MNIST is a very well-known dataset.
Revised: MNIST is a very well-known dataset.

Original: It contains 60,000 images (train set) and 10,000 images (test set) of handwritten digits belonging to 10 classes (numbers 0-9) of 28x28x1 pixels per image.
Revised: It contains 60,000 images (train set) and 10,000 images (test set) of handwritten digits belonging to 10 classes (numbers 0-9) of 28x28x1 pixels per image.

Original: It is widely used for proof-of-concepts of various model architectures and training protocols related to image classification or generation.
Revised: It is widely used for proof-of-concepts of various model architectures and training protocols related to image classification or generation.

Original: This is because it is easily available through TensorFlow and
Revised: This is because it is easily available through TensorFlow and

Original: PyTorch, smaller images make it easier to implement on simplistic hardware and it has well-defined modes in the dataset.
Revised: PyTorch, smaller images make it easier to implement on simplistic hardware and it has well-defined modes in the dataset.

Original: This is the dataset we used for our experiments.
Revised: This is the dataset we used for our experiments.

Original: The train set was used during the training procedures and the test set was used to evaluate our models.
Revised: The train set was used during the training procedures and the test set was used to evaluate our models.

Original: We resized the images to 32x32 to maintain the dimensions as a power of 2, to maintain even dimensions while using strided convolutions.
Revised: We resized the images to 32x32 to maintain the dimensions as a power of 2, to maintain even dimensions while using strided convolutions.

Original: We also normalized pixel values to lie between -1 and 1, as this has been shown to perform better in WGAN-GP (Gulrajani et al., 2017).
Revised: We also normalized pixel values to lie between -1 and 1, as this has been shown to perform better in WGAN-GP (Gulrajani et al., 2017).

Original: 3. T HE TVD LOSS COMPONENT
Revised: 3. T HE TVD LOSS COMPONENT

Original: The Total Variation Distance (TVD) is a quantity that measures the distance between two categorical distributions and is defined as:
Revised: The Total Variation Distance (TVD) is a quantity that measures the distance between two categorical distributions and is defined as:

Original: [Equation]
Revised: [Equation]

Revised: Where P and Q are two categorical distribution vectors, X are the number of categories in the distribution and P ( x ) and Q ( x ) denote the probability of class x in the distribution P and Q respectively.

Original: We assumed each class in the MNIST dataset is a mode.
Revised: We assumed each class in the MNIST dataset is a mode.

Original: To enforce mode separation among individual generators, we augmented the generator loss in the original WGAN-GP paper (Gulrajani et al., 2017) with a TVD based component.
Revised: To enforce mode separation among individual generators, we augmented the generator loss in the original WGAN-GP paper (Gulrajani et al., 2017) with a TVD based component.

Original: We chose this as the preferred metric because it has well defined bounds ( TV D ( P, Q ) ∈ [0 , 1] ) and is easy to compute for categorical distributions.
Revised: We chose this as the preferred metric because it has well defined bounds ( TV D ( P, Q ) ∈ [0 , 1] ) and is easy to compute for categorical distributions.

Original: It is also commutative, that is TV D ( i, j ) =
Revised: It is also commutative, that is TV D ( i, j ) =

Original: TV D ( j, i ) .
Revised: TV D ( j, i ) .

Original: We used a pre-trained MNIST classifier ( C ) to obtain softmax scores for each generated batch ˜ x
Revised: Subsequently, we defined the distance metric: [Equation]

Original: i resulting in n categorical distributions.
Revised: which is just the average pairwise TVD.

Original: We needed the classifier to be simple and fast.
Revised: Here, C (˜ x i ) and C (˜ x j )) are categorical distributions generated by a pre-trained MNIST classifier (C) on ˜ x i and ˜ x j , the images generated by generator i and j respectively.

Original: So, we trained a very basic convolution neural network to perform this loss computation.
Revised: The classifier is a very basic convolution neural network.

Original: Refer to Appendix A for more information and convergence graphs.
Revised: Refer to Appendix A for more information on classifier training.

Original: We obtained a peak accuracy of 99% after training.
Revised: We obtained a peak accuracy of 99% after training.

Original: Subsequently, we defined the distance metric ∆ = 1 N(cid:80) ni =1 (cid:80) nj = i +1 TV D ( C (˜ x i ) , C (˜ x j )) , which is just the average pairwise TVD (where N is the number of unique pairs of categorical distributions using ’ n ’ generators).
Revised: N = n ( n − 1) / 2 is the number of unique pairs of categorical distributions using ’ n ’ generators.

Original: Finally, we introduced this term to the loss of each generator to get:
Revised: Finally, we introduced this term to the loss of each generator to get:

Original: [Equation]
Revised: [Equation]

Original: where the first term is the original generator loss from the WGAN-GP paper.
Revised: where the first term is the original generator loss from the WGAN-GP paper.

Original: The hyperparameter γ is the distance weight.
Revised: The hyperparameter γ is the distance weight.

Original: We carried out experiments to find the optimal γ which produces the best possible FID (refer to Section 4).
Revised: We carried out experiments to find the optimal γ which produces the best possible FID (refer to Section 4).

Original: 3. T HE TRAINING ALGORITHM
Revised: 3. T HE TRAINING ALGORITHM

Original: The training algorithm remained the same for all the experiments.
Revised: The training algorithm remained the same for all the experiments.

Original: However, in each experiment, the effect of each newly introduced parameter (namely, γ and n ) was tested.
Revised: However, in each experiment, the effect of each newly introduced parameter (namely, γ and n ) was tested.

Original: The specific values used for each experiment, are specified in Section 4.
Revised: The specific values used for each experiment, are specified in Section 4.

Original: All other hyperparameters are constant throughout all experiments.
Revised: All other hyperparameters are constant throughout all experiments.

Original: The batch size m has to be varied to maintain divisibility by n in step 4 of the algorithm.
Revised: The batch size m has to be varied to maintain divisibility by n in step 4 of the algorithm.

Original: However, we restricted this value to one of two possible values [500 , 504] , to maintain approximate similarity.
Revised: However, we restricted this value to one of two possible values [500 , 504] , to maintain approximate similarity.

Original: Specifically, for n = 1 , 2 , 4 , 5 , 10 , m = 500 and for n = 3 , 6 , 7 , 8 , 9 , m = 504 .
Revised: Specifically, for n = 1 , 2 , 4 , 5 , 10 , m = 500 and for n = 3 , 6 , 7 , 8 , 9 , m = 504 .

Original: We used the generator learning rate and discriminator learning rate α g = α d = 0 . 0002 .
Revised: We used the generator learning rate and discriminator learning rate α g = α d = 0 . 0002 .

Original: We chose the number of discriminator training iterations per generator iteration, n d = 3 , following the recommendations of
Revised: We chose the number of discriminator training iterations per generator iteration, n d = 3 , following the recommendations of

Original: Gulrajani et al.
Revised: Gulrajani et al.

Original: Also following their recommendations, we used the Adam optimizer, setting β 1 = 0 .
Revised: Also following their recommendations, we used the Adam optimizer, setting β 1 = 0 .

Original: 5 and β 2 = 0 .
Revised: 5 and β 2 = 0 .

Original: 9 for all our experiments.
Revised: 9 for all our experiments.

Original: The ◦ symbol denotes concatenation along batch dimension.
Revised: The ◦ symbol denotes concatenation along batch dimension.

Original: λ
Revised: λ

Original: = 10 is the gradient penalty weight as described in Eq.
Revised: = 10 is the gradient penalty weight as described in Eq.

Original: equation 3.
Revised: equation 3.

Original: The algorithm pseudocode can be found in Algorithm 1 below.
Revised: The algorithm pseudocode can be found in Algorithm 1 below.

Original: Algorithm1 Training algorithm
Revised: Algorithm

Original: Require: β 1 , β 2 , α , the Adam parameters, γ the distance weight, n the number of generators, n d the number of discriminator iterations per generator iteration, λ the GP coefficient and m the batch size, α g , α d , the learning rates and a pre-trained classifier (C).
Revised: Require: β 1 , β 2 , α , the Adam parameters; γ the distance weight; n the number of generators; n d the number of discriminator iterations per generator iteration; λ the GP coefficient; m the batch size; α g , α d , the learning rates; a pre-trained classifier (C).

Original: Require: initial discriminator parameters w 0 , initial generator parameters θ 0 ,i ∀ i = 1 , 2 , ..., n 1: while θ i have not converged do 2: for t = 1 , ..., n d do 3:
Revised: Require: initial discriminator parameters w 0 , initial generator parameters θ 0 ,i ∀ i = 1 , 2 , ..., n 1: while θ i have not converged do 2: for t = 1 , ..., n d do 3:

Original: Sample real data batch x ∼ P r , latent variable batch
Revised: Sample real data batch x ∼ P r , latent variable batch

Original: z ∼ p
Revised: z ∼ p

Original: ( z ) and ϵ ∼ U [0 , 1] .
Revised: ( z ) and ϵ ∼ U [0 , 1] .

Original: Split the latent vector batch into n sub-batches ( z 1 , ..., z n ) 5: ˜ x i ← G i ( z i ) ∀ i = 1 , ..., n 6: ˜ x ← ˜ x 1 ◦ ˜ x 2 ◦ ... ◦
Revised: Split the latent vector batch into n sub-batches ( z 1 , ..., z n ) 5: ˜ x i ← G i ( z i ) ∀ i = 1 , ..., n 6: ˜ x ← ˜ x 1 ◦ ˜ x 2 ◦ ... ◦

Original: ˜ x n 7: ˆ x ← ϵ x + (1 − ϵ )˜ x 8:
Revised: ˜ x n 7: ˆ x ← ϵ x + (1 − ϵ )˜ x 8:

Original: L d ← 1 m (cid:80) mi =

Revised: L d ← 1 m (cid:80) mi =

Original: [ D w (˜ x ) − D w ( x ) + λ ( ||∇ ˆ x D (ˆ x ) || 2 − 1) 2 ] 9: w ← Adam ( ∇ w L d , w, α, β 1 , β 2 ) 10: end for 11: sample n latent variable batches
Revised: [ D w (˜ x ) − D w ( x ) + λ ( ||∇ ˆ x D (ˆ x ) || 2 − 1) 2 ] 9: w ← Adam ( ∇ w L d , w, α, β 1 , β 2 ) 10: end for 11: sample n latent variable batches

Original: z i ∼ p ( z )  12: ˜ x i ← G i ( z i ) ∀ i = 1 , ..., n 13: if γ ̸ = 0 then 14: ∆ ← 1 N (cid:80) ni
Revised: z i ∼ p ( z ) ∀ i = 1 , 2 , ..., n 12: ˜ x i ← G i ( z i ) ∀ i = 1 , ..., n 13: if γ ̸ = 0 then 14: ∆ ← 1 N (cid:80) ni

Original: =1 (cid:80) nj
Revised: =1 (cid:80) nj

Original: =
Revised: =

Original: i +1 TV D
Revised: i +1 TV D

Original: ( C (˜ x i ) , C (˜ x j )) 15:
Revised: ( C (˜ x i ) , C (˜ x j )) 15:

Original: L g,i ← 1 m (cid:80) mk =1 [ − D
Revised: L g,i ← 1 m (cid:80) mk =1 [ − D

Original:  ( G i ( z )) − γ ∆] 16: else 17:
Revised: (˜ x ( k ) i ) + γ (1 − ∆)] ∀ i = 1 , 2 , ..., n 16: else 17:

Original: L g,i ← 1 m (cid:80) m k =1 − D  ( G i ( z )) 18: end if 19: θ i ← Adam ( ∇ θ i L g,i , θ i , α, β 1 , β 2 )  20: end while
Revised: L g,i ← 1 m (cid:80) mk =1 − D (˜ x ( k ) i ) ∀ i = 1 , 2 , ..., n 18: end if 19: θ i ← Adam ( ∇ θ i L g,i , θ i , α, β 1 , β 2 ) ∀ i = 1 , 2 , ..., n 20: end while

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: For all the experiments, we used the same model architectures for all the generators and the discriminators, the details of which can be found in Appendix B.
Revised: For all the experiments below, we used the same model architectures for all the generators and the discriminators, the details of which can be found in Appendix B.

Original: We performed 3 broad categories of experiments -
Revised: We performed 3 broad categories of experiments -

Original: Baseline experiments as a proof of concept of our new method.
Revised: Baseline experiments as a proof of concept of our new method.

Original: Optimizing for the hyperparameter γ .
Revised: Optimizing for the hyperparameter γ .

Original: Experiments comparing number of generators versus performance (in terms of FID).
Revised: Experiments comparing number of generators versus performance (in terms of FID).

Original: In the baseline experiments, we ran 2 tests, with 2 generators each.
Revised: In the baseline experiments, we ran two types of tests:

Original:  In the first test, we set the γ = 0 . This test is termed as batch concatenation , because it is equivalent to the followingEach Generator ( G i ) generates batch size / n images, during each iteration of Discriminator ( D ) training, where n is the number of generators.
Revised: • Model size experiments • TVD loss experiments In the first type, we ran the original WGAN-GP procedure, with bigger generators and compared performance with 4 small generators using our method.

Original: These individual generations ( ˜ x i ) are of size ( batch size / n , 32 , 32 , 1) . We concatenate them along the batch dimension to generate ˜ x , i.e.:
Revised: We used two different ways to make the generators bigger - using dense layers to increase depth and using residual blocks. When using 4 small generators, we used γ = 12 . 5 .

Original: [Equation]
Revised: We kept the discriminator the same throughout these experiments. All three models were trained for 70 epochs.

Original: This ˜ x is then used in the WGAN-GP objective function equation 3. In the second test, we set γ = 5 . 0 , an arbitrary value to see the impact of the new loss component on convergence.
Revised: In the second type, we ran two experiments, one with γ =(termed as batch concatenation ) and one with γ = 5 . 0 , an arbitrary value to see the impact of the new loss component on convergence.

Original: Here we trained all models until convergence was reached.
Revised: Here we trained all models until convergence was reached.

Original: In the second category, we fixed the number of generators n = 5 and the number of epochs = 70 .
Revised: In the second category, we fixed the number of generators n = 5 and the number of epochs = 70 .

Original: We carried out 3 tests:
Revised: We carried out 3 tests:

Original: • Constant γ throughout training at discrete values [0 , 5 , 10 , 11 , 12 , 13 , 14 , 15 , 20] .
Revised: • Constant γ throughout training at discrete values [0 , 5 , 10 , 11 , 12 , 13 , 14 , 15 , 20] .

Original: • Linear decay from γ = 20 to γ = 5 , so that the effective weight is equal to the average of the two ( γ eff = 12 . 5 ), which we found to be close to the optimal for the setup.
Revised: • Linear decay from γ = 20 to γ = 5 , so that the effective weight is equal to the average of the two ( γ eff = 12 . 5 ), which we found to be close to the optimal for the setup.

Original: • Sudden death, where γ = 20 for the first 20 epochs of training and then it is equal to 0 for the remainder.
Revised: • Sudden death, where γ = 20 for the first 20 epochs of training and then it is equal to 0 for the remainder

Original: In the final category, we swept the number of generators n = [1 , 2 , ..., 10] to see how it affects the performance.
Revised: In the final category, we swept the number of generators n = [1 , 2 , ..., 10] to see how it affects the performance.

Original: In this category of experiments, the constant parameter γ = 12 . 2 was chosen after the optimizations performed in the second category of experiments.
Revised: In this category of experiments, the constant parameter γ = 12 . 2 was chosen after the optimizations performed in the second category of experiments.

Original: Here, we trained all models until convergence was reached.
Revised: Here, we trained all models until convergence was reached.

Original:  R ESULTS AND DISCUSSION
Revised:  R ESULTS AND DISCUSSION

Original: If we run our experiments, with n = 1 , we can also evaluate the state-of-the-art FID for a vanilla
Revised: If we run our experiments, with n = 1 , we can also evaluate the state-of-the-art FID for a vanilla

Original: WGAN-GP.
Revised: WGAN-GP.

Original: We performed a session of training under these conditions and as can be observed in the
Revised: We performed a session of training under these conditions and as can be observed in the

Original: Fig.
Revised: Fig.

Original: 4, the current state-of-the-art WGAN-GP gives us an FID ∼ 4 . 25 .
Revised: 4, the current state-of-the-art WGAN-GP gives us an FID ∼ 4 . 25 .

Original: In all the future discussions, we will compare the obtained values to this one.
Revised: In all the future discussions, we will compare the obtained values to this one.

Original: 5. B ASELINE EXPERIMENTS 
Revised: 5. B ASELINE EXPERIMENTS 5.1. M ODEL SIZE EXPERIMENTS

Revised: In all experiments the discriminator was identical and had 4 , 306 , 177 parameters.

Revised: In the first experiment, we trained 4 generators using the TVD loss method ( γ = 12 . 5 ).

Revised: Each generator model was the same as presented in B and had 910 , 660 parameters for a total of 3 , 642 , 640 parameters.

Original: We ran both of the two generators experiments until convergence 3 times, to test robustness.
Revised: Running these 4 small generators resulted in a final average FID = 2 . 6 . In the second experiment, we added an extra dense layer to the generator model, resulting in 69 , 657 , 924 parameters ( 76. times the original generator). After training using the WGAN-GP protocol, it converged to an average

Revised: FID = 5 . 56 .

Revised: In the third experiment, we replaced the generator with a much deeper network that uses residual blocks.

Revised: The total number of parameters in the generator is 3 , 882 , 049 ( 4.3 times the original generator).

Revised: This model did not converge because the discriminator was easily overpowered.

Revised: The average FID = 23 . 42 .

Revised: This model converges if we add more parameters to the discriminator.

Revised: The models used in the latter two experiments are also available in B. From the above results, it is clear that two or more competing generators exceed the performance of bigger and more complex ones.

Revised: Our method can be efficiently parallelized due to the architecture.

Revised: . TVD LOSS EXPERIMENTS

Revised: We ran both of the two generators experiments until convergence 3 times, to test robustness.

Original: The convergence graphs for one training session are presented in Fig.
Revised: The convergence graphs for one training session are presented in Fig.

Original: Fig.
Revised: Fig.

Original: 1b shows the convergence graph for batch concatenation (b.c.)
Revised: 1b shows the convergence graph for batch concatenation (b.c.)

Original: + TVD and Fig.
Revised: + TVD and Fig.

Original: 1a shows just batch concatenation (b.c.)
Revised: 1a shows just batch concatenation (b.c.)

Original: based generator loss.
Revised: based generator loss.

Original: We see from the graphs that both of the generators are converging at a similar pace (the lines overlap with each other), under both loss functions.
Revised: We see from the graphs that both of the generators are converging at a similar pace (the lines overlap with each other), under both loss functions.

Original: The discriminator loss going toindicates that it predicts negative and positive values with almost equal probability (meaning the system has converged).
Revised: The discriminator loss going toindicates that it predicts negative and positive values with almost equal probability (meaning the system has converged).

Original: The network managed to converge all 3 times for both loss functions ( γ =and γ = 5 ).
Revised: The network managed to converge all 3 times for both loss functions ( γ =and γ = 5 ).

Revised: [Figure]

Original: After convergence, we measured 3 metrics : FID, calculated according to Eq.
Revised: After convergence, we measured 3 metrics : FID, calculated according to Eq.

Original: equation 5;
Revised: equation 5;

Original: TV D ( C ( G 1 ( z 1 )) , C ( G 2 ( z 2 ))) and; average class probabilities (softmax) for each generators’ generations, as predicted by C. These metrics are presented in Tables 1 and 2. 
Revised: TV D ( C ( G 1 ( z 1 )) , C ( G 2 ( z 2 ))) and; average class probabilities (softmax) for each generators’ generations, as predicted by C. These metrics are presented in Tables 1 and 2. [Table]

Original: [Table]
Revised: [Table]

Original: We see in Table 2 that the two generators specialize in generating disjoint subsets of classes.
Revised: We see in Table 2 that the two generators specialize in generating disjoint subsets of classes.

Original: The actual samples generated by each generator can be seen in Fig.2.
Revised: The actual samples generated by each generator can be seen in Fig.2.

Original: We see that they are indeed from separate classes for different generators and of very high quality.
Revised: We see that they are indeed from separate classes for different generators and of very high quality.

Original: We can also observe (Table 1) how introducing a small non-zero constant γ = 5 drastically improves the FID as well as the TVD.
Revised: We can also observe (Table 1) how introducing a small non-zero constant γ = 5 drastically improves the FID as well as the TVD.

Original: [Figure] [Table] 
Revised: [Table] [Figure]

Original: 5. TVD LOSS WEIGHT EXPERIMENTS 5.2. D ISCRETE CONSTANT
Revised: 5. TVD LOSS WEIGHT EXPERIMENTS 5.2. D ISCRETE CONSTANT

Original: In this experiment we swept gamma across the following values - [0 , 5 , 10 , 11 , 12 , 13 , 14 , 15 , 20] and kept it constant throughout the 70 epochs of training.
Revised: In this experiment we swept gamma across the following values - [0 , 5 , 10 , 11 , 12 , 13 , 14 , 15 , 20] and kept it constant throughout the 70 epochs of training.

Original: We fixed the number of generators to 5.
Revised: We fixed the number of generators to 5.

Original: We evaluated each time using the FID metric.
Revised: We evaluated each time using the FID metric.

Original: The plot of the parametric sweep is seen in Figure 3.
Revised: The plot of the parametric sweep is seen in Figure 3.

Original: We can see that there is an optimum value for the TVD loss weight (approximately γ = 12 . 2 ).
Revised: We can see that there is an optimum value for the TVD loss weight (approximately γ = 12 . 2 ).

Original: 5.2. L INEAR DECAY
Revised: 5.2. L INEAR DECAY

Original: In this experiment [Table] we again fixed the number of generators to 5 and the number of epochs to 70.
Revised: In this experiment, we again fixed the number of generators to 5 and the number of epochs to 70.

Original: We linearly decayed the parameter γ from 20 to 5, giving us an effective γ eff = 12 .
Revised: We linearly decayed the parameter γ from 20 to 5, giving us an effective γ eff = 12 .

Original: We evaluated the FID and TVD after training is complete and received FID = 3 . 5379 and TV D = 0 . 

Revised: We evaluated the FID and TVD after training is complete and received FID = 3 . 5379 and TV D = 0 .

Original: .This method clearly gives an improvement over the state-of-the-art FID as well as optimal mode separation.
Revised: .This method clearly gives an improvement over the state-of-the-art FID as well as optimal mode separation.

Original: 5.2. S UDDEN DEATH
Revised: 5.2. S UDDEN DEATH

Original: Keeping the other elements of the setup exactly the same, we now keep γ = 20 for the firstepochs of training and then γ = 0 for the remaining 50.
Revised: Keeping the other elements of the setup exactly the same, we now keep γ = 20 for the firstepochs of training and then γ = 0 for the remaining 50.

Original: We end up with FID = 2 .
Revised: We end up with FID = 2 .

Original: 6685 and
Revised: 6685 and

Original: TV D = 0 .
Revised: TV D = 0 .

Original: 9998 , which again is an improvement over the state-of-the-art FID along with great mode separation.
Revised: 9998 , which again is an improvement over the state-of-the-art FID along with great mode separation.

Original: Thisresultis interesting as it suggests that one does not need to enforce mode separation (or compute the pairwise TVD) for all of the training epochs.
Revised: This result is interesting as it suggests that one does not need to enforce mode separation (or compute the pairwise TVD) for all of the training epochs.

Original: Once the modes have sufficiently separated, we can remove the TVD loss component and the generators will keep generating separate modes and in a much better quality than current state-of-the-art.
Revised: Once the modes have sufficiently separated, we can remove the TVD loss component and the generators will keep generating separate modes and in a much better quality than current state-of-the-art.

Original: [Figure]

Original: 5. N UMBER OF GENERATORS SWEEP
Revised: 5. N UMBER OF GENERATORS SWEEP

Original: From the above experiments we found an optimal value for γ = 12 .
Revised: From the above experiments we found an optimal value for γ = 12 .

Original: 2 which we fixed and kept constant for this experiment.
Revised: 2 which we fixed and kept constant for this experiment.

Original: We changed n from 1 to 10 (1 representing current state-of-the-art).
Revised: We changed n from 1 to 10 (1 representing current state-of-the-art).

Original: We trained each setup until convergence or 250 epochs, whichever came first.
Revised: We trained each setup until convergence or 250 epochs, whichever came first.

Original: We set the early stopping criterion as: less than 0.01 change in the objective for 5 consecutive epochs .
Revised: We set the early stopping criterion as: less than 0.01 change in the objective for 5 consecutive epochs .

Original: We decided to do this because when we tried running for a fixed number of epochs, we realized that for n = 3 , 6 , 7 , 9 the generators did not converge in the same number of epochs.
Revised: We decided to do this because when we tried running for a fixed number of epochs, we realized that for n = 3 , 6 , 7 , 9 the generators did not converge in the same number of epochs.

Original: Here we notice that the mode separation problem is much more difficult because of the non-divisibility of 10 (the number of modes) by n, which results in one or more modes of the dataset being assigned to a different generator during different training epochs.
Revised: Here we notice that the mode separation problem is much more difficult because of the non-divisibility of 10 (the number of modes) by n, which results in one or more modes of the dataset being assigned to a different generator during different training epochs.

Original: As we can also see in Fig.
Revised: As we can also see in Fig.

Original: 4, the FID values for these specific values of n are much higher, denoting a much worse performance.
Revised: 4, the FID values for these specific values of n are much higher, denoting a much worse performance.

Original: The FID for n = 4 and n = 5 is the lowest, showing us that having more generators does not imply better performance and this is another parameter that can be optimized.
Revised: The FID for n = 4 and n = 5 is the lowest, showing us that having more generators does not imply better performance and this is another parameter that can be optimized.

Original:  C ONCLUSIONS
Revised:  C ONCLUSIONS

Original: We introduced a novel approach in training WGANs in a collaborative manner which we call CoGANs.
Revised: We introduced a novel approach in training WGANs in a collaborative manner which we call ComMutE-GANs.

Original: This approach can be used to model complex, multi-modal datasets in a better way.
Revised: This approach is inspired by the principles of co-creativity.

Original: We showed that using our approach outperforms state-of-the-art GANs on the MNIST dataset in terms of FID and manages to improve overall stability, robustness and performance of GANs.
Revised: Our approach gives improved generation performance under space constraints as we saw in section 5.1.1. This method also manages to improve overall stability, robustness and performance of GANs.

Original: We also introduced a new loss component that can be added to the generator loss during training.
Revised: We also introduced a new loss component that can be added to the generator loss during training.

Original: We observed that even adding this loss for a short amount of time in the initial training can drastically improve performance.
Revised: We observed that even adding this loss for a short amount of time in the initial training can drastically improve performance.

Original: We can also tune the weight of this loss to fit the number of generators we plan to use.
Revised: We can also tune the weight of this loss to fit the number of generators we plan to use.

Original: A major disadvantage of the current method is the fact that we need a pre-trained classifier to calculate a loss during training, which is not efficient.
Revised: A major disadvantage of the current method is the fact that we need a pre-trained classifier to calculate a loss during training, which is not efficient.

Original: Majority of the future work will involve finding better losses to enforce mode separation without the use of a classifier.
Revised: Majority of the future work will involve finding better losses to enforce mode separation without the use of a classifier.

Original: The eventual goal of this research is to use multiple simple generator models to generate multi-modal data rather than a single large, complex network.
Revised: The eventual goal of this research is to use multiple simple generator models to generate multi-modal data rather than a single large, complex network.

Original: This research is a also contribution towards modelling co-creativity in humans.
Revised: This research is a also contribution towards modelling co-creativity in humans.

Original: Study of this model can help us better understand and implement creativity promoting interactions.
Revised: Study of this model can help us better understand and implement creativity promoting interactions

Original: A A PPENDIX - C LASSIFIER TRAINING
Revised: A A PPENDIX - C LASSIFIER TRAINING

Original: We present specific information about the classifier in this section.
Revised: We present specific information about the classifier in this section.

Original: The network architecture Fig.and convergence graphs Fig. 6 are presented below.
Revised: The network architecture Fig.and convergence graphs Fig. 6 are presented below.

Original: Each conv block contains a convolutional layer with a 5x5 filter and number of filters going from 64 to 512 in powers of 2 .
Revised: Each conv block contains a convolutional layer with a 5x5 filter and number of filters going from 64 to 512 in powers of 2 .

Original: Each convolutional layer is followed by BatchNormalization , ReLU and Dropout with drop prob = 0 .
Revised: Each convolutional layer is followed by BatchNormalization , ReLU and Dropout with drop prob = 0 .

Original: The MLP block contains 2 layers of 128 neurons each followed by a 10 neuron output layer.
Revised: The MLP block contains 2 layers of 128 neurons each followed by a 10 neuron output layer.

Original: As observed in 6b, the peak accuracy of our classifier was ∼ 99% .
Revised: As observed in 6b, the peak accuracy of our classifier was ∼ 99% .

Original: B A PPENDIX - G ENERATOR AND DISCRIMINATOR MODELS
Revised: B A PPENDIX - G ENERATOR AND DISCRIMINATOR MODELS

Original: We implemented very simple Generator and Discriminator .
Revised: We implemented very simple Generator and Discriminator .

Original: The exact models are presented in Fig.
Revised: The exact models are presented in Fig.

Original: 7a and Fig.
Revised: 7a and Fig.

Original: 7b.
Revised: 7b.

Original: Each conv block is exactly the same as the one implemented for the Classifier , except the use of Layer normalization instead of Batch normalization , as recommended in Arjovsky et al.
Revised: Each conv block is exactly the same as the one implemented for the Classifier , except the use of Layer normalization instead of Batch normalization , as recommended in Arjovsky et al.

Original: Each upsample block consists of a 2D Upsampling layer with a factor of 2x followed by a convolutional layer with filter size 3x3 and stride 1x
Revised: Each upsample block consists of a 2D Upsampling layer with a factor of 2x followed by a convolutional layer with filter size 3x3 and stride 1x1.

Original: These operations are followed by LeakyReLU activation with α = 
Revised: These operations are followed by LeakyReLU activation with α = 0 .

Original: Batch normalization is applied after the activation final upsample block also uses dropout with drop prob =  final upsample block uses the tanh activation function to squash the values 
Revised: Batch normalization is applied after the activation. The final upsample block also uses dropout with drop prob = 0 . The final upsample block uses the tanh activation function to squash the values between -1 and 1.

Revised: In the model size experiments (Section 5.1.1) we extend the generator model by one extra dense layer of size 16 , 384 to the structure above.

Revised: The final model can be seen in fig 8.

Revised: We also replaced the above mentioned generator with a generator that uses residual blocks in the generation.

Revised: The full architecture can be seen in fig 9.

Original: [Figure]
Revised: [Figure]

Original: [Figure]
Revised: [Figure]

Original: C A PPENDIX - O THER EXPERIMENTS
Revised: C A PPENDIX - O THER EXPERIMENTS

Original: Setting γ = 0 and n = 4 , we ran the setup until convergence.
Revised: Setting γ = 0 and n = 4 , we ran the setup until convergence.

Original: This method is an extended version of the batch concatenation (b.c.) method as described in 4, to four generators instead of two.
Revised: This method is an extended version of the batch concatenation (b.c.) method as described in 4, to four generators instead of two.

Original: The convergence graphs show that all 4 generators converge at the same pace (the lines overlap) and the discriminator loss converges to 0 (Fig. 
Revised: The convergence graphs show that all 4 generators converge at the same pace (the lines overlap) and the discriminator loss converges to 0 (Fig. 10).

Original: Our method improves the state-of-the-art with an
Revised: Our method improves the state-of-the-art with an

Original: FID = 2 .
Revised: FID = 2 .

Original: 9439 implying that high quality images are generated that match the dataset distribution.
Revised: 9439 implying that high quality images are generated that match the dataset distribution.

Original: Table 3 and Fig.
Revised: Table 3 and Fig.

Original: 9 show the average class probabilities and correspondingly generated samples.
Revised: 11 show the average class probabilities and correspondingly generated samples.

Original: This method of training results in an average pairwise TV D = 0 .
Revised: This method of training results in an average pairwise TV D = 0 .

Original: In Table 3, the values in bold represent which generator gives the maximum probability over that class.
Revised: In Table 3, the values in bold represent which generator gives the maximum probability over that class.

Original: The number 2 is generated by generator 1, 2 and 3 and the ”workload” required for this digit is being distributed among the three of them.
Revised: The number 2 is generated by generator 1, 2 and 3 and the ”workload” required for this digit is being distributed among the three of them.

Original: We can see that for γ = 0 , the TVD metric goes down as expected, but we can still obtain a comparable FID to what we can achieve using the TVD loss component.[Figure] [Table] [Table]
Revised: We can see that for γ = 0 , the TVD metric goes down as expected, but we can still obtain a comparable FID to what we can achieve using the TVD loss component.

Original: [Table]
Revised: [Table]

Original: [Table] [Table] [Table] [Table]
Revised: generator00.

Original: [Table] [Table] [Table] [Table]
Revised: generator38. generator00. generator00.

Original: [Table] [Table] [Table] [Table]
Revised: [Figure]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]