Original: MCL-GAN: Generative Adversarial Networks with Multiple Specialized Discriminators
Revised: MCL-GAN: Generative Adversarial Networks with Multiple Specialized Discriminators

Original: Anonymous Author(s)

Original: AfﬁliationAddress email

Original: Abstract
Revised: Abstract

Original: We propose a  generative adversarial network with multiple discriminators, whichcollaborate to represent a real dataset more effectively.
Revised: We propose a framework of generative adversarial networks with multiple discriminators, which collaborate to represent a real dataset more effectively.

Original: This approach facilitateslearning a generator consistent with the underlying data distribution based on realimages and thus mitigates the chronic mode collapse problem.
Revised: Our approach facilitates learning a generator consistent with the underlying data distribution based on real images and thus mitigates the chronic mode collapse problem.

Original: From the inspirationof multiple choice learning, we guide each discriminator to have expertise in thesubset of the entire data and allow the generator to ﬁnd reasonable correspondencesbetween the latent and real data spaces automatically withoutthe extra supervisionfor training examples.
Revised: From the inspiration of multiple choice learning, we guide each discriminator to have expertise in a subset of the entire data and allow the generator to ﬁnd reasonable correspondences between the latent and real data spaces automatically without extra supervision for training examples.

Original: Despite the use of multiple discriminators, the backbonenetworks are shared across the discriminators and the increase of training costis marginal.
Revised: Despite the use of multiple discriminators, the backbone networks are shared across the discriminators and the increase in training cost is marginal.

Original: We demonstrate the effectiveness of our algorithm using multipleevaluation metrics in the standard datasets for diverse tasks.
Revised: We demonstrate the effectiveness of our algorithm using multiple evaluation metrics in the standard datasets for diverse tasks.

Original:  Introduction
Revised:  Introduction

Original: Generative models learn to represent a probability distribution of data. With recent advances ofdeep generative models, Generative Adversarial Networks (GANs) [1] and Variational Autoencoders(VAEs) [2] have shown impressive achievements inunconditional generation of high-dimensionalrealistic images as well as various conditional generation tasks including image-to-image translation [3–5], image inpainting [6], image super-resolution [7], etc .
Revised: With recent advances in deep generative models, Generative Adversarial Networks (GANs) [1] and Variational Autoencoders (VAEs) [2] have shown impressive achievements in generation of high-dimensional realistic images as well as various computer vision applications including image-toimage translation [3–5], image inpainting [6], image super-resolution [7], etc .

Original: GANs have  received a lot of attention due to their interesting framework of minimax games, wheretwo agents, a generator and a discriminator, compete against each other.
Revised: GANs have particularly received a lot of attention due to their interesting framework of minimax games, where two agents, a generator and a discriminator, compete against each other.

Original: Speciﬁcally, a discriminatordistinguishes whether a sample comes from the real dataset or the generator while the generatorattempts to deceive the discriminator.
Revised: Furthermore, compared to VAEs, GANs tend to produce acute, high-quality images.

Original: In theory, the generator learns the real data distribution byreaching an equilibrium point of the minimax game.
Revised: In theory, the generator learns the real data distribution by reaching an equilibrium point of the minimax game.

Original: It is known that GANs produce acute, highquality images compared to VAEs.

Original: However, in practice, the alternating training procedure does notguarantee the convergence to the optimal solution and often experiences mode collapsing, failing tocover the multiple modes of real data or, even worse, reachingat trivial solutions.
Revised: However, in practice, the alternating training procedure does not guarantee the convergence to the optimal solution and often experiences mode collapsing, failing to cover the multiple modes of real data or, even worse, reaching trivial solutions.

Original: This paper focuses on the mode collapse problem in training GANs by adopting multiple collaboratingdiscriminators.
Revised: This paper focuses on the mode collapse problem in training GANs by adopting multiple collaborating discriminators.

Original: Each discriminator is learned to specialize in a subset of reference data space, whichis identiﬁed automatically via the training procedure, so the ensemble of discriminators provide notonly the differentiation of fake data, but also more accurate predictions over the clusters of real data.
Revised: Each discriminator learns to be specialized in a subset of reference data space, which is identiﬁed automatically during training, so the ensemble of discriminators provides not only the differentiation of fake data but also more accurate predictions over the clusters of real data.

Original: In this respect, a generator is encouraged to produce diverse modes that deceive a set of discriminators.
Revised: In this respect, a generator is encouraged to produce diverse modes that deceive a set of discriminators.

Original: We employ  Multiple Choice Learning (MCL)  to learn multiple discriminators that are trained ona subset of training data. The generatoris updated via a set of expert models, each of which isassociated with a subset of the true and generated examples closest to the expert.
Revised: To achieve these goals, we employ a Multiple Choice Learning (MCL) framework to learn multiple discriminators and update the generator via a set of expert discriminators, where each discriminator is associated with a subset of the true and generated examples.

Original: Our approach basedon a single generator and multiple discriminators is called MCL-GAN, which is optimized by thestandard objective of GAN combined with the objective for MCL in the discriminator side.
Revised: Our approach, based on a single generator and multiple discriminators, is called MCL-GAN, which is optimized by the standard objective of GANs combined with the objective of MCL on the discriminator side.

Original: There are several GAN literatures that employ multiple discriminators [8–13].
Revised: There are several GAN literature that adopts multiple discriminators [8–13].

Original: Among them, GMAN [9] is closely related with our approach in the sense that it utilizes the ensemble predictionof discriminators.
Revised: Among them, GMAN [9] is closely related to our approach in the sense that it utilizes an ensemble prediction of discriminators.

Original: It explores multi-discriminator extensions of GANs with diverse versions of theaggregated prediction of discriminators—from a harsh trainer to a lenient teacher with a softenedcriteria.
Revised: It explores multi-discriminator extensions of GANs with diverse versions of the aggregated prediction of discriminators—from a harsh trainer to a lenient teacher with softened criteria.

Original: Meanwhile, there exist signiﬁcant differences in the method of ensembling from our approach.
Revised: Meanwhile, there exist signiﬁcant differences in the method of ensembling from our approach.

Original: While GMAN focuses on the loss to the generator with parallel learning of discriminators,our strategy specializes  each discriminator for more informative feedbacks to the generator.
Revised: While GMAN focuses on the loss to the generator with parallel learning of discriminators, our strategy specializes in each discriminator for more informative feedback to the generator.

Original: The training algorithm of the proposed method is inspired by Multiple Choice Learning [14], whichis known to be effective in learning specialized models with high oracle accuracy in recognition tasks.
Revised: The training algorithm of the proposed approach is inspired by MCL [14], which is known to be effective in learning specialized models with high oracle accuracy in recognition tasks.

Original: Encouraged by this beneﬁt, [15–18] apply MCL or its variations [19, 20] to produce diverse andaccurate outputs in several applications.
Revised: Encouraged by this beneﬁt, [15–18] apply MCL or its variations [19,20] to the production of diverse and accurate outputs in several applications.

Original: For instance, Mun et al .
Revised: For instance, Mun et al .

Original: [16] propose  MCL-KD frameworkto come up with the visual question answering (VQA) systems based on multiple models that arespecialized in different types of visual reasonings.
Revised: [16] propose an MCL-KD framework to come up with a visual question answering (VQA) system based on multiple models that are specialized in different types of visual reasoning.

Original: DiverseNet [17] introduces the control parameteras an input that diversiﬁes the outputs of networks with an MCL loss.
Revised: DiverseNet [17] introduces a control parameter as an input that diversiﬁes the outputs of the network with an MCL loss.

Original: While these works generatemultiple outputs explicitly and select one of predictions or take their ensemble at inference time, ourapproach adopts a unique strategy for diversifying the mode with no additional cost at inference time.
Revised: While these works generate multiple outputs explicitly and select one of them or take their ensemble at inference time, our approach adopts a unique strategy for diversifying the mode with no additional cost at inference time.

Original: The proposed method takes an advantage of MCLtechniques for GANs, which has hardly beenexplored before.
Revised: The proposed method takes an advantage of MCL for GANs, which has hardly been explored before.

Original: No supervision  such as class labels or other conditions are assumed unlike theaforementioned works.
Revised: Our approach does not require any supervision for model specialization, such as class labels and other conditions, unlike the aforementioned works.

Original: Our main contributions are summarized as follows:
Revised: Our main contributions are summarized as follows:

Original: • We propose a single-generator multi-discriminator GAN training algorithm to alleviate the mode collapse problem.
Revised: • We propose a single-generator multi-discriminator GAN training algorithm to alleviate the mode collapse problem.

Original: Our approach providesthe simple yet effective learning objectives based on MCL to achieve the goal.
Revised: Our approach provides simple yet effective learning objectives based on MCL to achieve the goal.

Original: • We present a balanced discriminator assignment strategy to facilitate the robust convergence of models and preserve the multi-modality of training data, where the number of the
Revised: • We present a balanced discriminator assignment strategy to facilitate the robust convergence of models and preserve the multi-modality of training data, where the number of discriminators is determined adaptively.

Original: discriminators is determined adaptively.

Original: • The proposed method is applicable to many GAN variants since there is no constraint onnetwork architectures or loss functions.
Revised: • The proposed method is applicable to many GAN variants since there is no constraint on network architectures or loss functions.

Original: Our method requiresa small extra overhead andtrains the model with computational efﬁciency via feature sharing in the discriminators.
Revised: Our method requires small extra overhead and trains models efﬁciently via feature sharing in the discriminators.

Original:  Related work
Revised:  Related work

Original: This section discusses the mode collapse issue in GANs and the GAN models with multiple discriminators or generators.
Revised: This section discusses the mode collapse issue in GANs and the GAN models with multiple discriminators or generators.

Original: 2.1 Handling mode collapse for diversity
Revised: 2.1 Handling mode collapse for diversity

Original: Many variations of GANs propose either novel metrics for the discriminator loss or better alternativesof the discriminator design.
Revised: Many variations of GANs propose either novel metrics for the discriminator loss or better alternatives to the discriminator design.

Original: For example, LSGAN [21] substitutes the least square function for thebinary cross-entropy function as the discriminator loss.
Revised: For example, LSGAN [21] substitutes the least square function for the binary cross-entropy loss as the discriminator objective.

Original: WGAN [22] introduces a critic function basedon the Earth-Mover’s distance rather than a binary classiﬁer, and WGAN-GP [23] improves WGANby adding a gradient penalty term.
Revised: WGAN [22] introduces a critic function based on the Earth-Mover’s distance rather than a binary classiﬁer, and WGAN-GP [23] improves WGAN by adding a gradient penalty term.

Original: PacGAN [24] augments the discriminator’s input by packingsamples for a single label.
Revised: PacGAN [24] augments the discriminator’s input by packing samples for a single label.

Original: EBGAN [25] models the discriminator as an energy function, which isimplemented by the reconstruction loss of the autoencoder.
Revised: EBGAN [25] models the discriminator as an energy function, which is optimized by the reconstruction loss as in autoencoders.

Original: BiGAN [26], ALI [27], VEEGAN [28], 
Revised: BiGAN [26], ALI [27], VEEGAN [28], and

Original: Inclusive GAN [29] also learn reconstruction networks.
Revised: Inclusive GAN [29] also learn reconstruction networks.

Original: In particular, VEEGAN [28] autoencodesthe latent vectors to learn the inverse function of the generator and map both the true and generateddata to the latent distribution, i .
Revised: In particular, VEEGAN [28] autoencodes the latent vectors to learn the inverse function of the generator, and maps both the true and generated data to the latent distribution, e .

Original: e . a Gaussian.
Revised: g ., a Gaussian distribution.

Original: Inclusive GAN [29] learns a generator by matchingbetween real and fake examples in the feature space.
Revised: Inclusive GAN [29] learns a generator by matching between real and fake examples in the feature space.

Original: The mode collapse and diversity issue ofgenerated outputs has been addressed explicitly in [11,30,31].
Revised: The mode collapse and diversity issues of generated outputs have been addressed explicitly in [11,30,31].

Original: They formulate the diversity metricsthat encourage the mode exploration of the generators and derive the loss function using the metrics.
Revised: They formulate the diversity metrics that encourage the mode exploration of the generators and derive the loss function using the metrics.

Original: 2.2 GAN with multiple generators
Revised: 2.2 GAN with multiple generators

Original: Another line of research is the integration of multiple generators [32–35].
Revised: Another line of research is the integration of multiple generators [32–35].

Original: This approach representsthe data distribution with a mixture model enforcing each generator to cover a portion of the wholedata space.
Revised: This approach represents a data distribution with a mixture model enforcing each generator to cover a portion of the whole data space.

Original: It is naturally expected that mixture models approximate true distributions better than asingle model especially in high-dimensional spaces with multiple modes.
Revised: It is naturally expected that mixture models approximate true distributions better than a single model, especially in high-dimensional spaces with multiple modes.

Original: MAD-GAN [33] introduce an augmented classiﬁer as a discriminator, which predicts whether thesample is real and which generator the sample is drawn from, to encourage individual generatorsto learn distinctive modes.
Revised: MAD-GAN [33] introduces an augmented classiﬁer as a discriminator, which predicts whether a sample is real and which generator the sample is drawn from, to encourage individual generators to learn distinctive modes.

Original: MGAN [34] hasthe similar strategies to MAD-GAN, but constructsa separate branch in the discriminator to perform the two tasks.
Revised: MGAN [34] has similar strategies to MAD-GAN but constructs a separate branch in its discriminator to perform the two tasks.

Original: MEGAN [35] adopts a gatingnetwork that produces a one-hot vector to select the generator creating the best example.
Revised: MEGAN [35] adopts a gating network that produces a one-hot vector to select the generator creating the best example.

Original: P2GAN [36]sequentially adds a new generator to cover the missing modes ofthe real data.
Revised: P2GAN [36] sequentially adds a new generator to cover the missing modes of real data.

Original: 2.3 GAN with multiple discriminators
Revised: 2.3 GAN with multiple discriminators

Original: Multiple discriminators are often employed to improve the performance of a single generator [8–10,12].
Revised: Multiple discriminators are often employed to improve the performance of a single generator [8– 10,12].

Original: D2GAN [8] conducts a three player minimax game, where two discriminators are trained for thecompletely opposite objectives, minimizing  Kullback-Leibler (KL) divergence and the inverse KLdivergence between the true and generated data distributions.
Revised: D2GAN [8] conducts a three-player minimax game, where two discriminators are trained for the completely opposite objectives, minimizing the Kullback-Leibler (KL) divergence and the inverse KL divergence between the true and generated data distributions.

Original: The balancing of  two losses plays arole for seeking desirable and diverse modes at the same time.
Revised: The balancing of the two losses plays a role in seeking desirable and diverse modes.

Original: Albuquerque et al .
Revised: Albuquerque et al .

Original: [12] propose ageneral multi-objective optimization framework in the scenario with multiple discriminators.
Revised: [12] propose a general multi-objective optimization framework in the scenario with multiple discriminators.

Original: Theypresent the hypervolume maximization algorithm to obtain weighted gradients.
Revised: They present a hypervolume maximization algorithm to obtain weighted gradients.

Original: Neyshabur et al .
Revised: Neyshabur et al .

Original: [13]train a GAN based on multiple projections.
Revised: [13] train a GAN based on multiple projections.

Original: Each discriminator makes a decision for the randomlow-dimensional projection of a sample to address the instability of GAN training in high-dimensions.
Revised: Each discriminator determines a random low-dimensional projection of a sample to address the instability of GAN training in high-dimensions.

Original: GMAN [9] presents diverse aggregation methods of multiple discriminators, where both hard and softdiscriminator selection strategies are studied.
Revised: GMAN [9] presents diverse aggregation methods of multiple discriminators, where both hard and soft discriminator selection strategies are studied.

Original: Note that all the existing approaches learn the multiplediscriminators independently and they may have strong correlations, which may not be appropriatefor diversifyingthe generated samples.
Revised: Note that all the existing approaches learn the multiple discriminators independently and they may have strong correlations, which may not be appropriate for diversifying generated samples.

Original: The proposed approach, however, assigns each sample to thebest-suited discriminator throughthe interactions among the discriminators, and, consequently, eachdiscriminator becomes the expert model for the assigned examples.
Revised: Our approach, however, assigns each sample to the best-suited discriminator through interactions among the discriminators, and, consequently, each discriminator becomes an expert model for the assigned examples.

Original:  Multiple Choice Learning
Revised:  Multiple Choice Learning

Original: We present the main idea of MCL [37] and its extensions brieﬂy.
Revised: We present the main idea of MCL [37] and its extensions brieﬂy.

Original: Given a training dataset with Nsamples, D = { ( x i , y i ) } Ni =1 , M models, { f m } Mm =1 and a task-speciﬁc loss function, (cid:96) ( · , · ) , MCLminimizes the following oracle loss:
Revised: Given a training dataset with N samples, D = { ( x i , y i ) } Ni =1 , M models, { f m } Mm =1 and a task-speciﬁc loss function, (cid:96) ( · , · ) , MCL minimizes the following oracle loss:

Original: [Equation]
Revised: [Equation]

Original: In other words, only the model with the smallest error out of M candidates is selected for eachexample.
Revised: In other words, only the model with the smallest error out of M candidates is selected for each example.

Original: This optimization process makes each model f m  become an expert for a subset of D , thusleads to forming a natural cluster in D .
Revised: This optimization process makes each model f m ( · ) become an expert for a subset of D , thus leads to forming a natural cluster in D .

Original: A weakness of MCL is the possible mistakes caused bythe overconﬁdence issues.
Revised: A weakness of MCL is the possible mistakes caused by overconﬁdence issues.

Original: If non-specializedmodels make wrong predictions with high conﬁdences in the score aggregation process, the averagescores are misleading and the ensemble model may result in poor quality outputs.
Revised: If non-specialized models make wrong predictions with high conﬁdence in the score aggregation process, the average scores are misleading and the ensemble model may result in poor-quality outputs.

Original: To alleviate thelimitation, Conﬁdent Multiple Choice Learning (CMCL) [19] adopts a conﬁdent oracle loss thatenforces the predictions of a non-specialized model to be uniformly distributed using KL divergence,denoted by D KL .
Revised: To alleviate this limitation, Conﬁdent Multiple Choice Learning (CMCL) [19] adopts a conﬁdent oracle loss that enforces the predictions of a non-specialized model to be uniformly distributed using KL divergence, denoted by D KL .

Original: Assuming that f m  predicts the output distribution given data point x , i .
Revised: Assuming that f m ( · ) predicts the output distribution given data point x , i .

Original: e .,
Revised: e .,

Original: P m ( y | x ) ,the loss is modiﬁed as
Revised: P m ( y | x ) , the loss is modiﬁed as

Original: [Equation]
Revised: [Equation]

Original: where U ( y ) is the uniform distribution and the ﬂag variable v i,m ∈ { 0 , 1 } allows the choices of thespecialized models.
Revised: where U ( y ) is the uniform distribution and the ﬂag variable v i,m ∈ { 0 , 1 } allows the choices of the specialized models.

Original: Note that, if (cid:80) Mm =1 v i,m = k
Revised: Note that, if (cid:80) Mm =1 v i,m = k

Original: ( k < M ), each example is assigned to k models.
Revised: ( k < M ), each example is assigned to k models.

Original:  MCL-GAN
Revised:  MCL-GAN

Original: We describe our GAN structure with a generator G ( · ; θ ) and M discriminators { D m ( · ; φ m ) } Mm =1extended from the standard GAN.
Revised: We describe our GAN extension with a generator G ( · ; θ ) and M discriminators { D m ( · ; φ m ) } M m =1 .

Original: Let p z and p d be the distributions of the latent space and real dataspace, respectively.
Revised: Let p z and p d be the distributions in latent space and real data space, respectively.

Original: Given z ∼ p z , the generator produces a sample ˜ x
Revised: Given z ∼ p z , the generator produces a sample ˜ x

Original: = G ( z ; θ ) and M predictionsare made by the  discriminators for each real example x ∼ p d and fake sample ˜ x .
Revised: = G ( z ; θ ) and each of the M discriminators makes a prediction on a real example x ∼ p d or the fake sample ˜ x .

Original: Each prediction, D m ( x ; φ m ) , ranges in [0 , 1] and represents the probability that x belongs to the true data distribution.
Revised: Each prediction, D m ( x ; φ m ) , ranges in [0 , 1] and represents the probability that x belongs to the true data distribution.

Revised: We now describe the loss functions L D and L G used to train the discriminators and the generator, respectively.

Original: 4.1 Expert training
Revised: 4.1 Expert training

Original: Assuming that we draw N d real data and generate N g examples in each training batch, denoted by xand ˜ x , respectively, each network is trained as follows.
Revised: Assuming that we draw N d real data and generate N g examples in each training batch, denoted by x and ˜ x , respectively, each network is trained as follows.

Original: Discriminators Expert discriminators are the ones that predict the highest scores for an example.
Revised: Discriminators Expert discriminators are the ones that predict the highest scores for each example.

Original: With the indicator variable v i,m for  x i , the discriminators are trained to minimize the following lossfunction:
Revised: With the indicator variable v i,m for a real example x i , the discriminators are trained to minimize the following loss function:

Original: [Equation]
Revised: [Equation]

Original: ) where we choose k experts out of M discriminators for each example, i .
Revised: where we choose k experts out of M discriminators for each example, i .

Original: e ., (cid:80)
Revised: e ., (cid:80)

Original: Mm =1 v i,m = k .
Revised: Mm =1 v i,m = k .

Original: For afake sample, since all discriminators have to identify it correctly, the following loss is added to (3):
Revised: For a fake sample, since all discriminators have to identify it correctly, the following loss is added to (3):

Original: [Equation]
Revised: [Equation]

Original: Generator We train the generator using the gradients received from the expert models to encouragethe generator to ﬁnd the closest mode given z .
Revised: Generator We train the generator using the gradients received from the expert models to encourage the generator to ﬁnd the closest mode given z .

Original: With another indicator variable u j,m for z j , the expertloss for the generator is given by
Revised: With another indicator variable u j,m for z j , the expert loss for the generator is given by

Original: [Equation]
Revised: [Equation]

Original: 4.2 Non-expert training
Revised: 4.2 Non-expert training

Original: The non-expert discriminators should not be over-conﬁdent to real example while it is desirable toproduce higher scores for real samples than fake ones.
Revised: The non-expert discriminators should not be overconﬁdent to real examples while it is desirable to produce higher scores for real samples than fake ones.

Original: For this requirement, we give a uniform softlabel, e . g .,
Revised: For this requirement, we give a uniform soft label, e . g .,

Original: y =
Revised: y =

Original: 5] for non-expert discriminators and regularize them with some weight.
Revised: 5] for non-expert discriminators and regularize them with a certain weight.

Original: To beprecise, we obtain the following non-expert loss term corresponding to (3):
Revised: To be precise, we obtain the following non-expert loss term corresponding to (3):

Original: [Equation]
Revised: [Equation]

Original: with the same v i,m  deﬁned in (3) and (cid:96) ce ( · , y ) is the cross-entropy loss function given a target label y .
Revised: where v i,m is the same indicator variable deﬁned in (3) and (cid:96) ce ( · , y ) is the cross-entropy loss function given a target label y .

Original: The other counterpart for (5) is derived similarly as
Revised: The other counterpart for (5) is derived similarly as

Original: [Equation]
Revised: [Equation]

Original: The non-expert model training is effective to handle the overconﬁdence issue, but the model may stillsuffer from the data deﬁciency problem of the standard MCL framework because each discriminatorcan see only a subset of the whole dataset.
Revised: The non-expert model training is effective to handle the overconﬁdence issue, but the model may still suffer from the data deﬁciency problem frequently happening in the standard MCL algorithms because each discriminator can observe only a subset of the whole dataset.

Original: To ameliorate this limitation, our discriminators share theparameters of all layers for feature extraction while branching the last layer only.
Revised: To alleviate this limitation, our discriminators share the parameters of all layers in the feature extractor while branching the last layer only.

Original: This implementationis also sensible in that the discriminators partially have the same objective to distinguish the fakeexamples.
Revised: This implementation is also sensible since the discriminators partially have the same objective to distinguish fake examples.

Original: The common representations of all real samples are likely to be learned in the earlier layersdespite being clustered inthe different subsets whereas the critical information for the high-levelclassiﬁcation is often found in the last layer.
Revised: The common representations of all real samples are prone to be learned in the earlier layers despite being clustered in different subsets, whereas the critical information for high-level understanding is often encoded in deeper layers.

Original: Moreover, the number of training parameters and trainingtime are saved signiﬁcantly while  taking advantage of ensemble learning.
Revised: Moreover, the number of model parameters and training time are saved signiﬁcantly while still taking advantage of ensemble learning.

Original: 4.3 Balanced assignment of discriminators
Revised: 4.3 Balanced assignment of discriminators

Original: On top of the adversarial losses, we introduce another loss for balanced updates of discriminators.
Revised: On top of the adversarial losses, we introduce another loss for balanced updates of discriminators.

Original: As there is no supervision for the specialized factor for certain discriminator, e .
Revised: As there is no supervision about the specialization target of a discriminator, e .

Original: g ., class labels orfeature embeddings, it may be difﬁcult to reasonably distribute real samples to expert models fromthe beginning.
Revised: g ., class labels or feature embeddings, it may be difﬁcult to reasonably distribute real samples to expert models from the beginning.

Original: Since the abilities of individual discriminators are severely off-balanced, they are highly prone to assign all samples to  few speciﬁc models.
Revised: Since the abilities of individual discriminators are severely off-balanced, they are highly prone to assign all samples to a few speciﬁc models.

Original: Especially at an early phase of training, themodel’s capability is more sensitive to the number of updates in the discriminators.
Revised: Especially at an early phase of training, the model’s capability is more sensitive to the number of updates in the discriminators.

Original: To tackle this challenge, we propose another loss, called the balance loss, which gives discriminatorsbalanced chances for updates.
Revised: To tackle this challenge, we propose another loss, called the balance loss, which gives discriminators equal chances for updates.

Original: Let the selection of expert discriminators approximately follow acategorical distribution with a parameter µ = [ µ 1 , . .
Revised: Let the selection of expert discriminators approximately follow a categorical distribution with a parameter µ = [ µ 1 , . .

Original: . , µ M ] .
Revised: . , µ M ] .

Original: Then the loss is computed by the KLdivergence of the probability distribution of discriminators for being selected as experts from µ .
Revised: Then the loss is computed by the KL divergence of the probability distribution of discriminators for being selected as experts from µ .

Original: Toobtain the probability for discriminator selection, we apply the softmax function to the vector of Mpredictions from discriminators for each example since the discriminator with the highest score isguaranteed to be chosen as an expert.
Revised: To obtain the probability for discriminator selection, we apply the softmax function to the vector of M predictions from discriminators for each example since the discriminator with the highest score is guaranteed to be chosen as an expert.

Original: We average these probability vectors over the training batch.i .
Revised: We average these probability vectors over the training batch. i .

Original: e ., q = 1 N d (cid:80) N d i =1 s ([ D 1 ( x i ) , . . .
Revised: e ., q = 1 N d (cid:80) N d i =1 s ([ D 1 ( x i ) , . . .

Original: , D M ( x i )]; τ ) , where s ( · ; τ ) denotes a vector-valued softmaxfunction with  temperature τ given an input vector.
Revised: , D M ( x i )]; τ ) , where s ( · ; τ ) denotes a vector-valued softmax function with a temperature τ given a logit.

Original: To sum up, the balance loss is given by
Revised: To sum up, the balance loss is given by

Original: [Equation]
Revised: [Equation]

Original: In practice, we set µ m = 1 M , ∀ m to update the discriminators evenly, which is because the truedistribution is unavailable.
Revised: In practice, we set µ m = 1 M , ∀ m to update the discriminators evenly, which is because the true distribution is unavailable.

Original: This assumption may not be congruent to the real distribution of thedataset and excessively forced assignment would not result in an optimal clustering for specialization.
Revised: This assumption may not be congruent to the real distribution of the dataset and excessively forced assignment would not result in an optimal clustering for specialization.

Original: We, therefore, decrease the weight for the balance loss gradually during training.
Revised: We, therefore, decrease the weight for the balance loss gradually during training.

Original: Eventually, eachexample will be naturally assigned to its best model with a very small weight of the balance loss.
Revised: Eventually, each example will be naturally assigned to its best model with a very small weight of the balance loss.

Original: This adjustment helps stabilize training and naturally cluster the reference data.
Revised: This adjustment helps stabilize training and naturally cluster the reference data.

Original: Note that the modelsare balanced within a few epochs and the weight reduction helps generate higher quality samples.
Revised: Note that the models are balanced within a few epochs and the weight reduction helps generate higher quality samples.

Original: Likewise, a small enforcement on the distribution of the generator’s output facilitates balancedgeneration when the statistics of generated samples are skewed.
Revised: Likewise, a small constraint on the distribution of the generator’s output facilitates balanced generation while preventing the statistics of generated samples from being skewed.

Original: For this case, we use the distributionof the discriminators’ assignments  instead of arbitrarily chosen µ , i .
Revised: For this case, we use the distribution of the discriminators’ assignments q instead of arbitrarily chosen µ , i .

Original: e .,
Revised: e .,

Original: [Equation]
Revised: [Equation]

Original: (9) where o = 1 N g (cid:80) N g j =1 s ([ D 1 ( G ( z j )) , . ..
Revised: where o = 1 N g (cid:80) N g j =1 s [ D 1 ( G ( z j )) , . .

Original: , D M ( G ( z j ))]; τ ) .
Revised: , D M ( G ( z j ))]; τ ) .

Original: 4.4 Total loss
Revised: 4.4 Total loss

Original: Altogether, the total loss is summarized  asfollows:
Revised: Altogether, the total losses for discriminators and the generator are summarized respectively as

Original: [Equation]
Revised: [Equation]

Original: ) where β is different for discriminators and generator.
Revised: where α and { β d , β g } are hyperparameters.

Original: Note that the proposed approach is applicableto any GAN formulations with the corresponding adversarial losses.
Revised: Note that the proposed approach is applicable to any GAN formulations with the corresponding adversarial losses.

Original: 4.5 Choice of number of discriminators
Revised: 4.5 Choice of number of discriminators

Original: A remaining concern is how to ﬁnd the optimal number of discriminators while such information isnot available in general as in many clustering tasks.
Revised: A remaining concern is how to ﬁnd the optimal number of discriminators while such information is not available in general as in many clustering tasks.

Original: If the number of discriminators is much largerthan the optimal one, it is more desirable to focus on training a subset of discriminators than dividingthe dataset into many minor clusters forcefully.
Revised: If the number of discriminators is much larger than the optimal one, it is more desirable to focus on training a subset of discriminators than dividing the dataset into many minor clusters forcefully.

Original: To ease this issue, wecan augment  (cid:96) 1 regularization loss to the discriminator  with  weight γ andencourage the sparsity in the discriminator selection for more desirable clustering results.
Revised: To ease this issue, we augment the (cid:96) 1 regularization loss to the discriminator side with a weight γ and encourage the sparsity in the discriminator selection for more desirable clustering results.

Original: Hence, evenin the case that we are given an excessively large number of discriminators, our algorithm convergesat good points by using a small number of discriminators in practice.
Revised: Hence, even in the case that we are given an excessively large number of discriminators, our algorithm converges at good points by using a small subset of discriminators in practice.

Original: It is true that this strategy maynot always lead to the optimal number of discriminators and has conﬂict with the balance loss in (8).
Revised: It is true that this strategy may not always lead to the optimal number of discriminators and have conﬂicts with the balance loss in (8).

Original: However, the balance loss fades away as training goes, and our model identiﬁes a proper number ofclusters by deactivating a subset of discriminators.
Revised: However, the balance loss fades away as training goes on, and our model identiﬁes a proper number of clusters by deactivating a fraction of discriminators adaptively.

Original: This sparsity loss may be useful when we learn onthe examples  drawn from unknown distributions.
Revised: This sparsity loss would be useful when we learn on real datasets, where the examples are drawn from unknown distributions.

Original:  Experiments
Revised:  Experiments

Original: We evaluate the performance of MCL-GAN on unconditional and conditional image generation.
Revised: We evaluate the performance of MCL-GAN on unconditional and conditional image generation.

Original: Notethat the asterisk(*) denotes the copied results from other papers throughout this section.
Revised: Note that the asterisk

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: 5.1 Synthetic dataset
Revised: 5.1 Synthetic dataset

Original: [Figure]

Original: We ﬁrst perform toy experiments to verify the main idea of MCL-GAN intuitively.
Revised: We ﬁrst perform toy experiments to verify the main idea of MCL-GAN intuitively.

Original: We consider amixture of 8 2D isotropic Gaussians whose centers are located on the circumference of a circle with aradius of √ 2 while their standard deviation in each dimension is 0.05.
Revised: We consider a mixture of 8 2D isotropic Gaussians whose centers are located on the circumference of a circle with a radius of √ 2 while their standard deviation in each dimension is 0.05.

Original: We employ 8 discriminatorsfor training with the standard GAN loss while utilizing 2 discriminators with the Hinge loss [38].
Revised: We employ 8 discriminators for training with the standard GAN loss while utilizing 2 discriminators with the Hinge loss [38].

Original: Wechoose one expert discriminator for each sample ( k = 1 ) in all experiments.
Revised: We choose one expert discriminator for each sample ( k = 1 ) in all experiments.

Original: Figure 1 illustrates the snapshots of generated examples through iterations by the baselines and
Revised: Figure 1 illustrates the snapshots of generated examples through iterations by the baselines and

Original: MCL-GANs.
Revised: MCL-GANs.

Original: Unlike the base models ( M = 1 ) that fail to cover all 8 modes, MCL-GANs learnto identify diverse modes quickly and produce the samples at all modes eventually.
Revised: Unlike the base models ( M = 1 ) that fail to cover all 8 modes, MCL-GANs learn to identify diverse modes quickly and produce the samples at all modes eventually.

Original: Note that the
Revised: Note that the

Original: Hinge loss tends to generate more diverse examples even with a single discriminator and MCL-GANalso requires less discriminators in the Hinge loss case to reconstruct the original data distribution.
Revised: Hinge loss tends to generate more diverse examples even with a single discriminator and MCL-GAN also requires less discriminators in the Hinge loss case to reconstruct the original data distribution.

Original: Figure 2 shows how MCL-GANs adaptively select discriminators, given an excessive number ofdiscriminators, e . g ., M = 20 .
Revised: Figure 2 shows how MCL-GANs adaptively select discriminators, given an excessive number of discriminators, e . g ., M = 20 .

Original: They mostly utilize 8 or 16 expert discriminators when γ = 0 and10 − 5 , respectively; it turns out that each mode is associated with 1 or 2 discriminators dependingon the value of γ .
Revised: They mostly utilize 8 or 16 expert discriminators when γ = 0 and 10 − 5 , respectively; it turns out that each mode is associated with 1 or 2 discriminators depending on the value of γ .

Original: We visualizes the detailed mapping between examples and discriminators in thesupplementary document.
Revised: We visualize the detailed mapping between examples and discriminators in Appendix A.

Original: This observation implies that MCL-GAN covers all modes effectivelywhile the (cid:96) 1 loss helps identify the proper number of discriminators to generate high-ﬁdelity data.
Revised: This observation implies that MCL-GAN covers all modes effectively while the (cid:96) 1 loss helps identify the proper number of discriminators to generate high-ﬁdelity data.

Original: 5.2 Unconditional GAN on image dataset 
Revised: 5.2 Unconditional GAN on image dataset 5.2.1 Experiment setup and evaluation protocol

Original: 5.2.1 Experiment setup and evaluation protocol

Original: We run the unconditional GAN experiment on four distinct datasets including MNIST [39], FashionMNIST [40], CIFAR-10 [41] and CelebA [42], where two types of network architectures areemployed—DCGAN [43] and StyleGAN2 [44].
Revised: We run the unconditional GAN experiment on four distinct datasets including MNIST [39], FashionMNIST [40], CIFAR-10 [41] and CelebA [42], where two types of network architectures are employed—DCGAN [43] and StyleGAN2 [44].

Original: The images are resized to 32 × 32 except for
Revised: The images are resized to 32 × 32 except for

Original: CelebA dataset, for which DCGAN and StyleGAN2 adopt 64 × 64 and 128 × 128 images, respectively.
Revised: CelebA dataset, for which DCGAN and StyleGAN2 adopt 64 × 64 and 128 × 128 images, respectively.

Original: For the StyleGAN2 experiments on CelebA, we use the ﬁrst and the last 30K images from the“align&cropped" version for the train and validation splits following [29].
Revised: For the StyleGAN2 experiments on CelebA, we use the ﬁrst and the last 30K images from the “align&cropped” version for the train and validation splits following [29].

Original: We apply our method to the
Revised: We apply our method to the

Original: DCGAN architecture with three different GAN loss functions: the vanilla GAN loss [1], LSGAN [21]and Hinge loss [38].
Revised: DCGAN architecture with three different GAN loss functions: the vanilla GAN loss [1], LSGAN [21] and Hinge loss [38].

Original: The supplementary document describes more details of our setting.
Revised: Appendix E describes more details of our setting.

Original: We adopt Precision Recall Distribution (PRD) [45] and Frèchet Inception Distance (FID) [46] as ourevaluation metrics. 
Revised: We adopt Precision Recall Distribution (PRD) [46] and Frèchet Inception Distance (FID) [47] as our evaluation metrics. The PRD curve provides more credible assessment on generative models than [Table]

Original: The PRD curve provides more credible assessment on generative models thansingle-valued metrics since it quantiﬁes the model in two folds—mode coverage (recall) and qualityof generated samples (precision).
Revised: [Table] [Table] single-valued metrics since it quantiﬁes the model in two folds—mode coverage (recall) and quality of generated samples (precision).

Original: We measure the recall and precision of each model by computingthe F 8 and F 1 / 8 scores from the PRD curve, respectively.
Revised: We measure the recall and precision of each model by computing the F 8 and F 1 / 8 scores from the PRD curve, respectively.

Original: We prefer higher scores and they are equalto 1 when the generated data distribution is identical to the reference.
Revised: We prefer higher scores and they are equal to 1 when the generated data distribution is identical to the reference.

Original: FID is a popular evaluationmetric of generative models, based on the distance between two datasets with multivariate Gaussianassumption.
Revised: FID is a popular evaluation metric of generative models, based on the distance between two datasets with multivariate Gaussian assumption.

Original: Lower FID scores mean that generated samples are closer to the reference data.[Table] [Table] [Table]
Revised: Lower FID scores mean that generated samples are closer to the reference data.

Original: 5.2.2 DCGAN backbone
Revised: 5.2.2 DCGAN backbone

Original: Table 1 summarizes the precision and recall scores of our methods compared to the baseline modelswith three different GAN objectives, when the number of discriminators is set to 5 or 10 ( M = 5 or 10 )and the number of experts is 1 ( k = 1 ).
Revised: Table 1 summarizes the precision and recall scores of our methods compared to the baseline models with three different GAN objectives, when the number of discriminators is set to 5 or 10 ( M = 5 or 10 ) and the number of experts is 1 ( k = 1 ).

Original: MCL-GAN achieves outstanding performance in terms ofboth metrics compared to the baseline and GMAN, which is an existing approach based on multiplediscriminators, on MNIST and CelebA. For Fashion-MNIST, we observe that MCL-GAN focuses onthe mode coverage (diversity) while GMAN cares about the image quality rather than the diversity.
Revised: MCL-GAN achieves outstanding performance in terms of both metrics compared to the baseline and GMAN, which is an existing approach based on multiple discriminators, on MNIST and CelebA. For Fashion-MNIST, we observe that MCL-GAN focuses on the mode coverage (diversity) while GMAN cares about the image quality rather than the diversity.

Original:  MCL-GAN outperformsall the compared GAN models based on DCGAN by large marginsexcept MGAN and MSGAN in terms ofthe FID scores on CIFAR-10, as presented in Table 2.
Revised: As presented in Table 2, MCL-GAN outperforms the compared GAN models (based on DCGAN) by large margins in terms of FID scores on CIFAR-10, while being competitive with MGAN, which relies on multiple generators. MCL-GAN also achieves better results than a clustering-based approach, i .

Original: Note that MGAN relies on multiple generators, 10 in this case, and MSGAN requires class labels for training.
Revised: e ., self-conditioned GAN [45], which requires additional computation and model parameters due to clustering features during training and conditioning the generator’s inputs with cluster label embedding.

Original: The results imply that MCL-GAN is effective to maintain the multi-modality in the underlyingdistribution with relatively small memory footprint and without extra supervision.
Revised: The results imply that MCL-GAN is effective to maintain the multi-modality in the underlying distribution with relatively small memory footprint and without extra supervision.

Original: To analyze the semantic quality of generated images, we present their classiﬁcation results given bythe pretrained classiﬁers in Table 3.
Revised: To analyze the semantic quality of generated images, we present their classiﬁcation results given by the pretrained classiﬁers in Table 3.

Original: We measure how much the predicted label distribution in eachtested dataset deviates from the true (uniform) distribution using the KL divergence. 
Revised: We measure how much the predicted label distribution in each tested dataset deviates from the true (uniform) distribution using the KL divergence. We also calculate [Figure]

Original: We also calculatethe average of class-wise LPIPS to measure the intra-class diversity of generated images.
Revised: [Table] [Table] the average of class-wise LPIPS to measure the intra-class diversity of generated images.

Original: The overallresults are favorable to MCL-GAN but there exist some misleading points.
Revised: The overall results are favorable to MCL-GAN but there exist some misleading points.

Original: For example, GMANtends to achieve overly high class-wise LPIPS scores due to many out-of-distribution examples thatconfuse the classiﬁer, as illustrated in the supplementary document.
Revised: For example, GMAN tends to achieve overly high class-wise LPIPS scores due to many out-of-distribution examples that confuse the classiﬁer, as illustrated in Appendix D.1.

Original: 5.2.3 StyleGAN2 backbone
Revised: 5.2.3 StyleGAN2 backbone

Original: Table 4 presents that MCL-GAN is also effective in the state-of-the-art backbone model and outperforms StyleGAN2 and its variation, Inclusive GAN [29], in terms of all metrics.
Revised: Table 4 presents that MCL-GAN is also effective in the state-of-the-art backbone model and outperforms StyleGAN2 and its variation, Inclusive GAN [29], in terms of all metrics.

Original: Note that Inclusive
Revised: Note that Inclusive

Original: GAN uses the sample-wise reconstruction loss by regarding each image as a mode.
Revised: GAN uses the sample-wise reconstruction loss by regarding each image as a mode.

Original: This strategyappears to improve recall but the model may suffer from sampling bias and scalability issue byestimating overly complex distributions.
Revised: This strategy appears to improve recall but the model may suffer from sampling bias and scalability issue by estimating overly complex distributions.

Original: The results imply that MCL-GAN enhances the convergenceand the reproducibility of large GAN models, practically leading to improved performance.[Figure] [Table]
Revised: The results imply that MCL-GAN enhances the convergence and the reproducibility of large GAN models, practically leading to improved performance.

Original: 5.2.4 Discriminators specialization
Revised: 5.2.4 Discriminators specialization

Original:  Figure 3 qualitatively presents how successfully the discriminators in MCL-GAN are specialized tosubsets of datasets.
Revised: Qualitative results Figure 3 qualitatively presents how successfully the discriminators in MCLGAN are specialized to subsets of datasets.

Original: We learn the model with 10 discriminators, and illustrate the generated imageswith their membership to discriminators; the images in the same row belong to the same discriminator.
Revised: We learn the model with 10 discriminators, and illustrate the generated images with their membership to discriminators; the images in the same row belong to the same discriminator.

Original: We observe semantic consistency of images within the same row in both MNIST and Fashion-MNISTclearly.
Revised: We observe semantic consistency of images within the same row in both MNIST and Fashion-MNIST clearly.

Original: We further analyze discriminators specialization using attribute annotations  of  CelebA in thesupplementary document.
Revised: Analysis using attribute annotations To examine how effective the discriminators represent the real data, we analyze the discriminator specialization statistics of MCL-GAN for individual attributes on CelebA dataset, which has 40 binary attribute annotations per image. We ﬁrst compute the normalized histogram of discriminator assignments for the images with a certain attribute, and then compute the cosine similarity between the histograms to obtain a 40 × 40 dimensional similarity matrix, where 40 is the number of attributes. Figure 4 illustrates a submatrix of the similarity matrix, where we choose a subset of attributes for better visualization. We discover that the discriminator specialization statistics highly coincide with the latent semantic similarities of examples.

Revised: Note that the four attributes more relevant to female—Arched_Eyebrows,

Original: 5.3 Conditioned image synthesis
Revised: Wearing_Earings, Wearing_Lipstick, and Heavy_Makeup—are clustered together, while another set of four attributes such as Goatee, Male, Mustache, and Sideburns construct a strong group. These results support that the individual discriminators learned by MCL-GAN are specialized to reasonable A r c h e d _ E y e b r o w s W e a r i n g _ E a rr i n g s

Revised: W e a r i n g _ L i p s t i c k

Revised: H e a v y _ M a k e u p B l o n d _ H a i r B l a c k _

Revised: H a i r G o a t e e M a l e

Revised: M u s t a c h e S

Revised: i d e b u r n s

Revised: Arched_Eyebrows

Revised: [Figure]

Revised: GAN on CelebA dataset.

Revised: [Table]

Original:  We apply MCL-GAN to image-to-image translation and text-to-image synthesis tasks, which requirecomplex architectures to generate high-resolution images.
Revised: [Table] subsets of the dataset, which is helpful to generate more realistic images congruent to the samples in the latent space. 5.3 Conditioned image synthesis We apply MCL-GAN to image-to-image translation and text-to-image synthesis tasks, which require complex architectures to generate high-resolution images.

Original: In this experiment, we adopt MSGAN [11],a technique with a mode-seeking regularizer, as an additional component which alleviates the modecollapse in conditional GANs.
Revised: In this experiment, we adopt MSGAN [11], a technique with a mode-seeking regularizer, as an additional component which alleviates the mode collapse in conditional GANs.

Original: Then, we observe whether the mode seeking and our multiple choicelearning create synergy, based on FID, NDB/JSD [47], and LPIPS [48] following [11]. 
Revised: Then, we observe whether the mode seeking and our multiple choice learning create synergy, based on FID, NDB/JSD [48], and LPIPS [49] following [11]. Note that NDB counts the number of statistically different bins based on the clusters made by k -means clustering.

Original: The deﬁnitionsand properties of these metrics are discussed in the supplementary ﬁle.
Revised: [Table]

Original: Image-to-image translation We choose DRIT [4, 49], an unpaired image-to-image translationmethod based on the cycle consistency, as our baseline.
Revised: Image-to-image translation We choose DRIT [4, 5], an unpaired image-to-image translation method based on the cycle consistency, as our baseline.

Original: We employ MCL-GAN with M = 3 andk{+
 and k+} = 1 for distinguishing real and translated images.
Revised: We employ MCL-GAN with M = {+
 and k+} = 1 for distinguishing real and translated images.

Original: As shown in Table 5, MCL-GAN signiﬁcantlyimproves the diversity measure, LPIPS, while achieving high-ﬁdelity data generation performancein terms of other metrics.
Revised: As shown in Table 5, MCL-GAN signiﬁcantly improves the diversity measure, LPIPS, while achieving high-ﬁdelity data generation performance in terms of other metrics.

Original: In particular, our approach works better on a more challenging task, cat283 (cid:10) dog, since it effectively handles the changes in both object shape and texture across domainsby specializing discriminators to a subset of modes.
Revised: In particular, our approach works better on a more challenging task, cat (cid:10) dog, since it effectively handles the changes in both object shape and texture across domains by specializing discriminators to a subset of modes.

Original: Note that the integration of MCL is mostlybeneﬁcial and the addition of the mode-seeking module often leads to extra performance gains.
Revised: Note that the integration of MCL is mostly beneﬁcial and the addition of the mode-seeking module often leads to extra performance gains.

Original: Text-to-image synthesis This experiment is based on StackGAN ++ [50] trained on CUB-2002011 [51] with a mode-seeking regularizer.
Revised: Text-to-image synthesis This experiment is based on StackGAN ++ [50] trained on CUB-2002011 [51] with a mode-seeking regularizer.

Original: StackGAN +
Revised: StackGAN +

Original: + has a hierarchical structure with aspecialized pair of a discriminator and a generator to a certain image resolution.
Revised: + has a hierarchical structure with a specialized pair of a discriminator and a generator to a certain image resolution.

Original: We adopt its 3-stageversion and trains an MCL-GAN with M = 3 and k = 1 only at the last stage, which handlesimages with size 256 × 256 .
Revised: We adopt its 3-stage version and trains an MCL-GAN with M = 3 and k = 1 only at the last stage, which handles images with size 256 × 256 .

Original: Table 6 illustrates that the integration of MCL improves performanceconsistently, especially in terms of the diversity measure, LPIPS, where we observe the merit of themode-seeking regularizer via the combination with MCL.[Table] [Table] [Table]
Revised: Table 6 illustrates that the integration of MCL improves performance consistently, especially in terms of the diversity measure, LPIPS, where we observe the merit of the mode-seeking regularizer via the combination with MCL.

Original: 5.4 Discussion
Revised: 5.4 Discussion

Original: MCL-GAN is a model-agnostic ensemble algorithm with multiple discriminators.
Revised: MCL-GAN is a model-agnostic ensemble algorithm with multiple discriminators.

Original: Our experimentsshow that the specialized discriminators to the subsets of training data are helpful compared toindependent training of the discriminators on the whole dataset (as in GMAN) or different strategiesof discriminator assignments, i .
Revised: Our experiments show that the specialized discriminators to the subsets of training data are helpful compared to independent training of the discriminators on the whole dataset (as in GMAN) or different strategies of discriminator assignments, i .

Original: e ., minimum-score discriminator selection, which is an oppositecriterion to MCL-GAN, or random selection.
Revised: e ., minimum-score discriminator selection, which is an opposite criterion to MCL-GAN, or random selection.

Original: Also, the performance of MCL-GAN is as competitiveas the method assigning discriminators based on the ground-truth class labels, which exhibits thereliability of the discriminator specialization by MCL.
Revised: Also, the performance of MCL-GAN is as competitive as the method assigning discriminators based on the ground-truth class labels, which exhibits the reliability of the discriminator specialization by MCL.

Original: Table 7 presents that MCL-GAN achievesoutstanding performance compared to other strategies especially for the recall metric, even comparedwith GT-Assign without relying on class labels for discriminator specialization.
Revised: Table 7 presents that MCL-GAN achieves outstanding performance compared to other strategies especially for the recall metric, even compared with GT-Assign without relying on class labels for discriminator specialization.

Original: The proposedapproach is also efﬁcient since it is free from any time-consuming clustering procedure for sampleassignment to discriminators during training, and incurs marginal extra cost despite the use of multiplediscriminators because the discriminators share the feature extractor.
Revised: The proposed approach is also efﬁcient since it is free from any time-consuming clustering procedure for sample assignment to discriminators during training, and incurs marginal extra cost despite the use of multiple discriminators because the discriminators share the feature extractor.

Original:  Conclusion
Revised:  Conclusion

Original: We presented a generative adversarial network framework with multiple discriminators, where eachdiscriminator behaves as an expert classiﬁer and covers a separate mode in the underlying distribution.
Revised: We presented a generative adversarial network framework with multiple discriminators, where each discriminator behaves as an expert classiﬁer and covers a separate mode in the underlying distribution.

Original: This idea is implemented by incorporating the concept of multiple choice learning.
Revised: This idea is implemented by incorporating the concept of multiple choice learning.

Original: The combinationof generative adversarial network and multiple choice learning turns out to be effective to alleviatethe mode collapse problem.
Revised: The combination of generative adversarial network and multiple choice learning turns out to be effective to alleviate the mode collapse problem.

Original: Also, the integration of the sparsity loss encourages our model to identifythe proper number of discriminators and estimate a desirable distribution.
Revised: Also, the integration of the sparsity loss encourages our model to identify the proper number of discriminators and estimate a desirable distribution.

Original: We demonstrated theeffectiveness of the proposed algorithm on various GAN models and datasets.
Revised: We demonstrated the effectiveness of the proposed algorithm on various GAN models and datasets.

Revised: Acknowledgements This work was partly supported by Samsung Electronics Co., Ltd., and Institute of Information & communications Technology Planning & Evaluation (IITP) grant and the

Revised: National Research Foundation of Korea (NRF) grant funded by the Korean government (MSIT)

Revised: [No.2022-0-00959, (Part 2) Few-Shot Learning of Causal Inference in Vision and Language for

Original: Checklist
Revised: Decision Making; No.2022R1A5A708390811]. Appendix A Mapping between examples and discriminators on synthetic data Figure 5 illustrates how the generated samples from a synthetic dataset based on a mixture with Gaussians are associated with 20 discriminators in the proposed MCL-GAN framework.

Revised: Note that this ﬁgure is an extended version of Figure 2 in the main paper.

Revised: The true data are represented in orange while other colors denote the expert discriminator incides for individual examples.

Revised: Without the (cid:96) 1 loss ( γ = 0 ), 16 discriminators are utilized to cluster the true distribution where exactly two discriminators are responsible for each mode; two distinct colors appear in each mode except the one (orange) corresponding to true data.

Revised: By adding the (cid:96) 1 loss with a small coefﬁcient, we discover only 8 discriminators are active in training, one for each mode.

Revised: [Figure]

Revised: B Analysis on hyperparameters B.1 Non-expert loss weight

Revised: Table 8 shows the effect of non-expert training regularization when training 10 discriminators with the standard GAN loss on MNIST.

Revised: The overall performance is robust to the variation of the hyperparameter α .

Revised: The best score is obtained with α = 0 .

Revised: while all positive α improves the precision scores, which reduces the overconﬁdence issue of non-expert discriminators.

Revised: [Table]

Revised: [Table]

Revised: weights

Revised: Let β d

Revised: and β g be two different versions of β in Eq.

Revised: (10) of the main paper; the former is for the discriminator and the latter is for the generator.

Revised: We conduct the ablation studies on several ( β d , β g ) combinations for 10 discriminators with Hinge loss on MNIST.

Revised: Table 9 shows that β d plays an important role in boosting the performance of GAN.

Revised: This is mainly because β d is responsible for distributing the chances of being an expert to multiple discriminators; only a few discriminators are utilized in training if β d is zero or too small.

Revised: β g has a smaller effect on the performance than β d .

Revised: However, it helps improve recall scores without sacriﬁcing precision and leads to the best score when ( β d , β g ) =

Revised: Note that the performance does not change drastically for all cases with the positive value of β d , which surpass a single discriminator GAN by large margins.

Revised: [Table]

Revised: B.3 Number of experts per example

Revised: We evaluate our model by varying the number of experts k with M = 5 and

Revised: [Table]

Revised: and present the results on MNIST and CIFAR-10 in Table 10.

Revised: The number of optimal k may be different in each dataset, however, choosing too many experts tend to drop the recall scores.

Revised: Figure 6 shows how the specialization characteristics of discriminators differ by the number of experts per sample,

Revised: i

Revised: [Table]

Revised: e ., k ∈ { 1 , 3 , 5 } , when there are 10 discriminators on MNIST and Fashion-MNIST.

Revised: As k increases, the models get less specialized by sharing more data each other so the subclusters become less distinctive.

Revised: [Table]

Revised: B.4 Number of discriminators and (cid:96) 1 loss weight

Revised: We conduct the experiment under a various number of discriminators and illustrate the results in

Revised: Table 11.

Revised: It turns out that the performance of the proposed method is fairly robust to the number of discriminators quantitatively and adding the (cid:96) 1 loss does not incur noticeable differences in terms of precision/recall measure.

Revised: However, interestingly, the (cid:96) 1 loss plays a crucial role in ﬁnding modes in the underlying distribution.

Revised: Figure 7 illustrates the impact of the (cid:96) 1 loss on MNIST and FashionMNIST when we train the model with 40 discriminators.

Revised: According to our results, only a fraction of the discriminators are specialized to data, and the number of active discriminators is coherent to the number of classes in the dataset.

Original: For all authors...
Revised: We conduct experiments on MNIST with M = 10 , 20 , 40 using (cid:96) 1 regularization multiple times and observe that the accuracies (precision and recall) are very stable regardless of the number of discriminators for expert training as in Table 12. Note that we performed each experiment 4 times with random initialization. [Figure]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: C Computational overheads

Revised: Table 13 compares the training time per iteration and memory usage with the DCGAN [43] and

Revised: StyleGAN2 [44] backbone when training on CIFAR-10 dataset (32 × 32 sized) with the batch size of 64.

Revised: While required resources and training time increases by several folds for GMAN [9] with 5 or 10 discriminators, additional overheads are marginal for MCL-GAN compared to the singlediscriminator baseline setting due to feature sharing.

Revised: Note that the increases in overheads become more negligible in case of MCL-GAN with larger capacity of base architecture such as StyleGAN unlike GMAN that suffers from the scalability issue.

Revised: We used a machine with a Titan Xp GPU for the measurement.

Revised: [Table]

Revised: D Qualitative results D.1 Unconditional GAN on image datasets

Revised: We investigate the quality of images generated by MCL-GAN and compare its performance with

Revised: GMAN [9] together with real data on MNIST [39], Fashion-MNIST [40], CIFAR-10 [41] and

Original: (a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s
Revised: CelebA [42] in Figure 8. The samples are drawn randomly rather than cherry-picked. We observe clear differences between MCL-GAN and GMAN on MNIST and Fashion-MNIST. For MNIST, the generated images by GMAN are sometimes hard to recognize or look artiﬁcial (too thin and crisp) compared to real ones.

Original: contributions and scope?
Revised: The images in Fashion-MNIST are lacking in diversity; the types of generated bags and shoes are rather simple. Meanwhile, MCL-GAN generates the images that are faithful to the true distribution in semantics and diversity and are indistinguishable from real examples. For CIFAR-10, MCL-GAN generate relatively clear images and some of them are recognizable as vehicles or animals (see Figure 9) whereas most images obtained from GMAN look incomplete and noisy.

Revised: For CelebA, GMAN produces high quality images, but we discover more distorted and unnatural images than MCL-GAN.

Revised: Some selected examples from MCL-GAN with DCGAN backbone on

Revised: Fashion-MNIST, CIFAR-10, and CelebA are displayed in Figure 9.

Original: [Yes]
Revised: Figure10 shows random samples generated by MCL-GAN with StyleGAN2 backbone on CIFAR-10 and CelebA30K. D.2 Conditioned image synthesis Figure 11 and 12 qualitatively compare the diversity of the generated images between the baselines and MCL-GANs. For all methods including the baselines, mode-seeking regularization [11] is applied.

Original: (b) Did you describe the limitations of your work?
Revised: As shown in Figure 11(a), images generated by MCL-GAN have more variations in edges and expressions of dogs. Regarding Yosemitee results (Figure 11(b)) which the shapes of the contents are ﬁxed, colors are more diverse and vivid in MCL-GAN results. For Figure 12, we ﬁx the text code for each text description to remove the diversity effect of text embedding and produce images with the same set of latent vectors. MCL-GAN produces more diverse bird images, in terms of shape, orientation and size with high quality.

Original: [Yes] See Section 4.5 and supplementary
Revised: We present more qualitative results of MCL-GAN in Figureand 14. E Implementation details E.1 Synthetic data We reuse the experimental design and implementation 1 following [23]. E.2 Unconditional GAN on image datasets

Original: material.
Revised: DCGAN backbone We mostly follow the training convention proposed in DCGAN [43]. We use Adam optimizer [52] with β = (0 . 5 , 0 . 999) and set 64 and 128 as size of the mini-batch for real data and latent vectors, respectively. We use the same learning rate and temperature in balance loss for all networks, i .

Original: (c) Did you discuss any potential negative societal impacts of your work?
Revised: e ., lr = 0 . τ = 0 . 1 for LSGAN experiments and lr = 0 . 0002 , τ = 1 . 0 for the others. The weights for balance loss of discriminators, β d , is chosen in the range [0 . 05 , 1 . 0] and we choose the best performance.

Revised: For the weights for balance loss of generator, β g = 0 produces fairly good results on all cases while positive β g gives particularly signiﬁcant improvement in some

Revised: LSGAN and Hinge loss experiments.

Revised: We choose β g ∈ { 1 . 0 , 2 . 0 } for LSGAN experiments on all datasets and β g ∈ { 5 . 0 , 10 . 0 } for Hinge loss experiments on MNIST.

Revised: StyleGAN2 backbone We adopt the conﬁguration E architecture among the StyleGAN2 variations and use default hyperparameters for training using the ofﬁcial implementation 2 without applying data augmentation option.

Revised: We set the batch size at 64 and 16 for CIFAR-10 and CelebA30K, respectively.

Revised: GMAN settings We used the ofﬁcial implementation 3 of GMAN.

Revised: Among its varients, we use three versions that use the arithmetic mean of softmax, i .

Revised: e

Revised: ., GMAN-1, GMAN-0 and GMAN*, and choose the best scores among them to report in Table 1 and 2.

Revised: For differentiating discriminators, we apply different dropout rates in [0 . 4 , 0 . 6] and split of mini-batches for the input of discriminators while adopting the same architectures as DCGAN.

Revised: E.3 Conditioned image synthesis

Revised: We apply the MCL components to the ofﬁcial codes of DRIT ++ 4 , StackGAN ++ 5 and MSGANand use the default settings of their original implementations.

Revised: F Evaluation details F.1 Unconditional GAN on image datasets

Revised: Evaluation metrics We measure precision/recall based on Precision Recall Distribution (PRD) [46].

Revised: We adopt F 8 and F 1 / 8 scores from the PRD curve as a recall and precision of each model, respectively.

Revised: We use the ofﬁcial implementations of PRD 7 and FID 8 for the measurement.

Revised: DCGAN backbone We run the the experiment on MNIST [39], Fashion-MNIST [40], CIFAR10 [41] and CelebA [42] until 40, 50, 150 and 30 epochs, respectively.

Revised: We generate 60K random examples for MNIST and Fashion-MNIST and 50K random samples for the other datasets, and then compare them with the reference datasets with the same number of examples.

Revised: StyleGAN2 backbone We run the the experiment on CIFAR-10 [41] and CelebA30K [42] until 300 epochs and choose the best model in terms of FID.

Revised: We generate 50K and 30K random examples for CIFAR-10 and CelebA30K, respectively, and then compare them with the whole train (/validation) set.

Revised: We do not use the truncation trick when generating samples for quantitative evaluations.

Revised: F.2 Conditioned image synthesis

Revised: We measure FID, NDB/JSD 9 and LPIPS 10 using their ofﬁcial implementations.

Original: [Yes] Seesupplementary material. (d) Have you read the ethics review guidelines and ensured that your paper conforms tothem?
Revised: We follow all evaluation details in MSGAN [11] which is referenced for comparision. G Limitations The proposed method carries additional hyperparameters including the weights for several loss terms and the number of discriminators, and one might question the robustness of MCL-GAN with respect to the variations of the hyperparameters.

Original: [Yes]

Original: If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results? [Yes] See Section 4. (b) Did you include complete proofs of all theoretical results?
Revised: From our analysis of the hyperparameter setting, presented in Appendix B, the performance of the proposed method improves signiﬁcantly by the expert training and the balanced assignment of discriminators while the rest of the loss terms make stable contributions over a wide range of their weights.

Original: [N/A] If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or as a URL)?
Revised: Also, since MCL-GAN adjusts the number of active discriminators that participate in learning as experts, its performance is robust to the number of discriminators.

Original: [Yes] See supplementary material. (b) Did you specify all the training details (e
Revised: Although not included in the scope of this paper, there are some settings that need further investigation.

Original: g., data splits, hyperparameters, how theywere chosen)? [Yes] See Section 2 and supplementary material.
Revised: For example, how to extend the multi-discriminator environment with an extremely small dataset and compatibility with recent data augmentation techniques are not discovered but are worth studying.

Original: (c) Did you report error bars (e g., with respect to the random seed after running experiments multiple times)? [Yes] Partly.
Revised: Deep generative models have some potentials to be used for adverse or abusive applications.

Original: Section 5.3.
Revised: Although our work involves unconditional image generations based on face datasets, this is rather a generic framework based on GANs to mitigate the mode collapse and dropping problems hampering sample diversity.

Original: (d) Did you include the total amount of compute and  the type of resources used (e g., typeof GPUs, internal cluster, or cloud provider)?
Revised: Our algorithm is not directly related to particular applications with ethical issues, and we believe that the proposed approach can alleviate the bias and fairness issues by identifying the minority groups in a dataset effectively.

Original: [Yes] See supplementary material If you are using existing assets (e g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
Revised: PerceptualSimilarity

Original: [Yes]
Revised: [Figure]

Original: (b) Did you mention the license of the assets?

Original: [N/A]
Revised: [Figure]

Original: (c) Did you include any new assets either in the supplemental material or as a URL? [N/A] (d) Did you discuss whether and how consent was obtained from people whose data you’re using/curating?
Revised: [Figure]

Original: [N/A]
Revised: [Figure]

Original: (e) Did you discuss whether the data you are using/curating contains personally identiﬁable information or offensive content?
Revised: [Figure]

Original: [N/A]
Revised: [Figure]

Original: If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?
Revised: [Figure]

Original: [N/A]
Revised: [Figure]

Original: (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable? [N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount
Revised: [Table]

Original: spent on participant compensation?

Original: [N/A]
Revised: [Figure]