Original: MCL-GAN: Generative Adversarial Networks with Multiple Specialized Discriminators
Revised: MCL-GAN: Generative Adversarial Networks with Multiple Specialized Discriminators

Original: Abstract
Revised: Abstract

Original: We propose a framework of generative adversarial networks with multiple discriminators, which collaborate to represent a real dataset more effectively.
Revised: We propose a framework of generative adversarial networks with multiple discriminators, which collaborate to represent a real dataset more effectively.

Original: Our approach facilitates learning a generator consistent with the underlying data distribution based on real images and thus mitigates the chronic mode collapse problem.
Revised: Our approach facilitates learning a generator consistent with the underlying data distribution based on real images and thus mitigates the chronic mode collapse problem.

Original: From the inspiration of multiple choice learning, we guide each discriminator to have expertise in a subset of the entire data and allow the generator to ﬁnd reasonable correspondences between the latent and real data spaces automatically without extra supervision for training examples.
Revised: From the inspiration of multiple choice learning, we guide each discriminator to have expertise in a subset of the entire data and allow the generator to ﬁnd reasonable correspondences between the latent and real data spaces automatically without extra supervision for training examples.

Original: Despite the use of multiple discriminators, the backbone networks are shared across the discriminators and the increase in training cost is marginal.
Revised: Despite the use of multiple discriminators, the backbone networks are shared across the discriminators and the increase in training cost is marginal.

Original: We demonstrate the effectiveness of our algorithm using multiple evaluation metrics in the standard datasets for diverse tasks.
Revised: We demonstrate the effectiveness of our algorithm using multiple evaluation metrics in the standard datasets for diverse tasks.

Original:  Introduction
Revised:  Introduction

Original: With recent advances in deep generative models, Generative Adversarial Networks (GANs) [1] and Variational Autoencoders (VAEs) [2] have shown impressive achievements in generation of high-dimensional realistic images as well as various computer vision applications including image-toimage translation [3–5], image inpainting [6], image super-resolution [7], etc .
Revised: With recent advances in deep generative models, Generative Adversarial Networks (GANs) [1] and Variational Autoencoders (VAEs) [2] have shown impressive achievements in generation of high-dimensional realistic images as well as various computer vision applications including image-toimage translation [3–5], image inpainting [6], image super-resolution [7], etc .

Original: GANs have particularly received a lot of attention due to their interesting framework of minimax games, where two agents, a generator and a discriminator, compete against each other.
Revised: GANs have particularly received a lot of attention due to their interesting framework of minimax games, where two agents, a generator and a discriminator, compete against each other.

Original: Furthermore, compared to VAEs, GANs tend to produce acute, high-quality images.
Revised: Furthermore, compared to VAEs, GANs tend to produce acute, high-quality images.

Original: In theory, the generator learns the real data distribution by reaching an equilibrium point of the minimax game.
Revised: In theory, the generator learns the real data distribution by reaching an equilibrium point of the minimax game.

Original: However, in practice, the alternating training procedure does not guarantee the convergence to the optimal solution and often experiences mode collapsing, failing to cover the multiple modes of real data or, even worse, reaching trivial solutions.
Revised: However, in practice, the alternating training procedure does not guarantee the convergence to the optimal solution and often experiences mode collapsing, failing to cover the multiple modes of real data or, even worse, reaching trivial solutions.

Original: This paper focuses on the mode collapse problem in training GANs by adopting multiple collaborating discriminators.
Revised: This paper focuses on the mode collapse problem in training GANs by adopting multiple collaborating discriminators.

Original: Each discriminator learns to be specialized in a subset of reference data space, which is identiﬁed automatically during training, so the ensemble of discriminators provides not only the differentiation of fake data but also more accurate predictions over the clusters of real data.
Revised: Each discriminator learns to be specialized in a subset of reference data space, which is identiﬁed automatically during training, so the ensemble of discriminators provides not only the differentiation of fake data but also more accurate predictions over the clusters of real data.

Original: In this respect, a generator is encouraged to produce diverse modes that deceive a set of discriminators.
Revised: In this respect, a generator is encouraged to produce diverse modes that deceive a set of discriminators.

Original: To achieve these goals, we employ a Multiple Choice Learning (MCL)  framework to learn multiple discriminators and update the generator via a set of expert discriminators, where each discriminator is associated with a subset of the true and generated examples.
Revised: To achieve these goals, we employ a Multiple Choice Learning (MCL) [8] framework to learn multiple discriminators and update the generator via a set of expert discriminators, where each discriminator is associated with a subset of the true and generated examples.

Original: Our approach, based on a single generator and multiple discriminators, is called MCL-GAN, which is optimized by the standard objective of GANs combined with the objective of MCL on the discriminator side.
Revised: Our approach, based on a single generator and multiple discriminators, is called MCL-GAN, which is optimized by the standard objective of GANs combined with the objective of MCL on the discriminator side.

Original: There are several GAN literature that adopts multiple discriminators [8–13].
Revised: There are several GAN literature that adopts multiple discriminators [9–14].

Original: Among them, GMAN [9] is closely related to our approach in the sense that it utilizes an ensemble prediction of discriminators.
Revised: Among them, GMAN [10] is closely related to our approach in the sense that it utilizes an ensemble prediction of discriminators.

Original: It explores multi-discriminator extensions of GANs with diverse versions of the aggregated prediction of discriminators—from a harsh trainer to a lenient teacher with softened criteria.
Revised: It explores multi-discriminator extensions of GANs with diverse versions of the aggregated prediction of discriminators—from a harsh trainer to a lenient teacher with softened criteria.

Original: Meanwhile, there exist signiﬁcant differences in the method of ensembling from our approach.
Revised: Meanwhile, there exist signiﬁcant differences in the method of ensembling from our approach.

Original: While GMAN focuses on the loss to the generator with parallel learning of discriminators, our strategy specializes in each discriminator for more informative feedback to the generator.
Revised: While GMAN focuses on the loss to the generator with parallel learning of discriminators, our strategy specializes in each discriminator for more informative feedback to the generator.

Original: The training algorithm of the proposed approach is inspired by MCL [14], which is known to be effective in learning specialized models with high oracle accuracy in recognition tasks.
Revised: The training algorithm of the proposed approach is inspired by MCL [15], which is known to be effective in learning specialized models with high oracle accuracy in recognition tasks.

Original: Encouraged by this beneﬁt, [15–18] apply MCL or its variations [19,20] to the production of diverse and accurate outputs in several applications.
Revised: Encouraged by this beneﬁt, [16–19] apply MCL or its variations [20,21] to the production of diverse and accurate outputs in several applications.

Original: For instance, Mun et al .
Revised: For instance, Mun et al .

Original: [16] propose an MCL-KD framework to come up with a visual question answering (VQA) system based on multiple models that are specialized in different types of visual reasoning.
Revised: [17] propose an MCL-KD framework to come up with a visual question answering (VQA) system based on multiple models that are specialized in different types of visual reasoning.

Original: DiverseNet [17] introduces a control parameter as an input that diversiﬁes the outputs of the network with an MCL loss.
Revised: DiverseNet [18] introduces a control parameter as an input that diversiﬁes the outputs of the network with an MCL loss.

Original: While these works generate multiple outputs explicitly and select one of them or take their ensemble at inference time, our approach adopts a unique strategy for diversifying the mode with no additional cost at inference time.
Revised: While these works generate multiple outputs explicitly and select one of them or take their ensemble at inference time, our approach adopts a unique strategy for diversifying the mode with no additional cost at inference time.

Original: The proposed method takes an advantage of MCL for GANs, which has hardly been explored before.
Revised: The proposed method takes an advantage of MCL for GANs, which has hardly been explored before.

Original: Our approach does not require any supervision for model specialization, such as class labels and other conditions, unlike the aforementioned works.
Revised: Our approach does not require any supervision for model specialization, such as class labels and other conditions, unlike the aforementioned works.

Original: Our main contributions are summarized as follows:
Revised: Our main contributions are summarized as follows:

Original: • We propose a single-generator multi-discriminator GAN training algorithm to alleviate the mode collapse problem.
Revised: • We propose a single-generator multi-discriminator GAN training algorithm to alleviate the mode collapse problem.

Original: Our approach provides simple yet effective learning objectives based on MCL to achieve the goal.
Revised: Our approach provides simple yet effective learning objectives based on MCL to achieve the goal.

Original: • We present a balanced discriminator assignment strategy to facilitate the robust convergence of models and preserve the multi-modality of training data, where the number of discriminators is determined adaptively.
Revised: • We present a balanced discriminator assignment strategy to facilitate the robust convergence of models and preserve the multi-modality of training data, where the number of discriminators is determined adaptively.

Original: • The proposed method is applicable to many GAN variants since there is no constraint on network architectures or loss functions.
Revised: • The proposed method is applicable to many GAN variants since there is no constraint on network architectures or loss functions.

Original: Our method requires small extra overhead and trains models efﬁciently via feature sharing in the discriminators.
Revised: Our method requires small extra overhead and trains models efﬁciently via feature sharing in the discriminators.

Original:  Related work
Revised:  Related work

Original: This section discusses the mode collapse issue in GANs and the GAN models with multiple discriminators or generators.
Revised: This section discusses the mode collapse issue in GANs and the GAN models with multiple discriminators or generators.

Original: 2.1 Handling mode collapse for diversity
Revised: 2.1 Handling mode collapse for diversity

Original: Many variations of GANs propose either novel metrics for the discriminator loss or better alternatives to the discriminator design.
Revised: Many variations of GANs propose either novel metrics for the discriminator loss or better alternatives to the discriminator design.

Original: For example, LSGAN [21] substitutes the least square function for the binary cross-entropy loss as the discriminator objective.
Revised: For example, LSGAN [22] substitutes the least square function for the binary cross-entropy loss as the discriminator objective.

Original: WGAN [22] introduces a critic function based on the Earth-Mover’s distance rather than a binary classiﬁer, and WGAN-GP [23] improves WGAN by adding a gradient penalty term.
Revised: WGAN [23] introduces a critic function based on the Earth-Mover’s distance rather than a binary classiﬁer, and WGAN-GP [24] improves WGAN by adding a gradient penalty term.

Original: PacGAN [24] augments the discriminator’s input by packing samples for a single label.
Revised: PacGAN [25] augments the discriminator’s input by packing samples for a single label.

Original: EBGAN [25] models the discriminator as an energy function, which is optimized by the reconstruction loss as in autoencoders.
Revised: EBGAN [26] models the discriminator as an energy function, which is optimized by the reconstruction loss as in autoencoders.

Original: BiGAN[26], ALI [27], VEEGAN [28],  and
Revised: BiGAN [27], ALI [28], VEEGAN [29], and

Original: Inclusive GAN [29] also learn reconstruction networks.
Revised: Inclusive GAN [30] also learn reconstruction networks.

Original: In particular, VEEGAN [28] autoencodes the latent vectors to learn the inverse function of the generator, and maps both the true and generated data to the latent distribution, e .
Revised: In particular, VEEGAN [29] autoencodes the latent vectors to learn the inverse function of the generator, and maps both the true and generated data to the latent distribution, e .

Original: g ., a Gaussian distribution.
Revised: g ., a Gaussian distribution.

Original: Inclusive GAN [29] learns a generator by matching between real and fake examples in the feature space.
Revised: Inclusive GAN [30] learns a generator by matching between real and fake examples in the feature space.

Original: The mode collapse and diversity issues of generated outputs have been addressed explicitly in [11,30,31].
Revised: The mode collapse and diversity issues of generated outputs have been addressed explicitly in [12,31,32].

Original: They formulate the diversity metrics that encourage the mode exploration of the generators and derive the loss function using the metrics.
Revised: They formulate the diversity metrics that encourage the mode exploration of the generators and derive the loss function using the metrics.

Original: 2.2 GAN with multiple generators
Revised: 2.2 GAN with multiple generators

Original: Another line of research is the integration of multiple generators [32–35].
Revised: Another line of research is the integration of multiple generators [33–36].

Original: This approach represents a data distribution with a mixture model enforcing each generator to cover a portion of the whole data space.
Revised: This approach represents a data distribution with a mixture model enforcing each generator to cover a portion of the whole data space.

Original: It is naturally expected that mixture models approximate true distributions better than a single model, especially in high-dimensional spaces with multiple modes.
Revised: It is naturally expected that mixture models approximate true distributions better than a single model, especially in high-dimensional spaces with multiple modes.

Original: MAD-GAN [33] introduces an augmented classiﬁer as a discriminator, which predicts whether a sample is real and which generator the sample is drawn from, to encourage individual generators to learn distinctive modes.
Revised: MAD-GAN [34] introduces an augmented classiﬁer as a discriminator, which predicts whether a sample is real and which generator the sample is drawn from, to encourage individual generators to learn distinctive modes.

Original: MGAN [34] has similar strategies to MAD-GAN but constructs a separate branch in its discriminator to perform the two tasks.
Revised: MGAN [35] has similar strategies to MAD-GAN but constructs a separate branch in its discriminator to perform the two tasks.

Original: MEGAN [35] adopts a gating network that produces a one-hot vector to select the generator creating the best example.
Revised: MEGAN [36] adopts a gating network that produces a one-hot vector to select the generator creating the best example.

Original: P2GAN [36] sequentially adds a new generator to cover the missing modes of real data.
Revised: P2GAN [37] sequentially adds a new generator to cover the missing modes of real data.

Original: 2.3 GAN with multiple discriminators
Revised: 2.3 GAN with multiple discriminators

Original: Multiple discriminators are often employed to improve the performance of a single generator [8– 10,12].
Revised: Multiple discriminators are often employed to improve the performance of a single generator [9– 11,13].

Original: D2GAN [8] conducts a three-player minimax game, where two discriminators are trained for the completely opposite objectives, minimizing the Kullback-Leibler (KL) divergence and the inverse KL divergence between the true and generated data distributions.
Revised: D2GAN [9] conducts a three-player minimax game, where two discriminators are trained for the completely opposite objectives, minimizing the Kullback-Leibler (KL) divergence and the inverse KL divergence between the true and generated data distributions.

Original: The balancing of the two losses plays a role in seeking desirable and diverse modes.
Revised: The balancing of the two losses plays a role in seeking desirable and diverse modes.

Original: Albuquerque et al .
Revised: Albuquerque et al .

Original: [12] propose a general multi-objective optimization framework in the scenario with multiple discriminators.
Revised: [13] propose a general multi-objective optimization framework in the scenario with multiple discriminators.

Original: They present a hypervolume maximization algorithm to obtain weighted gradients.
Revised: They present a hypervolume maximization algorithm to obtain weighted gradients.

Original: Neyshabur et al .
Revised: Neyshabur et al .

Original: [13] train a GAN based on multiple projections.
Revised: [14] train a GAN based on multiple projections.

Original: Each discriminator determines a random low-dimensional projection of a sample to address the instability of GAN training in high-dimensions.
Revised: Each discriminator determines a random low-dimensional projection of a sample to address the instability of GAN training in high-dimensions.

Original: GMAN [9] presents diverse aggregation methods of multiple discriminators, where both hard and soft discriminator selection strategies are studied.
Revised: GMAN [10] presents diverse aggregation methods of multiple discriminators, where both hard and soft discriminator selection strategies are studied.

Original: Note that all the existing approaches learn the multiple discriminators independently and they may have strong correlations, which may not be appropriate for diversifying generated samples.
Revised: Note that all the existing approaches learn the multiple discriminators independently and they may have strong correlations, which may not be appropriate for diversifying generated samples.

Original: Our approach, however, assigns each sample to the best-suited discriminator through interactions among the discriminators, and, consequently, each discriminator becomes an expert model for the assigned examples.
Revised: Our approach, however, assigns each sample to the best-suited discriminator through interactions among the discriminators, and, consequently, each discriminator becomes an expert model for the assigned examples.

Original:  Multiple Choice Learning
Revised:  Multiple Choice Learning

Original: We present the main idea of MCL [37] and its extensions brieﬂy.
Revised: We present the main idea of MCL [8] and its extensions brieﬂy.

Original: Given a training dataset with N samples, D = { ( x i , y i ) } Ni =1 , M models, { f m } Mm =1 and a task-speciﬁc loss function, (cid:96) ( · , · ) , MCL minimizes the following oracle loss:
Revised: Given a training dataset with N samples, D = { ( x i , y i ) } Ni =1 , M models, { f m } Mm =1 and a task-speciﬁc loss function, (cid:96) ( · , · ) , MCL minimizes the following oracle loss:

Original: [Equation]
Revised: [Equation]

Original: In other words, only the model with the smallest error out of M candidates is selected for each example.
Revised: In other words, only the model with the smallest error out of M candidates is selected for each example.

Original: This optimization process makes each model f m ( · ) become an expert for a subset of D , thus leads to forming a natural cluster in D .
Revised: This optimization process makes each model f m ( · ) become an expert for a subset of D , thus leads to forming a natural cluster in D .

Original: A weakness of MCL is the possible mistakes caused by overconﬁdence issues.
Revised: A weakness of MCL is the possible mistakes caused by overconﬁdence issues.

Original: If non-specialized models make wrong predictions with high conﬁdence in the score aggregation process, the average scores are misleading and the ensemble model may result in poor-quality outputs.
Revised: If non-specialized models make wrong predictions with high conﬁdence in the score aggregation process, the average scores are misleading and the ensemble model may result in poor-quality outputs.

Original: To alleviate this limitation, Conﬁdent Multiple Choice Learning (CMCL) [19] adopts a conﬁdent oracle loss that enforces the predictions of a non-specialized model to be uniformly distributed using KL divergence, denoted by D KL .
Revised: To alleviate this limitation, Conﬁdent Multiple Choice Learning (CMCL) [20] adopts a conﬁdent oracle loss that enforces the predictions of a non-specialized model to be uniformly distributed using KL divergence, denoted by D KL .

Original: Assuming that f m ( · ) predicts the output distribution given data point x , i .
Revised: Assuming that f m ( · ) predicts the output distribution given data point x , i .

Original: e .,
Revised: e .,

Original: P m ( y | x ) , the loss is modiﬁed as
Revised: P m ( y | x ) , the loss is modiﬁed as

Original: [Equation]
Revised: [Equation]

Original: where U ( y ) is the uniform distribution and the ﬂag variable v i,m ∈ { 0 , 1 } allows the choices of the specialized models.
Revised: where U ( y ) is the uniform distribution and the ﬂag variable v i,m ∈ { 0 , 1 } allows the choices of the specialized models.

Original: Note that, if (cid:80) Mm =1 v i,m = k
Revised: Note that, if (cid:80) Mm =1 v i,m = k

Original: ( k < M ), each example is assigned to k models.
Revised: ( k < M ), each example is assigned to k models.

Original:  MCL-GAN
Revised:  MCL-GAN

Original: We describe our GAN extension with a generator G ( · ; θ ) and M discriminators { D m ( · ; φ m ) } M m =1 .
Revised: We describe our GAN extension with a generator G ( · ; θ ) and M discriminators { D m ( · ; φ m ) } M m =1 .

Original: Let p z and p d be the distributions in latent space and real data space, respectively.
Revised: Let p z and p d be the distributions in latent space and real data space, respectively.

Original: Given z ∼ p z , the generator produces a sample ˜ x
Revised: Given z ∼ p z , the generator produces a sample ˜ x

Original: = G ( z ; θ ) and each of the M discriminators makes a prediction on a real example x ∼ p d or the fake sample ˜ x .
Revised: = G ( z ; θ ) and each of the M discriminators makes a prediction on a real example x ∼ p d or the fake sample ˜ x .

Original: Each prediction, D m ( x ; φ m ) , ranges in [0 , 1] and represents the probability that x belongs to the true data distribution.
Revised: Each prediction, D m ( x ; φ m ) , ranges in [0 , 1] and represents the probability that x belongs to the true data distribution.

Original: We now describe the loss functions L D and L G for training discriminators and the generator, respectively.
Revised: We now describe the loss functions L D and L G for training discriminators and the generator, respectively.

Original: 4.1 Expert training
Revised: 4.1 Expert training

Original: Assuming that we draw N d real data and generate N g examples in each training batch, denoted by x and ˜ x , respectively, each network is trained as follows.
Revised: Assuming that we draw N d real data and generate N g examples in each training batch, denoted by x and ˜ x , respectively, each network is trained as follows.

Original: Discriminators Expert discriminators are the ones that predict the highest scores for each example.
Revised: Discriminators Expert discriminators are the ones that predict the highest scores for each example.

Original: With the indicator variable v i,m for a real example x i , the discriminators are trained to minimize the following loss function:
Revised: With the indicator variable v i,m for a real example x i , the discriminators are trained to minimize the following loss function:

Original: [Equation]
Revised: [Equation]

Original: where we choose k experts out of M discriminators for each example, i .
Revised: where we choose k experts out of M discriminators for each example, i .

Original: e ., (cid:80)
Revised: e ., (cid:80)

Original: Mm =1 v i,m = k .
Revised: Mm =1 v i,m = k .

Original: For a fake sample, since all discriminators have to identify it correctly, the following loss is added to (3):
Revised: For a fake sample, since all discriminators have to identify it correctly, the following loss is added to (3):

Original: [Equation]
Revised: [Equation]

Original: Generator We train the generator using the gradients received from the expert models to encourage the generator to ﬁnd the closest mode given z .
Revised: Generator We train the generator using the gradients received from the expert models to encourage the generator to ﬁnd the closest mode given z .

Original: With another indicator variable u j,m for z j , the expert loss for the generator is given by
Revised: With another indicator variable u j,m for z j , the expert loss for the generator is given by

Original: [Equation]
Revised: [Equation]

Original: 4.2 Non-expert training
Revised: 4.2 Non-expert training

Original: The non-expert discriminators should not be overconﬁdent to real examples while it is desirable to produce higher scores for real samples than fake ones.
Revised: The non-expert discriminators should not be overconﬁdent to real examples while it is desirable to produce higher scores for real samples than fake ones.

Original: For this requirement, we give a uniform soft label, e . g .,
Revised: For this requirement, we give a uniform soft label, e . g .,

Original: y =
Revised: y =

Original: 5] for non-expert discriminators and regularize them with a certain weight.
Revised: 5] for non-expert discriminators and regularize them with a certain weight.

Original: To be precise, we obtain the following non-expert loss term corresponding to (3):
Revised: To be precise, we obtain the following non-expert loss term corresponding to (3):

Original: [Equation]
Revised: [Equation]

Original: where v i,m is the same indicator variable deﬁned in (3) and (cid:96) ce ( · , y ) is the cross-entropy loss function given a target label y .
Revised: where v i,m is the same indicator variable deﬁned in (3) and (cid:96) ce ( · , y ) is the cross-entropy loss function given a target label y .

Original: The other counterpart for (5) is derived similarly as
Revised: The other counterpart for (5) is derived similarly as

Original: [Equation]
Revised: [Equation]

Original: The non-expert model training is effective to handle the overconﬁdence issue, but the model may still suffer from the data deﬁciency problem frequently happening in the standard MCL algorithms because each discriminator can observe only a subset of the whole dataset.
Revised: The non-expert model training is effective to handle the overconﬁdence issue, but the model may still suffer from the data deﬁciency problem frequently happening in the standard MCL algorithms because each discriminator can observe only a subset of the whole dataset.

Original: To alleviate this limitation, our discriminators share the parameters of all layers in the feature extractor while branching the last layer only.
Revised: To alleviate this limitation, our discriminators share the parameters of all layers in the feature extractor while branching the last layer only.

Original: This implementation is also sensible since the discriminators partially have the same objective to distinguish fake examples.
Revised: This implementation is also sensible since the discriminators partially have the same objective to distinguish fake examples.

Original: The common representations of all real samples are prone to be learned in the earlier layers despite being clustered in different subsets, whereas the critical information for high-level understanding is often encoded in deeper layers.
Revised: The common representations of all real samples are prone to be learned in the earlier layers despite being clustered in different subsets, whereas the critical information for high-level understanding is often encoded in deeper layers.

Original: Moreover, the number of model parameters and training time are saved signiﬁcantly while still taking advantage of ensemble learning.
Revised: Moreover, the number of model parameters and training time are saved signiﬁcantly while still taking advantage of ensemble learning.

Original: 4.3 Balanced assignment of discriminators
Revised: 4.3 Balanced assignment of discriminators

Original: On top of the adversarial losses, we introduce another loss for balanced updates of discriminators.
Revised: On top of the adversarial losses, we introduce another loss for balanced updates of discriminators.

Original: As there is no supervision about the specialization target of a discriminator, e .
Revised: As there is no supervision about the specialization target of a discriminator, e .

Original: g ., class labels or feature embeddings, it may be difﬁcult to reasonably distribute real samples to expert models from the beginning.
Revised: g ., class labels or feature embeddings, it may be difﬁcult to reasonably distribute real samples to expert models from the beginning.

Original: Since the abilities of individual discriminators are severely off-balanced, they are highly prone to assign all samples to a few speciﬁc models.
Revised: Since the abilities of individual discriminators are severely off-balanced, they are highly prone to assign all samples to a few speciﬁc models.

Original: Especially at an early phase of training, the model’s capability is more sensitive to the number of updates in the discriminators.
Revised: Especially at an early phase of training, the model’s capability is more sensitive to the number of updates in the discriminators.

Original: To tackle this challenge, we propose another loss, called the balance loss, which gives discriminators equal chances for updates.
Revised: To tackle this challenge, we propose another loss, called the balance loss, which gives discriminators equal chances for updates.

Original: Let the selection of expert discriminators approximately follow a categorical distribution with a parameter µ = [ µ 1 , . .
Revised: Let the selection of expert discriminators approximately follow a categorical distribution with a parameter µ = [ µ 1 , . .

Original: . , µ M ] .
Revised: . , µ M ] .

Original: Then the loss is computed by the KL divergence of the probability distribution of discriminators for being selected as experts from µ .
Revised: Then the loss is computed by the KL divergence of the probability distribution of discriminators for being selected as experts from µ .

Original: To obtain the probability for discriminator selection, we apply the softmax function to the vector of M predictions from discriminators for each example since the discriminator with the highest score is guaranteed to be chosen as an expert.
Revised: To obtain the probability for discriminator selection, we apply the softmax function to the vector of M predictions from discriminators for each example since the discriminator with the highest score is guaranteed to be chosen as an expert.

Original: We average these probability vectors over the training batch.
Revised: We average these probability vectors over the training batch.

Original: i .
Revised: i .

Original: e ., q = 1 N d (cid:80) N
Revised: e ., q = 1 N d (cid:80) N

Original: d
Revised: d

Original: i =1 s ([ D 1 ( x i ) , . . .
Revised: i =1 s ([ D 1 ( x i ) , . . .

Original: , D M ( x i )]; τ ) , where s ( · ; τ ) denotes a vector-valued softmax function with a temperature τ given a logit.
Revised: , D M ( x i )]; τ ) , where s ( · ; τ ) denotes a vector-valued softmax function with a temperature τ given a logit.

Original: To sum up, the balance loss is given by
Revised: To sum up, the balance loss is given by

Original: [Equation]
Revised: [Equation]

Original: In practice, we set µ m = 1 M , ∀ m to update the discriminators evenly, which is because the true distribution is unavailable.
Revised: In practice, we set µ m = 1 M , ∀ m to update the discriminators evenly, which is because the true distribution is unavailable.

Original: This assumption may not be congruent to the real distribution of the dataset and excessively forced assignment would not result in an optimal clustering for specialization.
Revised: This assumption may not be congruent to the real distribution of the dataset and excessively forced assignment would not result in an optimal clustering for specialization.

Original: We, therefore, decrease the weight for the balance loss gradually during training.
Revised: We, therefore, decrease the weight for the balance loss gradually during training.

Original: Eventually, each example will be naturally assigned to its best model with a very small weight of the balance loss.
Revised: Eventually, each example will be naturally assigned to its best model with a very small weight of the balance loss.

Original: This adjustment helps stabilize training and naturally cluster the reference data.
Revised: This adjustment helps stabilize training and naturally cluster the reference data.

Original: Note that the models are balanced within a few epochs and the weight reduction helps generate higher quality samples.
Revised: Note that the models are balanced within a few epochs and the weight reduction helps generate higher quality samples.

Original: Likewise, a small constraint on the distribution of the generator’s output facilitates balanced generation while preventing the statistics of generated samples from being skewed.
Revised: Likewise, a small constraint on the distribution of the generator’s output facilitates balanced generation while preventing the statistics of generated samples from being skewed.

Original: For this case, we use the distribution of the discriminators’ assignments q instead of arbitrarily chosen µ , i .
Revised: For this case, we use the distribution of the discriminators’ assignments q instead of arbitrarily chosen µ , i .

Original: e .,
Revised: e .,

Original: [Equation]
Revised: [Equation]

Original: where o = 1 N g
Revised: where o = 1 N g

Original: (cid:80) N g j =1 s
Revised: (cid:80) N g j =1 s

Original: [ D 1 ( G ( z j )) , . .
Revised: [ D 1 ( G ( z j )) , . .

Original: , D M ( G ( z j ))]; τ ) .
Revised: , D M ( G ( z j ))]; τ ) .

Original: 4.4 Total loss
Revised: 4.4 Total loss

Original: Altogether, the total losses for discriminators and the generator are summarized respectively as
Revised: Altogether, the total losses for discriminators and the generator are summarized respectively as

Original: [Equation]
Revised: [Equation]

Original: where α and { β d , β g } are hyperparameters.
Revised: where α and { β d , β g } are hyperparameters.

Original: Note that the proposed approach is applicable to any
Revised: Note that the proposed approach is applicable to any

Original: GAN formulations with the corresponding adversarial losses.
Revised: GAN formulations with the corresponding adversarial losses.

Original: 4.5 Choice of number of discriminators
Revised: 4.5 Choice of number of discriminators

Original: A remaining concern is how to ﬁnd the optimal number of discriminators while such information is not available in general as in many clustering tasks.
Revised: A remaining concern is how to ﬁnd the optimal number of discriminators while such information is not available in general as in many clustering tasks.

Original: If the number of discriminators is much larger than the optimal one, it is more desirable to focus on training a subset of discriminators than dividing the dataset into many minor clusters forcefully.
Revised: If the number of discriminators is much larger than the optimal one, it is more desirable to focus on training a subset of discriminators than dividing the dataset into many minor clusters forcefully.

Original: To ease this issue, we augment the (cid:96) 1 regularization loss to the discriminator side with a weight γ and encourage the sparsity in the discriminator selection for more desirable clustering results.
Revised: To ease this issue, we augment the (cid:96) 1 regularization loss to the discriminator side with a weight γ and encourage the sparsity in the discriminator selection for more desirable clustering results.

Original: Hence, even in the case that we are given an excessively large number of discriminators, our algorithm converges at good points by using a small subset of discriminators in practice.
Revised: Hence, even in the case that we are given an excessively large number of discriminators, our algorithm converges at good points by using a small subset of discriminators in practice.

Original: It is true that this strategy may not always lead to the optimal number of discriminators and have conﬂicts with the balance loss in (8).
Revised: It is true that this strategy may not always lead to the optimal number of discriminators and have conﬂicts with the balance loss in (8).

Original: However, the balance loss fades away as training goes on, and our model identiﬁes a proper number of clusters by deactivating a fraction of discriminators adaptively.
Revised: However, the balance loss fades away as training goes on, and our model identiﬁes a proper number of clusters by deactivating a fraction of discriminators adaptively.

Original: This sparsity loss would be useful when we learn on real datasets, where the examples are drawn from unknown distributions.
Revised: This sparsity loss would be useful when we learn on real datasets, where the examples are drawn from unknown distributions.

Original:  Experiments
Revised:  Experiments

Original: We evaluate the performance of MCL-GAN on unconditional and conditional image generation.
Revised: We evaluate the performance of MCL-GAN on unconditional and conditional image generation.

Original: In the results reported throughout this
Revised: In the results reported throughout this

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: 5.1 Results on synthetic datasets
Revised: 5.1 Results on synthetic datasets

Original: We ﬁrst perform toy experiments to verify the main idea of MCL-GAN intuitively.
Revised: We ﬁrst perform toy experiments to verify the main idea of MCL-GAN intuitively.

Original: We consider a mixture of 8 isotropic Gaussians in 2D whose centers are located on the circumference of a circle with a radius √ 2 while their standard deviation in each dimension is 0.05.
Revised: We consider a mixture of 8 isotropic Gaussians in 2D whose centers are located on the circumference of a circle with a radius √ 2 while their standard deviation in each dimension is 0.05.

Original: We employ 8 discriminators for training with the standard GAN loss while utilizing 2 discriminators with the Hinge loss [38].
Revised: We employ 8 discriminators for training with the standard GAN loss while utilizing 2 discriminators with the Hinge loss [38].

Original: We choose one expert discriminator for each sample ( k = 1 ) in all experiments.
Revised: We choose one expert discriminator for each sample ( k = 1 ) in all experiments.

Original: Figure 1 illustrates the snapshots of generated examples, given by the baseline methods and MCLGANs, through iterations.
Revised: Figure 1 illustrates the snapshots of generated examples, given by the baseline methods and MCLGANs, through iterations.

Original: While the base models ( M = 1 ) fail to cover all 8 modes, MCL-GANs manage to identify diverse modes quickly and produce samples belonging to all modes eventually.
Revised: While the base models ( M = 1 ) fail to cover all 8 modes, MCL-GANs manage to identify diverse modes quickly and produce samples belonging to all modes eventually.

Original: The models trained with the Hinge loss tend to generate more diverse samples even only with a single discriminator; MCL-GAN also requires less discriminators in the Hinge loss case to reconstruct the original data distribution.
Revised: The models trained with the Hinge loss tend to generate more diverse samples even only with a single discriminator; MCL-GAN also requires less discriminators in the Hinge loss case to reconstruct the original data distribution.

Original: Figure 2 illustrates how adaptively MCL-GANs select discriminators, given an excessive number of discriminators, e .
Revised: Figure 2 illustrates how adaptively MCL-GANs select discriminators, given an excessive number of discriminators, e .

Original: g .,
Revised: g .,

Original: M = 20 .
Revised: M = 20 .

Original: They utilize 8 and 16 expert discriminators when γ = 0 and 10 − 5 , respectively; each mode tends to be associated with 1 or 2 discriminators depending on the value of γ .
Revised: They utilize 8 and 16 expert discriminators when γ = 0 and 10 − 5 , respectively; each mode tends to be associated with 1 or 2 discriminators depending on the value of γ .

Original: We visualize the detailed mapping between examples and discriminators in Appendix A.1.
Revised: We visualize the detailed mapping between examples and discriminators in Appendix A.1.

Original: This observation implies that MCL-GAN covers all modes effectively while the (cid:96)loss helps identify the proper number of discriminators to generate high-ﬁdelity data.
Revised: This observation implies that MCL-GAN covers all modes effectively while the (cid:96)loss helps identify the proper number of discriminators to generate high-ﬁdelity data.

Original: 5.2 Unconditional GANs on image dataset 5.2.1 Experiment setup and evaluation protocol
Revised: 5.2 Unconditional GANs on image dataset 5.2.1 Experiment setup and evaluation protocol

Original: We run the unconditional GAN experiment on four distinct datasets including MNIST [39], FashionMNIST [40], CIFAR-10 [41] and CelebA [42], where two types of network architectures, DCGAN [43] and StyleGAN2 [44], are employed.
Revised: We run the unconditional GAN experiment on four distinct datasets including MNIST [39], FashionMNIST [40], CIFAR-10 [41] and CelebA [42], where two types of network architectures, DCGAN [43] and StyleGAN2 [44], are employed.

Original: Images are resized to 32 × 32 except for the CelebA dataset, for which DCGAN and StyleGAN2 adopt 64 × 64 and 128 × 128 images, respectively.
Revised: Images are resized to 32 × 32 except for the CelebA dataset, for which DCGAN and StyleGAN2 adopt 64 × 64 and 128 × 128 images, respectively.

Original: For the
Revised: For the

Original: StyleGAN2 experiments on CelebA, we use the ﬁrst and last 30K images from the “align&cropped” version of the train and validation splits following [29].
Revised: StyleGAN2 experiments on CelebA, we use the ﬁrst and last 30K images from the “align&cropped” version of the train and validation splits following [30].

Original: We apply our method to the DCGAN architecture with three different GAN loss functions: the vanilla GAN loss [1], the LSGAN loss [21], and the Hinge loss [38].
Revised: We apply our method to the DCGAN architecture with three different GAN loss functions: the vanilla GAN loss [1], the LSGAN loss [22], and the Hinge loss [38].

Original: Appendix C describes more details of our setting.
Revised: Appendix C describes more details of our setting.

Original: We adopt Precision Recall Distribution (PRD) [46] and Frèchet Inception Distance (FID) [47] as our evaluation metrics.
Revised: We adopt Precision Recall Distribution (PRD) [46] and Frèchet Inception Distance (FID) [47] as our evaluation metrics.

Original: The PRD curve provides a more credible assessment of generative models than
Revised: The PRD curve provides a more credible assessment of generative models than

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table]
Revised: [Table]

Original: single-valued metrics since it quantiﬁes the model in two folds—mode coverage (recall) and quality of generated samples (precision).
Revised: single-valued metrics since it quantiﬁes the model in two folds—mode coverage (recall) and quality of generated samples (precision).

Original: We measure the recall and precision of each model by computing the F 8 and F 1 / 8 scores from the PRD curve, respectively.
Revised: We measure the recall and precision of each model by computing the F 8 and F 1 / 8 scores from the PRD curve, respectively.

Original: We prefer higher scores and they are equal to 1 when the generated data distribution is identical to the reference.
Revised: We prefer higher scores and they are equal to 1 when the generated data distribution is identical to the reference.

Original: FID is a popular evaluation metric of generative models, based on the distance between two datasets with multivariate Gaussian assumption.
Revised: FID is a popular evaluation metric of generative models, based on the distance between two datasets with multivariate Gaussian assumption.

Original: Lower FID scores mean that generated samples are closer to the reference dataset.
Revised: Lower FID scores mean that generated samples are closer to the reference dataset.

Original: 5.2.2 DCGAN backbone
Revised: 5.2.2 DCGAN backbone

Original: Table 1 summarizes the precision and recall scores of our methods compared to other models with three different GAN objectives, when the number of discriminators is set to 5 or 10 ( M = 5 or 10 ) and the number of experts is 1 ( k = 1 ).
Revised: Table 1 summarizes the precision and recall scores of our methods compared to other models with three different GAN objectives, when the number of discriminators is set to 5 or 10 ( M = 5 or 10 ) and the number of experts is 1 ( k = 1 ).

Original: MCL-GAN achieves outstanding performance in terms of both metrics compared to the baseline and GMAN, which is an existing approach adopting multiple discriminators, on MNIST and CelebA. For Fashion-MNIST, we observe that MCL-GAN focuses on the mode coverage (diversity) while GMAN cares about image quality rather than diversity.
Revised: MCL-GAN achieves outstanding performance in terms of both metrics compared to the baseline and GMAN, which is an existing approach adopting multiple discriminators, on MNIST and CelebA. For Fashion-MNIST, we observe that MCL-GAN focuses on the mode coverage (diversity) while GMAN cares about image quality rather than diversity.

Original: As presented in Table 2, MCL-GAN outperforms the compared GAN models (based on DCGAN) by large margins in terms of FID scores on CIFAR-10, while being competitive with MGAN, which relies on multiple generators.
Revised: As presented in Table 2, MCL-GAN outperforms the compared GAN models (based on DCGAN) by large margins in terms of FID scores on CIFAR-10, while being competitive with MGAN, which relies on multiple generators.

Original: MCL-GAN also achieves better results than a clustering-based approach, e .
Revised: MCL-GAN also achieves better results than a clustering-based approach, e .

Original: g ., self-conditioned GAN [45], which requires additional computation due to clustering during training and the extra model parameters for conditioning the generator’s inputs with cluster membership.
Revised: g ., self-conditioned GAN [45], which requires additional computation due to clustering during training and the extra model parameters for conditioning the generator’s inputs with cluster membership.

Original: The results imply that MCL-GAN is effective to maintain multi-modality in the underlying distribution with a relatively small memory footprint and without extra supervision.
Revised: The results imply that MCL-GAN is effective to maintain multi-modality in the underlying distribution with a relatively small memory footprint and without extra supervision.

Original: To analyze the semantic quality of generated images, we present their classiﬁcation results given by the pretrained classiﬁers in Table 3.
Revised: To analyze the semantic quality of generated images, we present their classiﬁcation results given by the pretrained classiﬁers in Table 3.

Original: We measure how much the predicted label distribution in each
Revised: We measure how much the predicted label distribution in each

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: tested dataset deviates from the true (uniform) distribution using the KL divergence.
Revised: tested dataset deviates from the true (uniform) distribution using the KL divergence.

Original: We also calculate the average of class-wise LPIPS to measure the intra-class diversity of generated images.
Revised: We also calculate the average of class-wise LPIPS to measure the intra-class diversity of generated images.

Original: The overall results are favorable to MCL-GAN but there exist some misleading points.
Revised: The overall results are favorable to MCL-GAN but there exist some misleading points.

Original: For example, GMAN tends to achieve overly high class-wise LPIPS scores due to many out-of-distribution examples that confuse the classiﬁer, as illustrated in Appendix B.1.
Revised: For example, GMAN tends to achieve overly high class-wise LPIPS scores due to many out-of-distribution examples that confuse the classiﬁer, as illustrated in Appendix B.1.

Original: 5.2.3 StyleGAN2 backbone
Revised: 5.2.3 StyleGAN2 backbone

Original: Table 4 presents that MCL-GAN is also effective in the state-of-the-art backbone model and outperforms StyleGAN2 [44] and its variation, Inclusive GAN [29], in terms of all metrics.
Revised: Table 4 presents that MCL-GAN is also effective in the state-of-the-art backbone model and outperforms StyleGAN2 [44] and its variation, Inclusive GAN [30], in terms of all metrics.

Original: Inclusive
Revised: Inclusive

Original: GAN employs the sample-wise reconstruction loss by regarding each image as a mode.
Revised: GAN employs the sample-wise reconstruction loss by regarding each image as a mode.

Original: This strategy appears to improve recall but the model may suffer from sampling bias and scalability issues by estimating overly complex distributions.
Revised: This strategy appears to improve recall but the model may suffer from sampling bias and scalability issues by estimating overly complex distributions.

Original: The results imply that MCL-GAN enhances the convergence and the reproducibility of large GAN models, practically leading to improved performance.
Revised: The results imply that MCL-GAN enhances the convergence and the reproducibility of large GAN models, practically leading to improved performance.

Original: 5.2.4 Discriminators specialization
Revised: 5.2.4 Discriminators specialization

Original: Qualitative results Figure 3 qualitatively presents how successfully the discriminators in MCLGAN are specialized to subsets of datasets.
Revised: Qualitative results Figure 3 qualitatively presents how successfully the discriminators in MCLGAN are specialized to subsets of datasets.

Original: We learn the model with 10 discriminators, and illustrate the generated images with their memberships to discriminators; the images in the same row belong to the same discriminator.
Revised: We learn the model with 10 discriminators, and illustrate the generated images with their memberships to discriminators; the images in the same row belong to the same discriminator.

Original: We observe the semantic consistency of images within the same row in both
Revised: We observe the semantic consistency of images within the same row in both

Original: MNIST and Fashion-MNIST clearly.
Revised: MNIST and Fashion-MNIST clearly.

Original: Analysis using attribute annotations To examine how effective the discriminators represent the real data, we analyze the discriminator specialization statistics of MCL-GAN using the 40 binary attributes of each image in the CelebA dataset.
Revised: Analysis using attribute annotations To examine how effectively the discriminators represent the real data, we analyze the discriminator specialization statistics of MCL-GAN using thebinary attributes of each image in the CelebA dataset.

Original: We ﬁrst compute the normalized histogram of discriminator assignments for the images with a certain attribute, and then compute the cosine similarity between the histograms to obtain a 40 × 40 dimensional similarity matrix, where 40 is the number of attributes.
Revised: We ﬁrst compute the normalized histogram of discriminator assignments for the images with a certain attribute, and then compute the cosine similarity between the histograms to obtain a 40 × 40 dimensional similarity matrix, where 40 is the number of attributes.

Original: Figure 4 illustrates the similarity matrix, where we sample a subset of attributes for better visualization.
Revised: Figure 4 illustrates the similarity matrix, where we sample a subset of attributes for better visualization.

Original: We discover that the discriminator specialization statistics highly coincide with the latent semantic similarities of examples.
Revised: We discover that the discriminator specialization statistics highly coincide with the latent semantic similarities of examples.

Original: Speciﬁcally, the four attributes more relevant to female— Arched_Eyebrows, Wearing_Earings, Wearing_Lipstick, and Heavy_Makeup —are clustered together, while another set of four attributes such as Goatee, Male, Mustache, and Sideburns construct
Revised: Speciﬁcally, the four attributes more relevant to female— Arched_Eyebrows, Wearing_Earings, Wearing_Lipstick, and Heavy_Makeup —are clustered together, while another set of four attributes such as Goatee, Male, Mustache, and Sideburns construct

Original: A r c h e d _ E y e b r o w s
Revised: A r c h e d _ E y e b r o w s

Original: W e a r i n g _ E a rr i n g s
Revised: W e a r i n g _ E a rr i n g s

Original: W e a r i n g _ L i p s t i c k
Revised: W e a r i n g _ L i p s t i c k

Original: H e a v y _ M a k e u p B l o n d _ H a i r B l a c k _
Revised: H e a v y _ M a k e u p B l o n d _ H a i r B l a c k _

Original: H a i r G o a t e e M a l e
Revised: H a i r G o a t e e M a l e

Original: M u s t a c h e S
Revised: M u s t a c h e S

Original: i d e b u r n s
Revised: i d e b u r n s

Original: Arched_Eyebrows
Revised: Arched_Eyebrows

Original: [Figure]
Revised: [Figure]

Original: assignments of MCL-GAN on the CelebA dataset.
Revised: assignments of MCL-GAN on the CelebA dataset.

Original: [Table] [Table]
Revised: [Table] [Table]

Original: a strong group.
Revised: a strong group.

Original: These results support that the individual discriminators learned by MCL-GAN are specialized to reasonable subsets of the dataset
Revised: These results support that the individual discriminators learned by MCL-GAN are specialized to reasonable subsets of the dataset

Original: [Table]
Revised: [Table]

Original: which is helpful to generate more realistic images
Revised: which is helpful to generate more realistic images

Original: [Table]
Revised: [Table]

Original: 5.3 Conditional image synthesis
Revised: 5.3 Conditional image synthesis

Original: We apply MCL-GAN to image-to-image translation and text-to-image synthesis tasks, which require complex architectures to generate high-resolution images.
Revised: We apply MCL-GAN to image-to-image translation and text-to-image synthesis tasks, which require complex architectures to generate high-resolution images.

Original: In this experiment, we adopt MSGAN [11], a technique with a mode-seeking regularizer, as an additional component for alleviating the mode collapse in conditional GANs.
Revised: In this experiment, we adopt MSGAN [12], a technique with a mode-seeking regularizer, as an additional component for alleviating the mode collapse in conditional GANs.

Original: Then, we observe whether the mode seeking and our multiple choice learning create synergy, based on FID, NDB/JSD [48], and LPIPS [49] following [11].
Revised: Then, we observe whether the mode seeking and our multiple choice learning create synergy, based on FID, NDB/JSD [48], and LPIPS [49] following [12].

Original: Note that NDB counts the number of statistically different bins based on the clusters made by k -means clustering.
Revised: Note that NDB counts the number of statistically different bins based on the clusters made by k -means clustering.

Original: [Table]
Revised: [Table]

Original: Image-to-image translation We choose DRIT [4, 5], an unpaired image-to-image translation method based on the cycle consistency, as our baseline.
Revised: Image-to-image translation We choose DRIT [4, 5], an unpaired image-to-image translation method based on the cycle consistency, as our baseline.

Original: We employ MCL-GAN with M = 

Revised: We employ MCL-GAN with M =

Original: and k = 1 for distinguishing real and translated images.
Revised: and k = 1 for distinguishing real and translated images.

Original: As shown in Table 5, MCL-GAN signiﬁcantly improves the diversity measure, LPIPS, while achieving high-ﬁdelity data generation performance in terms of other metrics.
Revised: As shown in Table 5, MCL-GAN signiﬁcantly improves the diversity measure, LPIPS, while achieving high-ﬁdelity data generation performance in terms of other metrics.

Original: In particular, our approach works better on a more challenging task, cat (cid:10) dog, since it effectively handles the changes in both object shape and texture across domains by specializing discriminators to a subset of modes.
Revised: In particular, our approach works better on a more challenging task, cat (cid:10) dog, since it effectively handles the changes in both object shape and texture across domains by specializing discriminators to a subset of modes.

Original: Note that the integration of MCL is mostly beneﬁcial and the addition of the mode-seeking module often leads to extra performance gains.
Revised: Note that the integration of MCL is mostly beneﬁcial and the addition of the mode-seeking module often leads to extra performance gains.

Original: Text-to-image synthesis This experiment is based on StackGAN ++ [50] trained on CUB-2002011 [51] with a mode-seeking regularizer.
Revised: Text-to-image synthesis This experiment is based on StackGAN ++ [50] trained on CUB-2002011 [51] with a mode-seeking regularizer.

Original: StackGAN +
Revised: StackGAN +

Original: + has a hierarchical structure with a specialized pair of a discriminator and a generator to a certain image resolution.
Revised: + has a hierarchical structure with a specialized pair of a discriminator and a generator to a certain image resolution.

Original: We adopt its 3-stage version and trains an MCL-GAN with M = 3 and k = 1 only at the last stage, which handles images with size 256 × 256 .
Revised: We adopt its 3-stage version and trains an MCL-GAN with M = 3 and k = 1 only at the last stage, which handles images with size 256 × 256 .

Original: Table 6 illustrates that the integration of MCL improves performance consistently, especially in terms of the diversity measure, LPIPS, where we observe the merit of the mode-seeking regularizer via the combination with MCL.
Revised: Table 6 illustrates that the integration of MCL improves performance consistently, especially in terms of the diversity measure, LPIPS, where we observe the merit of the mode-seeking regularizer via the combination with MCL.

Original: 5.4 Discriminator assignment strategies
Revised: 5.4 Discriminator assignment strategies

Original: MCL-GAN is a model-agnostic ensemble algorithm with multiple discriminators, which is conceptually advantageous to other discriminator assignment strategies.
Revised: MCL-GAN is a model-agnostic ensemble algorithm with multiple discriminators, which is conceptually advantageous to other discriminator assignment strategies.

Original: In practice, our specialized discriminators to the subsets of training data outperforms independent training of discriminators on the whole dataset (as in GMAN) or different discriminator assignment strategies such as minimum-score discriminator selections, opposite to MCL-GAN, and random selections.
Revised: In practice, our specialized discriminators to the subsets of training data outperforms independent training of discriminators on the whole dataset (as in GMAN) or different discriminator assignment strategies such as minimum-score discriminator selections, opposite to MCL-GAN, and random selections.

Original: Also, the performance of
Revised: Also, the performance of

Original: MCL-GAN is as competitive as the method assigning discriminators based on the ground-truth class labels, which exhibits the reliability of the discriminator specialization by MCL.
Revised: MCL-GAN is as competitive as the method assigning discriminators based on the ground-truth class labels, which exhibits the reliability of the discriminator specialization by MCL.

Original: Table 7 presents that
Revised: Table 7 presents that

Original: MCL-GAN achieves outstanding performance compared to other strategies especially for the recall metric, even compared with GT-Assign without relying on class labels for discriminator specialization.
Revised: MCL-GAN achieves outstanding performance compared to other strategies especially for the recall metric, even compared with GT-Assign without relying on class labels for discriminator specialization.

Original: The proposed approach is also efﬁcient since it is free from any time-consuming clustering procedure for sample assignments to discriminators during training, and incurs marginal extra cost despite the use of multiple discriminators because the discriminators share the feature extractor.
Revised: The proposed approach is also efﬁcient since it is free from any time-consuming clustering procedure for sample assignments to discriminators during training, and incurs marginal extra cost despite the use of multiple discriminators because the discriminators share the feature extractor.

Original:  Conclusion
Revised:  Conclusion

Original: We presented a generative adversarial network framework with multiple discriminators, where each discriminator behaves as an expert classiﬁer and covers a separate mode in the underlying distribution.
Revised: We presented a generative adversarial network framework with multiple discriminators, where each discriminator behaves as an expert classiﬁer and covers a separate mode in the underlying distribution.

Original: This idea is implemented by incorporating the concept of multiple choice learning.
Revised: This idea is implemented by incorporating the concept of multiple choice learning.

Original: The combination of generative adversarial network and multiple choice learning turns out to be effective to alleviate the mode collapse problem.
Revised: The combination of generative adversarial network and multiple choice learning turns out to be effective to alleviate the mode collapse problem.

Original: Also, the integration of the sparsity loss encourages our model to identify the proper number of discriminators and estimate a desirable distribution with low complexity.
Revised: Also, the integration of the sparsity loss encourages our model to identify the proper number of discriminators and estimate a desirable distribution with low complexity.

Original: We demonstrated the effectiveness of the proposed algorithm on various GAN models and datasets.
Revised: We demonstrated the effectiveness of the proposed algorithm on various GAN models and datasets.

Original: Acknowledgements This work was partly supported by Samsung Electronics Co., Ltd., and the Institute of Information & communications Technology Planning & Evaluation (IITP) grants [No.20220-00959, (Part 2)
Revised: Acknowledgements This work was partly supported by Samsung Electronics Co., Ltd., and the Institute of Information & communications Technology Planning & Evaluation (IITP) grants [No.20220-00959, (Part 2)

Original: Few-Shot Learning of Causal Inference in Vision and Language for Decision
Revised: Few-Shot Learning of Causal Inference in Vision and Language for Decision

Original: Making; No.2021-0-01343, Artiﬁcial Intelligence Graduate School Program (Seoul National University)] and the National Research Foundation of Korea (NRF) grant [No.2022R1A5A708390811,Trustworthy Artiﬁcial Intelligence] funded by the Korean government (MSIT).
Revised: Making; No.2021-0-01343, Artiﬁcial Intelligence Graduate School Program (Seoul National University)] and the National Research Foundation of Korea (NRF) grant [No.2022R1A5A708390811,

Original: Appendix A Analysis A.1 Mapping between examples and discriminators on synthetic data

Original: Figure 5 illustrates how the generated samples from a synthetic dataset based on a mixture with 8 Gaussians are associated with 20 discriminators in the proposed MCL-GAN framework.

Original: Note that this ﬁgure is an extended version of Figure 2 in the main paper.

Original: The true data are represented in orange while other colors denote the expert discriminator coincides with individual examples.

Original: Without the (cid:96) 1 loss ( γ = 0 ), 16 discriminators are utilized to cluster the true distribution where exactly two discriminators are responsible for each mode; two distinct colors appear in each mode except the one (orange) corresponding to true data.

Original: By adding the (cid:96) 1 loss with a small coefﬁcient, we discover only 8 discriminators are active in training, one for each mode.

Original: [Figure]

Original: A.2 Hyperparameters

Original: [Figure] 1 Non-expert loss weight Table 8 shows the effect of non-expert training regularization when training 10 discriminators with the standard GAN loss on MNIST. The overall performance is robust to the variation of the hyperparameter α .
Revised: Trustworthy Artiﬁcial Intelligence] funded by the Korean government (MSIT).

Original: While all positive values of α improve the precision scores, which indicates that the overconﬁdence is reduced in non-expert discriminators, the best score is obtained with α = 0 .

Original: [Table]

Original: A.2.2 Balance loss weights

Original: We conduct the ablation studies on several ( β d , β g ) combinations for 10 discriminators with Hinge loss on MNIST.

Original: Table 9 shows that β d plays an important role in boosting the performance of GAN.

Original: This is mainly because β d is responsible for distributing the chances of being an expert to multiple discriminators; only a few discriminators are utilized in training if β d is zero or too small.

Original: β g has a smaller effect on the performance than β d .

Original: However, it helps improve recall scores without sacriﬁcing precision and leads to the best score when ( β d , β g ) =

Original: Note that the performance does not change drastically for all cases with the positive value of β d , which surpass a single discriminator

Original: GAN by large margins.

Original: [Table]

Original: A.2.3 Number of experts per example

Original: We evaluate our model by varying the number of experts k with M = 5 and

Original: [Table]

Original: and present the results on MNIST and CIFAR-10 in Table 10.

Original: The number of optimal k may be different in each dataset, however, choosing too many experts tend to drop the recall scores.

Original: Figure 6 shows how the specialization characteristics of discriminators differ by the number of experts per sample,

Original: i

Original: [Table]

Original: e ., k ∈ { 1 , 3 , 5 } , when there are 10 discriminators on MNIST and Fashion-MNIST.

Original: As k increases, the models get less specialized by sharing more data with each other so the subclusters become less distinctive.

Original: [Table]

Original: A.2.4 Number of discriminators and (cid:96) 1 loss weight

Original: We conduct the experiment under a various number of discriminators and illustrate the results in

Original: Table 11.

Original: It turns out that the performance of the proposed method is fairly robust to the number of discriminators quantitatively and adding the (cid:96) 1 loss does not incur noticeable differences in terms of precision/recall measure.

Original: However, interestingly, the (cid:96) 1 loss plays a crucial role in ﬁnding modes in the underlying distribution.

Original: Figure 7 illustrates the impact of the (cid:96) 1 loss on MNIST and FashionMNIST when we train the model with 40 discriminators.

Original: According to our results, only a fraction of the discriminators are specialized to data, and the number of active discriminators is coherent to the number of classes in the dataset.

Original: A.2.5 Stability to hyperparameter setting

Original: We conduct experiments on MNIST with M = 10 , 20 , 40 using (cid:96) 1 regularization multiple times and observe that the accuracies (precision and recall) are very stable regardless of the number of discriminators for expert training as in Table 12.

Original: Note that we performed each experiment 4 times with random initialization.

Original: [Figure]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: [Table]

Original: A.3 Computational Overheads

Original: Table 13 compares the training time per iteration and memory usage with the DCGAN [43] and

Original: StyleGAN2 [44] backbone when training on CIFAR-10 dataset (32 × 32 sized) with the batch size of 64.

Original: While required resources and training time increase by several folds for GMAN [9] with 5 or 10 discriminators, additional overheads are marginal for MCL-GAN compared to the singlediscriminator baseline setting due to feature sharing.

Original: Note that the increases in overheads become more negligible in case of MCL-GAN with a larger capacity of base architecture such as StyleGAN unlike GMAN that suffers from the scalability issue.

Original: We used a machine with a Titan Xp GPU for the measurement.

Original: [Table]

Original: B Qualitative Results B.1 Unconditional GAN on image datasets

Original: We investigate the quality of images generated by MCL-GAN and compare its performance with

Original: GMAN [9] together with real data on MNIST [39], Fashion-MNIST [40], CIFAR-10 [41] and

Original: CelebA [42] in Figure 8.

Original: The samples are drawn randomly rather than cherry-picked.

Original: We observe clear differences between MCL-GAN and GMAN on MNIST and Fashion-MNIST.

Original: For MNIST, the generated images by GMAN are sometimes hard to recognize or look artiﬁcial (too thin and crisp) compared to real ones.

Original: The images in Fashion-MNIST are lacking in diversity; the types of generated bags and shoes are rather simple.

Original: Meanwhile, MCL-GAN generates the images that are faithful to the true distribution in semantics and diversity and are indistinguishable from real examples.

Original: For CIFAR10, MCL-GAN generates relatively clear images and some of them are recognizable as vehicles or animals (see Figure 9) whereas most images obtained from GMAN look incomplete and noisy.

Original: For CelebA, GMAN produces high-quality images, but we discover more distorted and unnatural images than MCL-GAN.

Original: Some selected examples from MCL-GAN with DCGAN backbone on

Original: Fashion-MNIST, CIFAR-10, and CelebA are displayed in Figure 9.

Original: Figure10 shows random samples generated by MCL-GAN with StyleGAN2 backbone on CIFAR-10 and CelebA30K.

Original: B.2 Conditioned image synthesis

Original: Figure 11 and 12 qualitatively compare the diversity of the generated images between the baselines and MCL-GANs.

Original: For all methods including the baselines, mode-seeking regularization [11] is applied.

Original: As shown in Figure 11(a), images generated by MCL-GAN have more variations in edges and expressions of dogs.

Original: Regarding Yosemitee results (Figure 11(b)) which the shapes of the contents are ﬁxed, colors are more diverse and vivid in MCL-GAN results.

Original: For Figure 12, we ﬁx the text code for each text description to remove the diversity effect of text embedding and produce images with the same set of latent vectors.

Original: MCL-GAN produces more diverse bird images, in terms of shape, orientation and size with high quality.

Original: We present more qualitative results of MCL-GAN in Figureand 14.

Original: C Implementation Details C.1 Synthetic data

Original: We reuse the experimental design and implementation 1 following [23].

Original: C.2 Unconditional GANs on image datasets

Original: DCGAN backbone We mostly follow the training convention proposed in DCGAN [43].

Original: We use

Original: Adam optimizer [52] with β = (0 . 5 , 0 . 999) and set 64 and 128 as size of the mini-batch for real data and latent vectors, respectively.

Original: We use the same learning rate and temperature in balance loss for all networks, i .

Original: e ., lr = 0 .

Original: τ = 0 .

Original: 1 for LSGAN experiments and lr = 0 . 0002 , τ = 1 . 0 for the others.

Original: The weights for balance loss of discriminators, β d , is chosen in the range [0 . 05 , 1 . 0] and we choose the best performance.

Original: For the weights for balance loss of generator, β g = 0 produces fairly good results on all cases while positive β g gives particularly signiﬁcant improvement in some

Original: LSGAN and Hinge loss experiments.

Original: We choose β g ∈ { 1 . 0 , 2 . 0 } for LSGAN experiments on all datasets and β g ∈ { 5 . 0 , 10 . 0 } for Hinge loss experiments on MNIST.

Original: StyleGAN2 backbone We adopt the conﬁguration E architecture among the StyleGAN2 variations and use default hyperparameters for training using the ofﬁcial implementation 2 without applying data augmentation option.

Original: We set the batch size at 64 and 16 for CIFAR-10 and CelebA30K, respectively.

Original: GMAN settings We used the ofﬁcial implementation 3 of GMAN.

Original: Among its variants, we use three versions that use the arithmetic mean of softmax, i .

Original: e

Original: ., GMAN-1, GMAN-0 and GMAN*, and choose the best scores among them to report in Table 1 and 2.

Original: For differentiating discriminators, we apply different dropout rates in [0 . 4 , 0 . 6] and split of mini-batches for the input of discriminators while adopting the same architectures as DCGAN.

Original: C.3 Conditioned image synthesis

Original: We apply the MCL components to the ofﬁcial codes of DRIT ++ 4 , StackGAN ++ 5 and MSGANand use the default settings of their original implementations.

Original: D Evaluation Details D.1 Unconditional GANs on image datasets

Original: Evaluation metrics We measure precision/recall based on Precision Recall Distribution (PRD) [46].

Original: We adopt F 8 and F 1 / 8 scores from the PRD curve as a recall and precision of each model, respectively.

Original: We use the ofﬁcial implementations of PRD 7 and FID 8 for the measurement.

Original: DCGAN backbone We run the the experiment on MNIST [39], Fashion-MNIST [40], CIFAR10 [41] and CelebA [42] until 40, 50, 150 and 30 epochs, respectively.

Original: We generate 60K random examples for MNIST and Fashion-MNIST and 50K random samples for the other datasets, and then compare them with the reference datasets with the same number of examples.

Original: StyleGAN2 backbone We run the the experiment on CIFAR-10 [41] and CelebA30K [42] until 300 epochs and choose the best model in terms of FID.

Original: We generate 50K and 30K random examples for CIFAR-10 and CelebA30K, respectively, and then compare them with the whole train (/validation) set.

Original: We do not use the truncation trick when generating samples for quantitative evaluations.

Original: D.2 Conditioned image synthesis

Original: We measure FID, NDB/JSD 9 and LPIPS 10 using their ofﬁcial implementations.

Original: We follow all evaluation details in MSGAN [11] which is referenced for comparision.

Original: E Limitations

Original: The proposed method carries additional hyperparameters including the weights for several loss terms and the number of discriminators, and one might question the robustness of MCL-GAN with respect to the variations of the hyperparameters.

Original: From our analysis of the hyperparameter setting, presented in Appendix A.2, the performance of the proposed method improves signiﬁcantly by the expert training and the balanced assignment of discriminators while the rest of the loss terms make stable contributions over a wide range of their weights.

Original: Also, since MCL-GAN adjusts the number of active discriminators that participate in learning as experts, its performance is robust to the number of discriminators.

Original: Although not included in the scope of this paper, there are some settings that need further investigation.

Original: For example, how to extend the multi-discriminator environment with an extremely small dataset and compatibility with recent data augmentation techniques are not discovered but are worth studying.

Original: F Potential Societal Impact

Original: Deep generative models have some potentials to be used for adverse or abusive applications.

Original: Although our work involves unconditional image generations based on face datasets, this is rather a generic framework based on GANs to mitigate the mode collapse and dropping problems hampering sample diversity.

Original: Our algorithm is not directly related to particular applications with ethical issues, and we believe that the proposed approach can alleviate the bias and fairness issues by identifying the minority groups in a dataset effectively.

Original: [Figure]

Original: [Figure]

Original: [Figure]

Original: [Figure]

Original: [Figure]

Original: [Figure]

Original: [Figure]

Original: [Figure]

Original: [Table]

Original: [Figure]