Original: A CTIVATION M AXIMIZATION G ENERATIVE A DVER -
Revised: A CTIVATION M AXIMIZATION G ENERATIVE A DVER -

Original: SARIAL N ETS
Revised: SARIAL N ETS

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: The information about class labels has been empirically shown useful in improving the sample quality of generative adversarial nets (GANs).
Revised: Class labels have been empirically shown useful in improving the sample quality of generative adversarial nets (GANs).

Original: In this paper, we mathematically study the properties of the current variants of GANs that make use of class label information.
Revised: In this paper, we mathematically study the properties of the current variants of GANs that make use of class label information.

Original: With class aware gradient and cross-entropy decomposition, we reveal how class labels and associated losses inﬂuence GAN’s training.
Revised: With class aware gradient and cross-entropy decomposition, we reveal how class labels and associated losses inﬂuence GAN’s training.

Original: Based on that, we propose Activation Maximization Generative Adversarial Networks (AM-GAN) as an advanced solution.
Revised: Based on that, we propose Activation Maximization Generative Adversarial Networks (AM-GAN) as an advanced solution.

Original: A set of controlled experiments have been conducted to validate our analysis and evaluate the effectiveness of our solution, where AM-GAN outperforms other strong baselines and achieves state-of-the-art Inception Score (8.91) on CIFAR-10.
Revised: Comprehensive experiments have been conducted to validate our analysis and evaluate the effectiveness of our solution, where AM-GAN outperforms other strong baselines and achieves state-of-the-art Inception Score (8.91) on CIFAR-10.

Original: In addition, we demonstrate that, with the Inception ImageNet classiﬁer, Inception Score mainly tracks the diversity of the generator and there is, however, no reliable evidence that it can reﬂect the true sample quality.
Revised: In addition, we demonstrate that, with the Inception ImageNet classiﬁer, Inception Score mainly tracks the diversity of the generator, and there is, however, no reliable evidence that it can reﬂect the true sample quality.

Original: We thus propose a new metric, called AM Score to provide more accurate estimation on the sample quality.
Revised: We thus propose a new metric, called AM Score, to provide more accurate estimation on the sample quality.

Original: Our proposed model also outperforms the baseline methods in the new metric.
Revised: Our proposed model also outperforms the baseline methods in the new metric.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Generative adversarial nets (GANs) (Goodfellow et al., 2014) as a new way for learning generative models, has recently shown promising results in various challenging tasks, such as realistic image generation (Nguyen et al., 2016b;
Revised: Generative adversarial nets (GANs) (Goodfellow et al., 2014) as a new way for learning generative models, has recently shown promising results in various challenging tasks, such as realistic image generation (Nguyen et al., 2016b;

Original: Zhang et al., 2016;
Revised: Zhang et al., 2016;

Original: Gulrajani et al., 2017), conditional image generation (Huang et al., 2016b; Cao et al., 2017; Isola et al., 2016), image manipulation (Zhu et al., 2016) and text generation (Yu et al., 2016).
Revised: Gulrajani et al., 2017), conditional image generation (Huang et al., 2016b; Cao et al., 2017; Isola et al., 2016), image manipulation (Zhu et al., 2016) and text generation (Yu et al., 2016).

Original: Despite the great success, it is still challenging for the current GAN models to produce convincing samples when trained on datasets with high variability, even for image generation with low resolution, e.g., CIFAR-10.
Revised: Despite the great success, it is still challenging for the current GAN models to produce convincing samples when trained on datasets with high variability, even for image generation with low resolution, e.g., CIFAR-10.

Original: Meanwhile, people have empirically found taking advantages of class labels can signiﬁcantly improve the sample quality.
Revised: Meanwhile, people have empirically found taking advantages of class labels can signiﬁcantly improve the sample quality.

Original: There are three typical GAN models that make use of the label information: CatGAN (Springenberg, 2015) builds the discriminator as a multi-class classiﬁer; LabelGAN (Salimans et al., 2016) extends the discriminator with one extra class for the generated samples;
Revised: There are three typical GAN models that make use of the label information: CatGAN (Springenberg, 2015) builds the discriminator as a multi-class classiﬁer; LabelGAN (Salimans et al., 2016) extends the discriminator with one extra class for the generated samples;

Original: AC-GAN (Odena et al., 2016) jointly trains the real-fake discriminator and an auxiliary classiﬁer for the speciﬁc real classes.
Revised: AC-GAN (Odena et al., 2016) jointly trains the real-fake discriminator and an auxiliary classiﬁer for the speciﬁc real classes.

Original: By taking the class labels into account, these GAN models show improved generation quality and stability.
Revised: By taking the class labels into account, these GAN models show improved generation quality and stability.

Original: However, the mechanisms behind them have not been fully explored (Goodfellow, 2016).
Revised: However, the mechanisms behind them have not been fully explored (Goodfellow, 2016).

Original: In this paper, we mathematically study GAN models with the consideration of class labels.
Revised: In this paper, we mathematically study GAN models with the consideration of class labels.

Original: We derive the gradient of the generator’s loss w.r.t.
Revised: We derive the gradient of the generator’s loss w.r.t.

Original: class logits in the discriminator, named as class-aware gradient, for LabelGAN (Salimans et al., 2016) and further show its gradient tends to guide each generated sample towards being one of the speciﬁc real classes.
Revised: class logits in the discriminator, named as class-aware gradient, for LabelGAN (Salimans et al., 2016) and further show its gradient tends to guide each generated sample towards being one of the speciﬁc real classes.

Original: Moreover, we show that AC-GAN (Odena et al., 2016) can be viewed as a GAN model with hierarchical class discriminator.
Revised: Moreover, we show that AC-GAN (Odena et al., 2016) can be viewed as a GAN model with hierarchical class discriminator.

Original: Based on the analysis, we reveal some potential issues in the previous methods and accordingly propose a new method to resolve these issues.
Revised: Based on the analysis, we reveal some potential issues in the previous methods and accordingly propose a new method to resolve these issues.

Original: Speciﬁcally, we argue that a model with explicit target class would provide clearer gradient guidance to the generator than an implicit target class model like that in (Salimans et al., 2016).
Revised: Speciﬁcally, we argue that a model with explicit target class would provide clearer gradient guidance to the generator than an implicit target class model like that in (Salimans et al., 2016).

Original: Comparing with (Odena et al., 2016), we show that introducing the speciﬁc real class logits by replacing the overall real class logit in the discriminator usually works better than simply training an auxiliary classiﬁer.
Revised: Comparing with (Odena et al., 2016), we show that introducing the speciﬁc real class logits by replacing the overall real class logit in the discriminator usually works better than simply training an auxiliary classiﬁer.

Original: We argue that, in (Odena et al., 2016), adversarial training is missing in the auxiliary classiﬁer, which would make the model more likely to suffer mode collapse and produce low quality samples.
Revised: We argue that, in (Odena et al., 2016), adversarial training is missing in the auxiliary classiﬁer, which would make the model more likely to suffer mode collapse and produce low quality samples.

Original: We also experimentally ﬁnd that predeﬁned label tends to result in intra-class mode collapse and correspondingly propose dynamic labeling as a solution.
Revised: We also experimentally ﬁnd that predeﬁned label tends to result in intra-class mode collapse and correspondingly propose dynamic labeling as a solution.

Original: The proposed model is named as
Revised: The proposed model is named as

Original: Activation Maximization Generative Adversarial Networks (AM-GAN).
Revised: Activation Maximization Generative Adversarial Networks (AM-GAN).

Original: We empirically study the effectiveness of AM-GAN with a set of controlled experiments and the results are consistent with our analysis and, note that, AM-GAN achieves the state-of-the-art Inception Score (8.91) on CIFAR-10.
Revised: We empirically study the effectiveness of AM-GAN with a set of controlled experiments and the results are consistent with our analysis and, note that, AM-GAN achieves the state-of-the-art Inception Score (8.91) on CIFAR-10.

Original: In addition, through the experiments, we ﬁnd the commonly used metric needs further investigation.
Revised: In addition, through the experiments, we ﬁnd the commonly used metric needs further investigation.

Original: In our paper, we conduct a further study on the widely-used evaluation metric Inception Score (Salimans et al., 2016) and its extended metrics.
Revised: In our paper, we conduct a further study on the widely-used evaluation metric Inception Score (Salimans et al., 2016) and its extended metrics.

Original: We show that, with the Inception Model, Inception
Revised: We show that, with the Inception Model, Inception

Original: Score mainly tracks the diversity of generator, while there is no reliable evidence that it can measure the true sample quality.
Revised: Score mainly tracks the diversity of generator, while there is no reliable evidence that it can measure the true sample quality.

Original: We thus propose a new metric, called AM Score, to provide more accurate estimation on the sample quality as its compensation.
Revised: We thus propose a new metric, called AM Score, to provide more accurate estimation on the sample quality as its compensation.

Original: In terms of AM Score, our proposed method also outperforms other strong baseline methods.
Revised: In terms of AM Score, our proposed method also outperforms other strong baseline methods.

Original: The rest of this paper is organized as follows.
Revised: The rest of this paper is organized as follows.

Original: In Section 2, we introduce the notations and formulate the LabelGAN (Salimans et al., 2016) and AC-GAN ∗
Revised: In Section 2, we introduce the notations and formulate the LabelGAN (Salimans et al., 2016) and AC-GAN ∗

Original: Odena et al., 2016) as our baselines.
Revised: Odena et al., 2016) as our baselines.

Original: We then derive the class-aware gradient for LabelGAN, in Section 3, to reveal how class labels help its training.
Revised: We then derive the class-aware gradient for LabelGAN, in Section 3, to reveal how class labels help its training.

Original: In Section 4, we reveal the overlaid-gradient problem of LabelGAN and propose AM-GAN as a new solution, where we also analyze the properties of AM-GAN and build its connections to related work.
Revised: In Section 4, we reveal the overlaid-gradient problem of LabelGAN and propose AM-GAN as a new solution, where we also analyze the properties of AM-GAN and build its connections to related work.

Original: In Section 5, we introduce several important extensions, including the dynamic labeling as an alternative of predeﬁned labeling (i.e., class condition), the activation maximization view and a technique for enhancing the AC-GAN ∗ .
Revised: In Section 5, we introduce several important extensions, including the dynamic labeling as an alternative of predeﬁned labeling (i.e., class condition), the activation maximization view and a technique for enhancing the AC-GAN ∗ .

Original: We study Inception Score in Section 6 and accordingly propose a new metric AM Score.
Revised: We study Inception Score in Section 6 and accordingly propose a new metric AM Score.

Original: In Section 7, we empirically study AM-GAN and compare it to the baseline models with different metrics.
Revised: In Section 7, we empirically study AM-GAN and compare it to the baseline models with different metrics.

Original: Finally we conclude the paper and discuss the future work in
Revised: Finally we conclude the paper and discuss the future work in

Original: Section 8.
Revised: Section 8.

Original:  P RELIMINARIES
Revised:  P RELIMINARIES

Original: In the original GAN formulation (Goodfellow et al., 2014), the loss functions of the generator G and the discriminator D are given as:
Revised: In the original GAN formulation (Goodfellow et al., 2014), the loss functions of the generator G and the discriminator D are given as:

Original: [Equation]
Revised: [Equation]

Original: where D performs binary classiﬁcation between the real and the generated samples and D r ( x ) represents the probability of the sample x coming from the real data.
Revised: where D performs binary classiﬁcation between the real and the generated samples and D r ( x ) represents the probability of the sample x coming from the real data.

Original: 2. L ABEL GAN
Revised: 2. L ABEL GAN

Original: The framework (see Eq.
Revised: The framework (see Eq.

Original: (1)) has been generalized to multi-class case where each sample x has its associated class label y ∈ { 1 , . .
Revised: (1)) has been generalized to multi-class case where each sample x has its associated class label y ∈ { 1 , . .

Original: , K, K +1 } , and the K +1 th label corresponds to the generated samples (Salimans et al., 2016).
Revised: , K, K +1 } , and the K +1 th label corresponds to the generated samples (Salimans et al., 2016).

Original: Its loss functions are deﬁned as:
Revised: Its loss functions are deﬁned as:

Original: [Equation]
Revised: [Equation]

Original: where D i ( x ) denotes the probability of the sample x being class i .
Revised: where D i ( x ) denotes the probability of the sample x being class i .

Original: The loss can be written in the form of cross-entropy, which will simplify our later analysis:
Revised: The loss can be written in the form of cross-entropy, which will simplify our later analysis:

Original: [Equation]
Revised: [Equation]

Original: where D ( x ) =
Revised: where D ( x ) =

Original: [ D 1 ( x ) , D 2 ( x ) , ..., D K +1 ( x )] and v ( y ) =
Revised: [ D 1 ( x ) , D 2 ( x ) , ..., D K +1 ( x )] and v ( y ) =

Original: [ v 1 ( y ) , . . .
Revised: [ v 1 ( y ) , . . .

Original: , v K +

Revised: , v K +

Original: ( y )]
Revised: ( y )]

Original: with v i ( y )
Revised: with v i ( y )

Original: =
Revised: =

Original: if i (cid:54) =
Revised: if i (cid:54) =

Original: y and v i ( y )
Revised: y and v i ( y )

Original: = 1 if i = y .
Revised: = 1 if i = y .

Original: H is the cross-entropy, deﬁned as H ( p, q )
Revised: H is the cross-entropy, deﬁned as H ( p, q )

Original: =
Revised: =

Original: − (cid:80)
Revised: − (cid:80)

Original: i p
Revised: i p

Original: i log q i .
Revised: i log q i .

Original: We would refer the above model as LabelGAN (using class labels) throughout this paper.
Revised: We would refer the above model as LabelGAN (using class labels) throughout this paper.

Original: 2. AC-GAN ∗
Revised: 2. AC-GAN ∗

Original: Besides extending the original two-class discriminator as discussed in the above section, Odena et al.
Revised: Besides extending the original two-class discriminator as discussed in the above section, Odena et al.

Original: (2016) proposed an alternative approach, i.e., AC-GAN, to incorporate class label information, which introduces an auxiliary classiﬁer C for real classes in the original GAN framework.
Revised: (2016) proposed an alternative approach, i.e., AC-GAN, to incorporate class label information, which introduces an auxiliary classiﬁer C for real classes in the original GAN framework.

Original: With the core idea unchanged, we deﬁne a variant of AC-GAN as the following, and refer it as AC-GAN ∗ :
Revised: With the core idea unchanged, we deﬁne a variant of AC-GAN as the following, and refer it as AC-GAN ∗ :

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: x,
Revised: x,

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: where D r ( x ) and D f ( x ) = 1 − D r ( x ) are outputs of the binary discriminator which are the same as vanilla GAN, u ( · ) is the vectorizing operator that is similar to v ( · ) but deﬁned with K classes, and
Revised: where D r ( x ) and D f ( x ) = 1 − D r ( x ) are outputs of the binary discriminator which are the same as vanilla GAN, u ( · ) is the vectorizing operator that is similar to v ( · ) but deﬁned with K classes, and

Original: C ( x ) is the probability distribution over K real classes given by the auxiliary classiﬁer.
Revised: C ( x ) is the probability distribution over K real classes given by the auxiliary classiﬁer.

Original: In AC-GAN, each sample has a coupled target class y , and a loss on the auxiliary classiﬁer w.r.t.
Revised: In AC-GAN, each sample has a coupled target class y , and a loss on the auxiliary classiﬁer w.r.t.

Original: y is added to the generator to leverage the class label information.
Revised: y is added to the generator to leverage the class label information.

Original: We refer the losses on the auxiliary classiﬁer, i.e., Eq.
Revised: We refer the losses on the auxiliary classiﬁer, i.e., Eq.

Original: (7) and (9), as the auxiliary classiﬁer losses.
Revised: (7) and (9), as the auxiliary classiﬁer losses.

Original: The above formulation is a modiﬁed version of the original AC-GAN.
Revised: The above formulation is a modiﬁed version of the original AC-GAN.

Original: Speciﬁcally, we omit the auxiliary classiﬁer loss E ( x,y ) ∼ G [ H ( u ( y ) , C ( x ))] which encourages the auxiliary classiﬁer C to classify the fake sample x to its target class y .
Revised: Speciﬁcally, we omit the auxiliary classiﬁer loss E ( x,y ) ∼ G [ H ( u ( y ) , C ( x ))] which encourages the auxiliary classiﬁer C to classify the fake sample x to its target class y .

Original: Further discussions are provided in Section 5.3.
Revised: Further discussions are provided in Section 5.3.

Original: Note that we also adopt the − log( D r ( x )) loss in generator.
Revised: Note that we also adopt the − log( D r ( x )) loss in generator.

Original:  C LASS -A WARE G RADIENT
Revised:  C LASS -A WARE G RADIENT

Original: In this section, we introduce the class-aware gradient, i.e., the gradient of the generator’s loss w.r.t.
Revised: In this section, we introduce the class-aware gradient, i.e., the gradient of the generator’s loss w.r.t.

Original: class logits in the discriminator.
Revised: class logits in the discriminator.

Original: By analyzing the class-aware gradient of LabelGAN, we ﬁnd that the gradient tends to reﬁne each sample towards being one of the classes, which sheds some light on how the class label information helps the generator to improve the generation quality.
Revised: By analyzing the class-aware gradient of LabelGAN, we ﬁnd that the gradient tends to reﬁne each sample towards being one of the classes, which sheds some light on how the class label information helps the generator to improve the generation quality.

Original: Before delving into the details, we ﬁrst introduce the following lemma on the gradient properties of the cross-entropy loss to make our analysis clearer.
Revised: Before delving into the details, we ﬁrst introduce the following lemma on the gradient properties of the cross-entropy loss to make our analysis clearer.

Original: Lemma 1.
Revised: Lemma 1.

Original: With l being the logits vector and σ being the softmax function, let σ ( l ) be the current softmax probability distribution and ˆ p denote the target probability distribution, then
Revised: With l being the logits vector and σ being the softmax function, let σ ( l ) be the current softmax probability distribution and ˆ p denote the target probability distribution, then

Original: [Equation]
Revised: [Equation]

Original: For a generated sample x , the loss in LabelGAN is L lab G ( x )
Revised: For a generated sample x , the loss in LabelGAN is L lab G ( x )

Original: =
Revised: =

Original: H ([1 , 0] , [ D r ( x ) , D K +1 ( x )]) , as deﬁned in Eq.
Revised: H ([1 , 0] , [ D r ( x ) , D K +1 ( x )]) , as deﬁned in Eq.

Original: With Lemma 1, the gradient of L lab G ( x ) w.r.t.
Revised: With Lemma 1, the gradient of L lab G ( x ) w.r.t.

Original: the logits vector l ( x ) is given as:
Revised: the logits vector l ( x ) is given as:

Original: [Equation]
Revised: [Equation]

Original: With the above equations, the gradient of L lab G ( x ) w.r.t.
Revised: With the above equations, the gradient of L lab G ( x ) w.r.t.

Original: x is:
Revised: x is:

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: where
Revised: where

Original: [Equation]
Revised: [Equation]

Original: From the formulation, we ﬁnd that the overall gradient w.r.t.
Revised: From the formulation, we ﬁnd that the overall gradient w.r.t.

Original: a generated example x is 1 − D r ( x ) , which is the same as that in vanilla GAN (Goodfellow et al., 2014).
Revised: a generated example x is 1 − D r ( x ) , which is the same as that in vanilla GAN (Goodfellow et al., 2014).

Original: And the gradient on real classes is further distributed to each speciﬁc real class logit l k ( x ) according to its current probability ratio D k ( x ) D r ( x ) .
Revised: And the gradient on real classes is further distributed to each speciﬁc real class logit l k ( x ) according to its current probability ratio D k ( x ) D r ( x ) .

Original: [Figure]
Revised: [Figure]

Original: As such, the gradient naturally takes the label information into consideration: for a generated sample, higher probability of a certain class will lead to a larger step towards the direction of increasing the corresponding conﬁdence for the class.
Revised: As such, the gradient naturally takes the label information into consideration: for a generated sample, higher probability of a certain class will lead to a larger step towards the direction of increasing the corresponding conﬁdence for the class.

Original: Hence, individually, the gradient from the discriminator for each sample tends to reﬁne it towards being one of the classes in a probabilistic sense.
Revised: Hence, individually, the gradient from the discriminator for each sample tends to reﬁne it towards being one of the classes in a probabilistic sense.

Original: That is, each sample in LabelGAN is optimized to be one of the real classes, rather than simply to be real as in the vanilla GAN.
Revised: That is, each sample in LabelGAN is optimized to be one of the real classes, rather than simply to be real as in the vanilla GAN.

Original: We thus regard LabelGAN as an implicit target class model.
Revised: We thus regard LabelGAN as an implicit target class model.

Original: Reﬁning each generated sample towards one of the speciﬁc classes would help improve the sample quality.
Revised: Reﬁning each generated sample towards one of the speciﬁc classes would help improve the sample quality.

Original: Recall that there are similar inspirations in related work.
Revised: Recall that there are similar inspirations in related work.

Original: Denton et al.
Revised: Denton et al.

Original: (2015) showed that the result could be signiﬁcantly better if GAN is trained with separated classes.
Revised: (2015) showed that the result could be signiﬁcantly better if GAN is trained with separated classes.

Original: And AC-GAN (Odena et al., 2016) introduces an extra loss that forces each sample to ﬁt one class and achieves a better result.
Revised: And AC-GAN (Odena et al., 2016) introduces an extra loss that forces each sample to ﬁt one class and achieves a better result.

Original:  T HE P ROPOSED M ETHOD
Revised:  T HE P ROPOSED M ETHOD

Original: In LabelGAN, the generator gets its gradients from the K speciﬁc real class logits in discriminator and tends to reﬁne each sample towards being one of the classes.
Revised: In LabelGAN, the generator gets its gradients from the K speciﬁc real class logits in discriminator and tends to reﬁne each sample towards being one of the classes.

Original: However, LabelGAN actually suffers from the overlaid-gradient problem: all real class logits are encouraged at the same time.
Revised: However, LabelGAN actually suffers from the overlaid-gradient problem: all real class logits are encouraged at the same time.

Original: Though it tends to make each sample be one of these classes during the training, the gradient of each sample is a weighted averaging over multiple label predictors.
Revised: Though it tends to make each sample be one of these classes during the training, the gradient of each sample is a weighted averaging over multiple label predictors.

Original: As illustrated in Figure 1, the averaged gradient may be towards none of these classes.
Revised: As illustrated in Figure 1, the averaged gradient may be towards none of these classes.

Original: In multi-exclusive classes setting, each valid sample should only be classiﬁed to one of classes by the discriminator with high conﬁdence.
Revised: In multi-exclusive classes setting, each valid sample should only be classiﬁed to one of classes by the discriminator with high conﬁdence.

Original: One way to resolve the above problem is to explicitly assign each generated sample a single speciﬁc class as its target.
Revised: One way to resolve the above problem is to explicitly assign each generated sample a single speciﬁc class as its target.

Original: 4. AM-GAN
Revised: 4. AM-GAN

Original: Assigning each sample a speciﬁc target class y , the loss functions of the revised-version LabelGAN can be formulated as:
Revised: Assigning each sample a speciﬁc target class y , the loss functions of the revised-version LabelGAN can be formulated as:

Original: [Equation]
Revised: [Equation]

Original: where v ( y ) is with the same deﬁnition as in Section 2.1.
Revised: where v ( y ) is with the same deﬁnition as in Section 2.1.

Original: The model with aforementioned formulation is named as Activation Maximization Generative Adversarial Networks (AM-GAN) in our paper.
Revised: The model with aforementioned formulation is named as Activation Maximization Generative Adversarial Networks (AM-GAN) in our paper.

Original: And the further interpretation towards naming will be in Section 5.2.
Revised: And the further interpretation towards naming will be in Section 5.2.

Original: The only difference between
Revised: The only difference between

Original: AM-GAN and LabelGAN lies in the generator’s loss function.
Revised: AM-GAN and LabelGAN lies in the generator’s loss function.

Original: Each sample in AM-GAN has a speciﬁc target class, which resolves the overlaid-gradient problem.
Revised: Each sample in AM-GAN has a speciﬁc target class, which resolves the overlaid-gradient problem.

Original: AC-GAN (Odena et al., 2016) also assigns each sample a speciﬁc target class, but we will show that the AM-GAN and AC-GAN are substantially different in the following part of this section.
Revised: AC-GAN (Odena et al., 2016) also assigns each sample a speciﬁc target class, but we will show that the AM-GAN and AC-GAN are substantially different in the following part of this section.

Original: 4. L ABEL GAN + A UXILIARY C LASSIFIER
Revised: 4. L ABEL GAN + A UXILIARY C LASSIFIER

Original: Both LabelGAN and AM-GAN are GAN models with K +1 classes.
Revised: Both LabelGAN and AM-GAN are GAN models with K +1 classes.

Original: We introduce the following cross-entropy decomposition lemma to build their connections to GAN models with two classes and the K -classes models (i.e., the auxiliary classiﬁers).
Revised: We introduce the following cross-entropy decomposition lemma to build their connections to GAN models with two classes and the K -classes models (i.e., the auxiliary classiﬁers).

Original: Lemma 2.
Revised: Lemma 2.

Original: Given v =
Revised: Given v =

Original: [ v 1 , . .
Revised: [ v 1 , . .

Original: . , v K +1 ] , v 1: K (cid:44) [ v 1 , . .
Revised: . , v K +1 ] , v 1: K (cid:44) [ v 1 , . .

Original: . , v K ] , v r (cid:44) (cid:80) Kk
Revised: . , v K ] , v r (cid:44) (cid:80) Kk

Original: =1 v k , R ( v ) (cid:44) v 1: K /v r and
Revised: =1 v k , R ( v ) (cid:44) v 1: K /v r and

Original: F ( v ) (cid:44) [ v r , v K +1 ] , let ˆ p =
Revised: F ( v ) (cid:44) [ v r , v K +1 ] , let ˆ p =

Original: [ˆ p 1 , . . .
Revised: [ˆ p 1 , . . .

Original: , ˆ p K +1 ] , p
Revised: , ˆ p K +1 ] , p

Original: =
Revised: =

Original: [ p 1 , . .
Revised: [ p 1 , . .

Original: , p K +1 ] , then we have
Revised: , p K +1 ] , then we have

Original: [Equation]
Revised: [Equation]

Original: With Lemma 2, the loss function of the generator in AM-GAN can be decomposed as follows:
Revised: With Lemma 2, the loss function of the generator in AM-GAN can be decomposed as follows:

Original: [Equation]
Revised: [Equation]

Original: AuxiliaryClassiﬁerGLoss
Revised: AuxiliaryClassiﬁerGLoss

Original: [Equation]
Revised: [Equation]

Original: The second term of Eq.
Revised: The second term of Eq.

Original: (17) actually equals to the loss function of the generator in LabelGAN:
Revised: (17) actually equals to the loss function of the generator in LabelGAN:

Original: [Equation]
Revised: [Equation]

Original: Similar analysis can be adapted to the ﬁrst term and the discriminator.
Revised: Similar analysis can be adapted to the ﬁrst term and the discriminator.

Original: Note that v r ( x ) equals to one.
Revised: Note that v r ( x ) equals to one.

Original: Interestingly, we ﬁnd by decomposing the AM-GAN losses, AM-GAN can be viewed as a combination of LabelGAN and auxiliary classiﬁer (deﬁned in Section 2.2).
Revised: Interestingly, we ﬁnd by decomposing the AM-GAN losses, AM-GAN can be viewed as a combination of LabelGAN and auxiliary classiﬁer (deﬁned in Section 2.2).

Original: From the decomposition perspective, disparate to AM-GAN, AC-GAN is a combination of vanilla GAN and the auxiliary classiﬁer.
Revised: From the decomposition perspective, disparate to AM-GAN, AC-GAN is a combination of vanilla GAN and the auxiliary classiﬁer.

Original: The auxiliary classiﬁer loss in Eq.
Revised: The auxiliary classiﬁer loss in Eq.

Original: (17) can also be viewed as the cross-entropy version of generator loss in CatGAN: the generator of CatGAN directly optimizes entropy H ( R ( D ( x ))) to make each sample have a high conﬁdence of being one of the classes, while AM-GAN achieves this by the ﬁrst term of its decomposed loss H ( R ( v ( x )) , R ( D ( x ))) in terms of cross-entropy with given target distribution.
Revised: (17) can also be viewed as the cross-entropy version of generator loss in CatGAN: the generator of CatGAN directly optimizes entropy H ( R ( D ( x ))) to make each sample have a high conﬁdence of being one of the classes, while AM-GAN achieves this by the ﬁrst term of its decomposed loss H ( R ( v ( x )) , R ( D ( x ))) in terms of cross-entropy with given target distribution.

Original: That is, the AM-GAN is the combination of the cross-entropy version of CatGAN and
Revised: That is, the AM-GAN is the combination of the cross-entropy version of CatGAN and

Original: LabelGAN.
Revised: LabelGAN.

Original: We extend the discussion between AM-GAN and CatGAN in the Appendix B.
Revised: We extend the discussion between AM-GAN and CatGAN in the Appendix B.

Original: 4. N ON -H IERARCHICAL M
Revised: 4. N ON -H IERARCHICAL M

Original: ODEL
Revised: ODEL

Original: With the Lemma 2, we can also reformulate the AC-GAN ∗ as a K +1 classes model.
Revised: With the Lemma 2, we can also reformulate the AC-GAN ∗ as a K +1 classes model.

Original: Take the generator’s loss function as an example:
Revised: Take the generator’s loss function as an example:

Original: [Equation]
Revised: [Equation]

Original: In the K +1 classes model, the K +1 classes distribution is formulated as [ D r ( x ) · C ( x ) , D f ( x )] .
Revised: In the K +1 classes model, the K +1 classes distribution is formulated as [ D r ( x ) · C ( x ) , D f ( x )] .

Original: AC-GAN introduces the auxiliary classiﬁer in the consideration of leveraging the side information of class label, it turns out that the formulation of AC-GAN ∗ can be viewed as a hierarchical K + classes model consists of a two-class discriminator and a K -class auxiliary classiﬁer, as illustrated in
Revised: AC-GAN introduces the auxiliary classiﬁer in the consideration of leveraging the side information of class label, it turns out that the formulation of AC-GAN ∗ can be viewed as a hierarchical K + classes model consists of a two-class discriminator and a K -class auxiliary classiﬁer, as illustrated in

Original: Figure 2.
Revised: Figure 2.

Original: Conversely, AM-GAN is a non-hierarchical model.
Revised: Conversely, AM-GAN is a non-hierarchical model.

Original: All K +1 classes stay in the same level of the discriminator in AM-GAN.
Revised: All K +1 classes stay in the same level of the discriminator in AM-GAN.

Original: In the hierarchical model AC-GAN ∗ , adversarial training is only conducted at the real-fake twoclass level, while misses in the auxiliary classiﬁer.
Revised: In the hierarchical model AC-GAN ∗ , adversarial training is only conducted at the real-fake twoclass level, while misses in the auxiliary classiﬁer.

Original: Adversarial training is the key to the theoretical guarantee of global convergence p G = p data .
Revised: Adversarial training is the key to the theoretical guarantee of global convergence p G = p data .

Original: Taking the original GAN formulation as an instance, if generated samples collapse to a certain point x , i.e., p G ( x ) > p data ( x ) , then there must exit another point x (cid:48) with p G ( x (cid:48) )
Revised: Taking the original GAN formulation as an instance, if generated samples collapse to a certain point x , i.e., p G ( x ) > p data ( x ) , then there must exit another point x (cid:48) with p G ( x (cid:48) )

Original: <
Revised: <

Original: p data ( x (cid:48) ) .
Revised: p data ( x (cid:48) ) .

Original: Given the optimal D ( x ) = p data ( x ) p G ( x )+ p data ( x ) , the collapsed point x will get a relatively lower score.
Revised: Given the optimal D ( x ) = p data ( x ) p G ( x )+ p data ( x ) , the collapsed point x will get a relatively lower score.

Original: And with the existence of higher score points (e.g. x (cid:48) ), maximizing the generator’s expected score, in theory, has the strength to recover from the mode-collapsed state.
Revised: And with the existence of higher score points (e.g. x (cid:48) ), maximizing the generator’s expected score, in theory, has the strength to recover from the mode-collapsed state.

Original: In practice, the p G and p data are usually disjoint (Arjovsky & Bottou, 2017), nevertheless, the general behaviors stay the same: when samples collapse to a certain point, they are more likely to get a relatively lower score from the adversarial network.
Revised: In practice, the p G and p data are usually disjoint (Arjovsky & Bottou, 2017), nevertheless, the general behaviors stay the same: when samples collapse to a certain point, they are more likely to get a relatively lower score from the adversarial network.

Original: Without adversarial training in the auxiliary classiﬁer, a mode-collapsed generator would not get any penalties from the auxiliary classiﬁer loss.
Revised: Without adversarial training in the auxiliary classiﬁer, a mode-collapsed generator would not get any penalties from the auxiliary classiﬁer loss.

Original: In our experiments, we ﬁnd AC-GAN is more likely to get mode-collapsed, and it was empirically found reducing the weight (such as 0.1 used in Gulrajani et al.
Revised: In our experiments, we ﬁnd AC-GAN is more likely to get mode-collapsed, and it was empirically found reducing the weight (such as 0.1 used in Gulrajani et al.

Original: (2017)) of the auxiliary classiﬁer losses would help.
Revised: (2017)) of the auxiliary classiﬁer losses would help.

Original: In Section 5.3, we introduce an extra adversarial training in the auxiliary classiﬁer with which we improve AC-GAN ∗ ’s training stability and sample-quality in experiments.
Revised: In Section 5.3, we introduce an extra adversarial training in the auxiliary classiﬁer with which we improve AC-GAN ∗ ’s training stability and sample-quality in experiments.

Original: On the contrary, AM-GAN, as a non-hierarchical model, can naturally conduct adversarial training among all the class logits.
Revised: On the contrary, AM-GAN, as a non-hierarchical model, can naturally conduct adversarial training among all the class logits.

Original:  E XTENSIONS 5. D YNAMIC L ABELING
Revised:  E XTENSIONS 5. D YNAMIC L ABELING

Original: In the above section, we simply assume each generated sample has a target class.
Revised: In the above section, we simply assume each generated sample has a target class.

Original: One possible solution is like AC-GAN (Odena et al., 2016), predeﬁning each sample a class label, which substantially results in a conditional GAN.
Revised: One possible solution is like AC-GAN (Odena et al., 2016), predeﬁning each sample a class label, which substantially results in a conditional GAN.

Original: Actually, we could assign each sample a target class according to its current probability estimated by the discriminator.
Revised: Actually, we could assign each sample a target class according to its current probability estimated by the discriminator.

Original: A natural choice could be the class which is of the maximal probability currently: y ( x ) (cid:44) argmax i ∈{ 1 ,...,K } D i ( x ) for each generated sample x .
Revised: A natural choice could be the class which is of the maximal probability currently: y ( x ) (cid:44) argmax i ∈{ 1 ,...,K } D i ( x ) for each generated sample x .

Original: We name this dynamic labeling.
Revised: We name this dynamic labeling.

Original: According to our experiments, dynamic labeling brings important improvements to AM-GAN, and is applicable to other models that require target class for each generated sample, e.g. AC-GAN, as an alternative to predeﬁned labeling.
Revised: According to our experiments, dynamic labeling brings important improvements to AM-GAN, and is applicable to other models that require target class for each generated sample, e.g. AC-GAN, as an alternative to predeﬁned labeling.

Original: We experimentally ﬁnd GAN models with pre-assigned class label tend to encounter intra-class mode collapse.
Revised: We experimentally ﬁnd GAN models with pre-assigned class label tend to encounter intra-class mode collapse.

Original: In addition, with dynamic labeling, the GAN model remains generating from pure random noises, which has potential beneﬁts, e.g. making smooth interpolation across classes in the latent space practicable.
Revised: In addition, with dynamic labeling, the GAN model remains generating from pure random noises, which has potential beneﬁts, e.g. making smooth interpolation across classes in the latent space practicable.

Original: 5. T HE A CTIVATION M AXIMIZATION V IEW
Revised: 5. T HE A CTIVATION M AXIMIZATION V IEW

Original: Activation maximization is a technique which is traditionally applied to visualize the neuron(s) of pretrained neural networks (Nguyen et al., 2016a;b; Erhan et al., 2009).
Revised: Activation maximization is a technique which is traditionally applied to visualize the neuron(s) of pretrained neural networks (Nguyen et al., 2016a;b; Erhan et al., 2009).

Original: The GAN training can be viewed as an Adversarial Activation Maximization Process.
Revised: The GAN training can be viewed as an Adversarial Activation Maximization Process.

Original: To be more speciﬁc, the generator is trained to perform activation maximization for each generated sample on the neuron that represents the log probability of its target class, while the discriminator is trained to distinguish generated samples and prevents them from getting their desired high activation.
Revised: To be more speciﬁc, the generator is trained to perform activation maximization for each generated sample on the neuron that represents the log probability of its target class, while the discriminator is trained to distinguish generated samples and prevents them from getting their desired high activation.

Original: It is worth mentioning that the sample that maximizes the activation of one neuron is not necessarily of high quality.
Revised: It is worth mentioning that the sample that maximizes the activation of one neuron is not necessarily of high quality.

Original: Traditionally people introduce various priors to counter the phenomenon (Nguyen et al., 2016a;b).
Revised: Traditionally people introduce various priors to counter the phenomenon (Nguyen et al., 2016a;b).

Original: In GAN, the adversarial process of GAN training can detect unrealistic samples and thus ensures the high-activation is achieved by high-quality samples that strongly confuse the discriminator.
Revised: In GAN, the adversarial process of GAN training can detect unrealistic samples and thus ensures the high-activation is achieved by high-quality samples that strongly confuse the discriminator.

Original: We thus name our model the Activation Maximization Generative Adversarial Network (AM-GAN).
Revised: We thus name our model the Activation Maximization Generative Adversarial Network (AM-GAN).

Original: 5. AC-GAN ∗ +
Revised: 5. AC-GAN ∗ +

Original: Experimentally we ﬁnd AC-GAN easily get mode collapsed and a relatively low weight for the auxiliary classiﬁer term in the generator’s loss function would help.
Revised: Experimentally we ﬁnd AC-GAN easily get mode collapsed and a relatively low weight for the auxiliary classiﬁer term in the generator’s loss function would help.

Original: In the Section 4.3, we attribute mode collapse to the miss of adversarial training in the auxiliary classiﬁer.
Revised: In the Section 4.3, we attribute mode collapse to the miss of adversarial training in the auxiliary classiﬁer.

Original: From the adversarial activation maximization view: without adversarial training, the auxiliary classiﬁer loss that requires high activation on a certain class, cannot ensure the sample quality.
Revised: From the adversarial activation maximization view: without adversarial training, the auxiliary classiﬁer loss that requires high activation on a certain class, cannot ensure the sample quality.

Original: That is, in AC-GAN, the vanilla GAN loss plays the role for ensuring sample quality and avoiding mode collapse.
Revised: That is, in AC-GAN, the vanilla GAN loss plays the role for ensuring sample quality and avoiding mode collapse.

Original: Here we introduce an extra loss to the auxiliary classiﬁer in AC-GAN ∗ to enforce adversarial training and experimentally ﬁnd it consistently improve the performance:
Revised: Here we introduce an extra loss to the auxiliary classiﬁer in AC-GAN ∗ to enforce adversarial training and experimentally ﬁnd it consistently improve the performance:

Original: [Equation]
Revised: [Equation]

Original: where u ( · ) represents the uniform distribution, which in spirit is the same as CatGAN (Springenberg, 2015).
Revised: where u ( · ) represents the uniform distribution, which in spirit is the same as CatGAN (Springenberg, 2015).

Original: Recall that we omit the auxiliary classiﬁer loss E ( x,y ) ∼ G (cid:2) H (cid:0) u ( y )] in AC-GAN ∗ .
Revised: Recall that we omit the auxiliary classiﬁer loss E ( x,y ) ∼ G (cid:2) H (cid:0) u ( y )] in AC-GAN ∗ .

Original: According to our experiments, E ( x,y ) ∼ G [ H ( u ( y )] does improve AC-GAN ∗ ’s stability and make it less likely to get mode collapse, but it also leads to a worse Inception Score.
Revised: According to our experiments, E ( x,y ) ∼ G [ H ( u ( y )] does improve AC-GAN ∗ ’s stability and make it less likely to get mode collapse, but it also leads to a worse Inception Score.

Original: We will report the detailed results in
Revised: We will report the detailed results in

Original: Section 7.
Revised: Section 7.

Original: Our understanding on this phenomenon is that: by encouraging the auxiliary classiﬁer also to classify fake samples to their target classes, it actually reduces the auxiliary classiﬁer’s ability on providing gradient guidance towards the real classes, and thus also alleviates the conﬂict between the
Revised: Our understanding on this phenomenon is that: by encouraging the auxiliary classiﬁer also to classify fake samples to their target classes, it actually reduces the auxiliary classiﬁer’s ability on providing gradient guidance towards the real classes, and thus also alleviates the conﬂict between the

Original: GAN loss and the auxiliary classiﬁer loss.
Revised: GAN loss and the auxiliary classiﬁer loss.

Original:  E VALUATION M ETRICS
Revised:  E VALUATION M ETRICS

Original: One of the difﬁculties in generative models is the evaluation methodology (Theis et al., 2015).
Revised: One of the difﬁculties in generative models is the evaluation methodology (Theis et al., 2015).

Original: In this section, we conduct both the mathematical and the empirical analysis on the widely-used evaluation metric Inception Score (Salimans et al., 2016) and other relevant metrics.
Revised: In this section, we conduct both the mathematical and the empirical analysis on the widely-used evaluation metric Inception Score (Salimans et al., 2016) and other relevant metrics.

Original: We will show that Inception
Revised: We will show that Inception

Original: Score mainly works as a diversity measurement and we propose the AM Score as a compensation to
Revised: Score mainly works as a diversity measurement and we propose the AM Score as a compensation to

Original: Inception Score for estimating the generated sample quality.
Revised: Inception Score for estimating the generated sample quality.

Original: 6. I NCEPTION S CORE
Revised: 6. I NCEPTION S CORE

Original: As a recently proposed metric for evaluating the performance of generative models, Inception Score has been found well correlated with human evaluation (Salimans et al., 2016), where a publiclyavailable Inception model C pre-trained on ImageNet is introduced.
Revised: As a recently proposed metric for evaluating the performance of generative models, Inception Score has been found well correlated with human evaluation (Salimans et al., 2016), where a publiclyavailable Inception model C pre-trained on ImageNet is introduced.

Original: By applying the Inception model to each generated sample x and getting the corresponding class probability distribution C ( x ) ,
Revised: By applying the Inception model to each generated sample x and getting the corresponding class probability distribution C ( x ) ,

Original: Inception Score is calculated via
Revised: Inception Score is calculated via

Original: [Equation]
Revised: [Equation]

Original: where E x is short of E x ∼ G and ¯ C G
Revised: where E x is short of E x ∼ G and ¯ C G

Original: =
Revised: =

Original: E x
Revised: E x

Original: [ C ( x )] is the overall probability distribution of the generated samples over classes, which is judged by C , and KL denotes the Kullback-Leibler divergence.
Revised: [ C ( x )] is the overall probability distribution of the generated samples over classes, which is judged by C , and KL denotes the Kullback-Leibler divergence.

Original: As proved in Appendix D, E x (cid:2) KL (cid:0) C ( x ) (cid:107) ¯ C G
Revised: As proved in Appendix D, E x (cid:2) KL (cid:0) C ( x ) (cid:107) ¯ C G

Original: (cid:1)(cid:3) can be decomposed into two terms in entropy:
Revised: (cid:1)(cid:3) can be decomposed into two terms in entropy:

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: 6. T HE P ROPERTIES OF I NCEPTION M ODEL
Revised: 6. T HE P ROPERTIES OF I NCEPTION M ODEL

Original: A common understanding of how Inception Score works lies in that a high score in the ﬁrst term
Revised: A common understanding of how Inception Score works lies in that a high score in the ﬁrst term

Original: H ( ¯ C G ) indicates the generated samples have high diversity (the overall class probability distribution evenly distributed), and a high score in the second term − E
Revised: H ( ¯ C G ) indicates the generated samples have high diversity (the overall class probability distribution evenly distributed), and a high score in the second term − E

Original: x [ H ( C ( x ))] indicates that each individual sample has high quality (each generated sample’s class probability distribution is sharp, i.e., it can be classiﬁed into one of the real classes with high conﬁdence)
Revised: x [ H ( C ( x ))] indicates that each individual sample has high quality (each generated sample’s class probability distribution is sharp, i.e., it can be classiﬁed into one of the real classes with high conﬁdence)

Original: (Salimans et al., 2016).
Revised: (Salimans et al., 2016).

Original: However, taking CIFAR-10 as an illustration, the data are not evenly distributed over the classes under the Inception model trained on ImageNet, which is presented in Figure 4a.
Revised: However, taking CIFAR-10 as an illustration, the data are not evenly distributed over the classes under the Inception model trained on ImageNet, which is presented in Figure 4a.

Original: It makes Inception
Revised: It makes Inception

Original: Score problematic in the view of the decomposed scores, i.e., H ( ¯ C G ) and − E
Revised: Score problematic in the view of the decomposed scores, i.e., H ( ¯ C G ) and − E

Original: x [ H ( C ( x ))] .
Revised: x [ H ( C ( x ))] .

Original: Such as that one would ask whether a higher H ( ¯ C G ) indicates a better mode coverage and whether a smaller
Revised: Such as that one would ask whether a higher H ( ¯ C G ) indicates a better mode coverage and whether a smaller

Original: H ( C ( x )) indicates a better sample quality.
Revised: H ( C ( x )) indicates a better sample quality.

Original: [Figure]
Revised: [Figure]

Original: (c) increase usually tends to decrease practice as illustrated in (
Revised: (c) increase usually tends to decrease practice as illustrated in (

Original: [Figure]
Revised: [Figure]

Original: training images.
Revised: training images.

Original: a) ¯ C G over ImageNet classes; b) H ( C ( x )) distribution with ImageNet classiﬁer of each class; c) H ( C ( x )) distribution with CIFAR-10 classiﬁer of each class.
Revised: a) ¯ C G over ImageNet classes; b) H ( C ( x )) distribution with ImageNet classiﬁer of each class; c) H ( C ( x )) distribution with CIFAR-10 classiﬁer of each class.

Original: With the Inception model, the value of H ( C ( x )) score of CIFAR-10 training data is variant, which means, even in real data, it would still strongly prefer some samples than some others.
Revised: With the Inception model, the value of H ( C ( x )) score of CIFAR-10 training data is variant, which means, even in real data, it would still strongly prefer some samples than some others.

Original: H ( C ( x )) on a classiﬁer that pre-trained on CIFAR-10 has low values for all CIFAR-10 training data and thus can be used as an indicator of sample quality.
Revised: H ( C ( x )) on a classiﬁer that pre-trained on CIFAR-10 has low values for all CIFAR-10 training data and thus can be used as an indicator of sample quality.

Original: We experimentally ﬁnd that, as in Figure 3b, the value of H ( ¯ C G ) is usually going down during the training process, however, which is expected to increase.
Revised: We experimentally ﬁnd that, as in Figure 3b, the value of H ( ¯ C G ) is usually going down during the training process, however, which is expected to increase.

Original: And when we delve into the detail of
Revised: And when we delve into the detail of

Original: H ( C ( x )) for each speciﬁc sample in the training data, we ﬁnd the value of H ( C ( x )) score is also variant, as illustrated in Figure 4b, which means, even in real data, it would still strongly prefer some samples than some others.
Revised: H ( C ( x )) for each speciﬁc sample in the training data, we ﬁnd the value of H ( C ( x )) score is also variant, as illustrated in Figure 4b, which means, even in real data, it would still strongly prefer some samples than some others.

Original: The exp operator in Inception Score and the large variance of the value of
Revised: The exp operator in Inception Score and the large variance of the value of

Original: H ( C ( x )) aggravate the phenomenon.
Revised: H ( C ( x )) aggravate the phenomenon.

Original: We also observe the preference on the class level in Figure 4b, e.g., E x
Revised: We also observe the preference on the class level in Figure 4b, e.g., E x

Original: [ H ( C ( x ))]=2 . 14 for trucks, while E x [ H ( C ( x ))]=3 .
Revised: [ H ( C ( x ))]=2 . 14 for trucks, while E x [ H ( C ( x ))]=3 .

Original: 80 for birds.
Revised: 80 for birds.

Original: It seems, for an ImageNet Classiﬁer, both the two indicators of Inception Score cannot work correctly.
Revised: It seems, for an ImageNet Classiﬁer, both the two indicators of Inception Score cannot work correctly.

Original: Next we will show that Inception Score actually works as a diversity measurement.
Revised: Next we will show that Inception Score actually works as a diversity measurement.

Original: 6. I NCEPTION S CORE AS A D IVERSITY M EASUREMENT
Revised: 6. I NCEPTION S CORE AS A D IVERSITY M EASUREMENT

Original: Since the two individual indicators are strongly correlated, here we go back to Inception Score’s original formulation E
Revised: Since the two individual indicators are strongly correlated, here we go back to Inception Score’s original formulation E

Original: x [ KL ( C ( x ) (cid:107) ¯ C G )] .
Revised: x [ KL ( C ( x ) (cid:107) ¯ C G )] .

Original: In this form, we could interpret Inception Score as that it requires each sample’s distribution C ( x ) highly different from the overall distribution of the generator ¯
Revised: In this form, we could interpret Inception Score as that it requires each sample’s distribution C ( x ) highly different from the overall distribution of the generator ¯

Original: C G , which indicates a good diversity over the generated samples.
Revised: C G , which indicates a good diversity over the generated samples.

Original: As is empirically observed, a mode-collapsed generator usually gets a low Inception Score.
Revised: As is empirically observed, a mode-collapsed generator usually gets a low Inception Score.

Original: In an extreme case, assuming all the generated samples collapse to a single point, then C ( x )=
Revised: In an extreme case, assuming all the generated samples collapse to a single point, then C ( x )=

Original: C G
Revised: C G

Original: and we would get the minimal Inception Score 1 . 0 , which is the exp result of zero.
Revised: and we would get the minimal Inception Score 1 . 0 , which is the exp result of zero.

Original: To simulate mode collapse in a more complicated case, we design synthetic experiments as following: given a set of N points { x 0 , x 1 , x 2 , ...,
Revised: To simulate mode collapse in a more complicated case, we design synthetic experiments as following: given a set of N points { x 0 , x 1 , x 2 , ...,

Original: x N − 1 } , with each point x
Revised: x N − 1 } , with each point x

Original: i adopting the distribution C ( x i )
Revised: i adopting the distribution C ( x i )

Original: = v ( i ) and representing class
Revised: = v ( i ) and representing class

Original: i , where v ( i ) is the vectorization operator of length N , as deﬁned in Section 2.1, we randomly drop m points, evaluate E x [ KL ( C ( x ) (cid:107) ¯ C G )] and draw the curve.
Revised: i , where v ( i ) is the vectorization operator of length N , as deﬁned in Section 2.1, we randomly drop m points, evaluate E x [ KL ( C ( x ) (cid:107) ¯ C G )] and draw the curve.

Original: As is showed in Figure 5, when
Revised: As is showed in Figure 5, when

Original: N − m increases, the value of E x [ KL ( C ( x ) (cid:107) ¯ C G )] monotonically increases in general, which means that it can well capture the mode dropping and the diversity of the generated distributions.
Revised: N − m increases, the value of E x [ KL ( C ( x ) (cid:107) ¯ C G )] monotonically increases in general, which means that it can well capture the mode dropping and the diversity of the generated distributions.

Original: [Figure]
Revised: [Figure]

Original: I
Revised: I

Original: [Figure]
Revised: [Figure]

Original: p t i
Revised: t i

Original: [Figure]
Revised: [Figure]

Original: n
Revised: n

Original: [Figure]
Revised: [Figure]

Original: c o r e ) (b)
Revised: c o r e ) (b)

Original: [Figure]
Revised: [Figure]

Original: One remaining question is that whether good mode coverage and sample diversity mean high quality of the generated samples.
Revised: One remaining question is that whether good mode coverage and sample diversity mean high quality of the generated samples.

Original: From the above analysis, we do not ﬁnd any evidence.
Revised: From the above analysis, we do not ﬁnd any evidence.

Original: A possible explanation is that, in practice, sample diversity is usually well correlated with the sample quality.
Revised: A possible explanation is that, in practice, sample diversity is usually well correlated with the sample quality.

Original: 6. AM S CORE WITH A CCORDINGLY P RETRAINED C LASSIFIER
Revised: 6. AM S CORE WITH A CCORDINGLY P RETRAINED C LASSIFIER

Original: Note that if each point x i has multiple variants such as x 1 i , x 2 i , x 3 i , one of the situations, where x 2 i and x 3 i are missing and only x 1 is generated, cannot be detected by E x [ KL ( C ( x ) (cid:107) ¯ C G )] score.
Revised: Note that if each point x i has multiple variants such as x 1 i , x 2 i , x 3 i , one of the situations, where x 2 i and x 3 i are missing and only x 1 is generated, cannot be detected by E x [ KL ( C ( x ) (cid:107) ¯ C G )] score.

Original: It means that with an accordingly pretrained classiﬁer, E x [ KL ( C ( x ) (cid:107) ¯ C G )] score cannot detect intra-class level mode collapse.
Revised: It means that with an accordingly pretrained classiﬁer, E x [ KL ( C ( x ) (cid:107) ¯ C G )] score cannot detect intra-class level mode collapse.

Original: This also explains why the Inception Network on ImageNet could be a good candidate C for CIFAR-10.
Revised: This also explains why the Inception Network on ImageNet could be a good candidate C for CIFAR-10.

Original: Exploring the optimal C is a challenge problem and we shall leave it as a future work.
Revised: Exploring the optimal C is a challenge problem and we shall leave it as a future work.

Original: However, there is no evidence that using an Inception Network trained on ImageNet can accurately measure the sample quality, as shown in Section 6.2.
Revised: However, there is no evidence that using an Inception Network trained on ImageNet can accurately measure the sample quality, as shown in Section 6.2.

Original: To compensate Inception Score, we propose to introduce an extra assessment using an accordingly pretrained classiﬁer.
Revised: To compensate Inception Score, we propose to introduce an extra assessment using an accordingly pretrained classiﬁer.

Original: In the accordingly pretrained classiﬁer, most real samples share similar H ( C ( x )) and 99.6% samples hold scores less than 0.05 as showed in Figure 4c, which demonstrates that H ( C ( x )) of the classiﬁer can be used as an indicator of sample quality.
Revised: In the accordingly pretrained classiﬁer, most real samples share similar H ( C ( x )) and 99.6% samples hold scores less than 0.05 as showed in Figure 4c, which demonstrates that H ( C ( x )) of the classiﬁer can be used as an indicator of sample quality.

Original: The entropy term on ¯ C G is actually problematic when training data is not evenly distributed over classes, for that argmin H ( ¯ C G ) is a uniform distribution.
Revised: The entropy term on ¯ C G is actually problematic when training data is not evenly distributed over classes, for that argmin H ( ¯ C G ) is a uniform distribution.

Original: To take the ¯ C train into account, we replace
Revised: To take the ¯ C train into account, we replace

Original: H ( ¯ C G ) with a KL divergence between ¯ C train and ¯ C G .
Revised: H ( ¯ C G ) with a KL divergence between ¯ C train and ¯ C G .

Original: So that
Revised: So that

Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: which requires ¯ C G close to ¯ C train and each sample x has a low entropy C ( x ) .
Revised: which requires ¯ C G close to ¯ C train and each sample x has a low entropy C ( x ) .

Original: The minimal value of
Revised: The minimal value of

Original: AM Score is zero, and the smaller value, the better.
Revised: AM Score is zero, and the smaller value, the better.

Original: A sample training curve of AM Score is showed in Figure 6, where all indicators in AM Score work as expected.
Revised: A sample training curve of AM Score is showed in Figure 6, where all indicators in AM Score work as expected.

Original: [Table] [Table]
Revised: [Table] [Table]

Original: High-value indicates obvious intra-class mode collapse.
Revised: High-value indicates obvious intra-class mode collapse.

Original: Please refer to the Figure 11 in the Appendix for the visual results.
Revised: Please refer to the Figure 11 in the Appendix for the visual results.

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: To empirically validate our analysis and the effectiveness of the proposed method, we conduct experiments on the image benchmark datasets including CIFAR-10 and Tiny-ImageNet 1 which comprises 200 classes with 500 training images per class.
Revised: To empirically validate our analysis and the effectiveness of the proposed method, we conduct experiments on the image benchmark datasets including CIFAR-10 and Tiny-ImageNet 2 which comprises 200 classes with 500 training images per class.

Original: For evaluation, several metrics are used throughout our experiments, including Inception Score with the ImageNet classiﬁer, AM Score with a corresponding pretrained classiﬁer for each dataset, which is a DenseNet (Huang et al., 2016a) model.
Revised: For evaluation, several metrics are used throughout our experiments, including Inception Score with the ImageNet classiﬁer, AM Score with a corresponding pretrained classiﬁer for each dataset, which is a DenseNet (Huang et al., 2016a) model.

Original: We also follow Odena et al.
Revised: We also follow Odena et al.

Original: and use the mean MS-SSIM (Wang et al., 2004) of randomly chosen pairs of images within a given class, as a coarse detector of intra-class mode collapse.
Revised: and use the mean MS-SSIM (Wang et al., 2004) of randomly chosen pairs of images within a given class, as a coarse detector of intra-class mode collapse.

Original: A modiﬁed DCGAN structure, as listed in the Appendix F, is used in experiments.
Revised: A modiﬁed DCGAN structure, as listed in the Appendix F, is used in experiments.

Original: Visual results of various models are provided in the Appendix considering the page limit, such as Figure 9, etc.
Revised: Visual results of various models are provided in the Appendix considering the page limit, such as Figure 9, etc.

Original: The repeatable experiment code is published for further research 2 .
Revised: The repeatable experiment code is published for further research 3 .

Original: 7. E XPERIMENTS ON CIFAR7.1. GAN WITH A UXILIARY C LASSIFIER
Revised: 7. E XPERIMENTS ON CIFAR7.1. GAN WITH A UXILIARY C LASSIFIER

Original: The ﬁrst question is whether training an auxiliary classiﬁer without introducing correlated losses to the generator would help improve the sample quality.
Revised: The ﬁrst question is whether training an auxiliary classiﬁer without introducing correlated losses to the generator would help improve the sample quality.

Original: In other words, with the generator only with the GAN loss in the AC-GAN ∗ setting.
Revised: In other words, with the generator only with the GAN loss in the AC-GAN ∗ setting.

Original: (referring as GAN ∗ )
Revised: (referring as GAN ∗ )

Original: As is shown in Table 1, it improves GAN’s sample quality, but the improvement is limited comparing to the other methods.
Revised: As is shown in Table 1, it improves GAN’s sample quality, but the improvement is limited comparing to the other methods.

Original: It indicates that introduction of correlated loss plays an essential role in the remarkable improvement of GAN training.
Revised: It indicates that introduction of correlated loss plays an essential role in the remarkable improvement of GAN training.

Original: 7.1. C OMPARISON A MONG D IFFERENT M ODELS
Revised: 7.1. C OMPARISON A MONG D IFFERENT M ODELS

Original: The usage of the predeﬁned label would make the GAN model transform to its conditional version, which is substantially disparate with generating samples from pure random noises.
Revised: The usage of the predeﬁned label would make the GAN model transform to its conditional version, which is substantially disparate with generating samples from pure random noises.

Original: In this experiment, we use dynamic labeling for AC-GAN ∗ , AC-GAN ∗ + and AM-GAN to seek for a fair comparison among different discriminator models, including LabelGAN and GAN.
Revised: In this experiment, we use dynamic labeling for AC-GAN ∗ , AC-GAN ∗ + and AM-GAN to seek for a fair comparison among different discriminator models, including LabelGAN and GAN.

Original: We keep the network structure and hyper-parameters the same for different models, only difference lies in the output layer of the discriminator, i.e., the number of class logits, which is necessarily different across models.
Revised: We keep the network structure and hyper-parameters the same for different models, only difference lies in the output layer of the discriminator, i.e., the number of class logits, which is necessarily different across models.

Original: As is shown in Table 1, AC-GAN ∗ achieves improved sample quality over vanilla GAN, but sustains mode collapse indicated by the value 0.61 in MS-SSIM as in Table 2.
Revised: As is shown in Table 1, AC-GAN ∗ achieves improved sample quality over vanilla GAN, but sustains mode collapse indicated by the value 0.61 in MS-SSIM as in Table 2.

Original: By introducing adversarial
Revised: By introducing adversarial

Original: Link for 
Revised: Link for code: https://

Original: [Table]
Revised: [Table]

Original: training in the auxiliary classiﬁer, AC-GAN ∗ + outperforms AC-GAN ∗ .
Revised: training in the auxiliary classiﬁer, AC-GAN ∗ + outperforms AC-GAN ∗ .

Original: As an implicit target class model, LabelGAN suffers from the overlaid-gradient problem and achieves a relatively higher per sample entropy (0.124) in the AM Score, comparing to explicit target class model AM-GAN (0.079) and AC-GAN ∗ + (0.102).
Revised: As an implicit target class model, LabelGAN suffers from the overlaid-gradient problem and achieves a relatively higher per sample entropy (0.124) in the AM Score, comparing to explicit target class model AM-GAN (0.079) and AC-GAN ∗ + (0.102).

Original: In the table, our proposed AM-GAN model reaches the best scores against these baselines.
Revised: In the table, our proposed AM-GAN model reaches the best scores against these baselines.

Original: We also test AC-GAN ∗ with decreased weight on auxiliary classiﬁer losses in the generator (relative to the GAN loss).
Revised: We also test AC-GAN ∗ with decreased weight on auxiliary classiﬁer losses in the generator (relative to the GAN loss).

Original: It achieves 7.19 in Inception Score, 0.23 in AM Score and 0.35 in MS-SSIM.
Revised: It achieves 7.19 in Inception Score, 0.23 in AM Score and 0.35 in MS-SSIM.

Original: The 0.35 in MS-SSIM indicates there is no obvious mode collapse, which also conform with our above analysis.
Revised: The 0.35 in MS-SSIM indicates there is no obvious mode collapse, which also conform with our above analysis.

Original: 7.1. I NCEPTION S CORE C OMPARING WITH R ELATED W ORK
Revised: 7.1. I NCEPTION S CORE C OMPARING WITH R ELATED W ORK

Original: AM-GAN achieves Inception Score 8.83 in the previous experiments, which signiﬁcantly outperforms the baseline models in both our implementation and their reported scores as in Table 3.
Revised: AM-GAN achieves Inception Score 8.83 in the previous experiments, which signiﬁcantly outperforms the baseline models in both our implementation and their reported scores as in Table 3.

Original: By further enhancing the discriminator with more ﬁlters in each layer, AM-GAN also outperforms the orthogonal work (Guillermo et al., 2017) that enhances the class label information via class splitting.
Revised: By further enhancing the discriminator with more ﬁlters in each layer, AM-GAN also outperforms the orthogonal work (Guillermo et al., 2017) that enhances the class label information via class splitting.

Original: As the result
Revised: As the result

Original: AM
Revised: AM

Original: GAN state of
Revised: GAN state of

Original: 7.1. D YNAMIC L ABELING AND C LASS C ONDITION
Revised: 7.1. D YNAMIC L ABELING AND C LASS C ONDITION

Original: It’s found in our experiments that GAN models with class condition (predeﬁned labeling) tend to encounter intra-class mode collapse (ignoring the noise), which is obvious at the very beginning of
Revised: It’s found in our experiments that GAN models with class condition (predeﬁned labeling) tend to encounter intra-class mode collapse (ignoring the noise), which is obvious at the very beginning of

Original: GAN training and gets exasperated during the process.
Revised: GAN training and gets exasperated during the process.

Original: In the training process of GAN, it is important to ensure a balance between the generator and the discriminator.
Revised: In the training process of GAN, it is important to ensure a balance between the generator and the discriminator.

Original: With the same generator’s network structures and switching from dynamic labeling to class condition, we ﬁnd it hard to hold a good balance between the generator and the discriminator: to avoid the initial intra-class mode collapse, the discriminator need to be very powerful; however, it usually turns out the discriminator is too powerful to provide suitable gradients for the generator and results in poor sample quality.
Revised: With the same generator’s network structures and switching from dynamic labeling to class condition, we ﬁnd it hard to hold a good balance between the generator and the discriminator: to avoid the initial intra-class mode collapse, the discriminator need to be very powerful; however, it usually turns out the discriminator is too powerful to provide suitable gradients for the generator and results in poor sample quality.

Original: Nevertheless, we ﬁnd a suitable discriminator and conduct a set of comparisons with it.
Revised: Nevertheless, we ﬁnd a suitable discriminator and conduct a set of comparisons with it.

Original: The results can be found in Table 1.
Revised: The results can be found in Table 1.

Original: The general conclusion is similar to the above, AC-GAN ∗ + still outperforms
Revised: The general conclusion is similar to the above, AC-GAN ∗ + still outperforms

Original: AC-GAN ∗ and our AM-GAN reaches the best performance.
Revised: AC-GAN ∗ and our AM-GAN reaches the best performance.

Original: It’s worth noticing that the AC-GAN ∗ does not suffer from mode collapse in this setting.
Revised: It’s worth noticing that the AC-GAN ∗ does not suffer from mode collapse in this setting.

Original: In the class conditional version, although with ﬁne-tuned parameters, Inception Score is still relatively low.
Revised: In the class conditional version, although with ﬁne-tuned parameters, Inception Score is still relatively low.

Original: The explanation could be that, in the class conditional version, the sample diversity still tends to decrease, even with a relatively powerful discriminator.
Revised: The explanation could be that, in the class conditional version, the sample diversity still tends to decrease, even with a relatively powerful discriminator.

Original: With slight intra-class mode collapse, the per-sample-quality tends to improve, which results in a lower AM Score.
Revised: With slight intra-class mode collapse, the per-sample-quality tends to improve, which results in a lower AM Score.

Original: A supplementary evidence, not very strict, of partial mode collapse in the experiments is that: the (cid:80) | ∂G ( z ) ∂z | is around 45.0 in dynamic labeling setting, while it is 25.0 in the conditional version.
Revised: A supplementary evidence, not very strict, of partial mode collapse in the experiments is that: the (cid:80) | ∂G ( z ) ∂z | is around 45.0 in dynamic labeling setting, while it is 25.0 in the conditional version.

Original: The LabelGAN does not need explicit labels and the model is the same in the two experiment settings.
Revised: The LabelGAN does not need explicit labels and the model is the same in the two experiment settings.

Original: But please note that both Inception Score and the AM Score get worse in the conditional version.
Revised: But please note that both Inception Score and the AM Score get worse in the conditional version.

Original: The only difference is that the discriminator becomes more powerful with an extended layer, which attests
Revised: The only difference is that the discriminator becomes more powerful with an extended layer, which attests

Original: [Figure]
Revised: [Figure]

Original: that the balance between the generator and discriminator is crucial.
Revised: that the balance between the generator and discriminator is crucial.

Original: We ﬁnd that, without the concern of intra-class mode collapse, using the dynamic labeling makes the balance between generator and discriminator much easier.
Revised: We ﬁnd that, without the concern of intra-class mode collapse, using the dynamic labeling makes the balance between generator and discriminator much easier.

Original: 7.1. T HE E
Revised: 7.1. T HE E

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: G [ H
Revised: G [ H

Original: [Figure]
Revised: [Figure]

Original: u
Revised: u

Original: [Figure]
Revised: [Figure]

Original: y ) , C ( x ))] LOSS
Revised: y ) , C ( x ))] LOSS

Original: Note that we report results of the modiﬁed version of AC-GAN, i.e., AC-GAN ∗ in Table 1.
Revised: Note that we report results of the modiﬁed version of AC-GAN, i.e., AC-GAN ∗ in Table 1.

Original: If we take the omitted loss E ( x,y ) ∼ G [ H ( u ( y ) , C ( x ))] back to AC-GAN ∗ , which leads to the original AC-GAN (see Section 2.2), it turns out to achieve worse results on both Inception Score and AM Score on
Revised: If we take the omitted loss E ( x,y ) ∼ G [ H ( u ( y ) , C ( x ))] back to AC-GAN ∗ , which leads to the original AC-GAN (see Section 2.2), it turns out to achieve worse results on both Inception Score and AM Score on

Original: CIFAR-10, though dismisses mode collapse.
Revised: CIFAR-10, though dismisses mode collapse.

Original: Speciﬁcally, in dynamic labeling setting, Inception
Revised: Speciﬁcally, in dynamic labeling setting, Inception

Original: Score decreases from 7 .
Revised: Score decreases from 7 .

Original: 41 to 6 .
Revised: 41 to 6 .

Original: 48 and the AM Score increases from 0 .
Revised: 48 and the AM Score increases from 0 .

Original: 17 to 0 .
Revised: 17 to 0 .

Original: 43 , while in predeﬁned class setting, Inception Score decreases from 7 .
Revised: 43 , while in predeﬁned class setting, Inception Score decreases from 7 .

Original: 79 to 7 . 66 and the AM Score increases from 0 .
Revised: 79 to 7 . 66 and the AM Score increases from 0 .

Original: 16 to 0 . 20 .
Revised: 16 to 0 . 20 .

Original: This performance drop might be because we use different network architectures and hyper-parameters from AC-GAN (Odena et al., 2016).
Revised: This performance drop might be because we use different network architectures and hyper-parameters from AC-GAN (Odena et al., 2016).

Original: But we still fail to achieve its report Inception Score, i.e., 8 . 25 , on CIFAR-10 when using the reported hyper-parameters in the original paper.
Revised: But we still fail to achieve its report Inception Score, i.e., 8 . 25 , on CIFAR-10 when using the reported hyper-parameters in the original paper.

Original: Since they do not publicize the code, we suppose there might be some unreported details that result in the performance gap.
Revised: Since they do not publicize the code, we suppose there might be some unreported details that result in the performance gap.

Original: We would leave further studies in future work.
Revised: We would leave further studies in future work.

Original: 7.1. T HE L EARNING P ROPERTY
Revised: 7.1. T HE L EARNING P ROPERTY

Original: We plot the training curve in terms of Inception Score and AM Score in Figure 7.
Revised: We plot the training curve in terms of Inception Score and AM Score in Figure 7.

Original: Inception Score and
Revised: Inception Score and

Original: AM Score are evaluated with the same number of samples 50 k , which is the same as Salimans et al.
Revised: AM Score are evaluated with the same number of samples 50 k , which is the same as Salimans et al.

Original: Comparing with Inception Score, AM Score is more stable in general.
Revised: Comparing with Inception Score, AM Score is more stable in general.

Original: With more samples,
Revised: With more samples,

Original: Inception Score would be more stable, however the evaluation of Inception Score is relatively costly.
Revised: Inception Score would be more stable, however the evaluation of Inception Score is relatively costly.

Original: A better alternative of the Inception Model could help solve this problem.
Revised: A better alternative of the Inception Model could help solve this problem.

Original: The AC-GAN ∗ ’s curves appear stronger jitter relative to the others.
Revised: The AC-GAN ∗ ’s curves appear stronger jitter relative to the others.

Original: It might relate to the counteract between the auxiliary classiﬁer loss and the GAN loss in the generator.
Revised: It might relate to the counteract between the auxiliary classiﬁer loss and the GAN loss in the generator.

Original: Another observation is that the AM-GAN in terms of Inception Score is comparable with LabelGAN and AC-GAN ∗ + at the beginning, while in terms of AM Score, they are quite distinguishable from each other.
Revised: Another observation is that the AM-GAN in terms of Inception Score is comparable with LabelGAN and AC-GAN ∗ + at the beginning, while in terms of AM Score, they are quite distinguishable from each other.

Original: 7. E XPERIMENTS ON T INY -I MAGE N ET
Revised: 7. E XPERIMENTS ON T INY -I MAGE N ET

Original: In the CIFAR-10 experiments, the results are consistent with our analysis and the proposed method outperforms these strong baselines.
Revised: In the CIFAR-10 experiments, the results are consistent with our analysis and the proposed method outperforms these strong baselines.

Original: We demonstrate that the conclusions can be generalized with experiments in another dataset Tiny-ImageNet.
Revised: We demonstrate that the conclusions can be generalized with experiments in another dataset Tiny-ImageNet.

Original: The Tiny-ImageNet consists with more classes and fewer samples for each class than CIFAR-10, which should be more challenging.
Revised: The Tiny-ImageNet consists with more classes and fewer samples for each class than CIFAR-10, which should be more challenging.

Original: We downsize Tiny-ImageNet samples from 64 × 64 to 32 ×and simply leverage the same network structure that used in CIFAR-10, and the experiment result is showed also in Table 1.
Revised: We downsize Tiny-ImageNet samples from 64 × 64 to 32 ×and simply leverage the same network structure that used in CIFAR-10, and the experiment result is showed also in Table 1.

Original: From the comparison, AM-GAN still outperforms other methods remarkably.
Revised: From the comparison, AM-GAN still outperforms other methods remarkably.

Original: And the AC-GAN ∗ + gains better performance than AC-GAN ∗ .
Revised: And the AC-GAN ∗ + gains better performance than AC-GAN ∗ .

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: In this paper, we analyze current GAN models that incorporate class label information.
Revised: In this paper, we analyze current GAN models that incorporate class label information.

Original: Our analysis shows that: LabelGAN works as an implicit target class model, however it suffers from the overlaidgradient problem at the meantime, and explicit target class would solve this problem.
Revised: Our analysis shows that: LabelGAN works as an implicit target class model, however it suffers from the overlaidgradient problem at the meantime, and explicit target class would solve this problem.

Original: We demonstrate that introducing the class logits in a non-hierarchical way, i.e., replacing the overall real class logit in the discriminator with the speciﬁc real class logits, usually works better than simply supplementing an auxiliary classiﬁer, where we provide an activation maximization view for GAN training and highlight the importance of adversarial training.
Revised: We demonstrate that introducing the class logits in a non-hierarchical way, i.e., replacing the overall real class logit in the discriminator with the speciﬁc real class logits, usually works better than simply supplementing an auxiliary classiﬁer, where we provide an activation maximization view for GAN training and highlight the importance of adversarial training.

Original: In addition, according to our experiments, predeﬁned labeling tends to lead to intra-class mode collapsed, and we propose dynamic labeling as an alternative.
Revised: In addition, according to our experiments, predeﬁned labeling tends to lead to intra-class mode collapsed, and we propose dynamic labeling as an alternative.

Original: Our extensive experiments on benchmarking datasets validate our analysis and demonstrate our proposed
Revised: Our extensive experiments on benchmarking datasets validate our analysis and demonstrate our proposed

Original: AM-GAN’s superior performance against strong baselines.
Revised: AM-GAN’s superior performance against strong baselines.

Original: Moreover, we delve deep into the widelyused evaluation metric Inception Score, reveal that it mainly works as a diversity measurement.
Revised: Moreover, we delve deep into the widelyused evaluation metric Inception Score, reveal that it mainly works as a diversity measurement.

Original: And we also propose AM Score as a compensation to more accurately estimate the sample quality.
Revised: And we also propose AM Score as a compensation to more accurately estimate the sample quality.

Original: In this paper, we focus on the generator and its sample quality, while some related work focuses on the discriminator and semi-supervised learning.
Revised: In this paper, we focus on the generator and its sample quality, while some related work focuses on the discriminator and semi-supervised learning.

Original: For future work, we would like to conduct empirical studies on discriminator learning and semi-supervised learning.
Revised: For future work, we would like to conduct empirical studies on discriminator learning and semi-supervised learning.

Original: We extend AM-GAN to unlabeled data in the Appendix C, where unsupervised and semi-supervised is accessible in the framework of
Revised: We extend AM-GAN to unlabeled data in the Appendix C, where unsupervised and semi-supervised is accessible in the framework of

Original: AM-GAN.
Revised: AM-GAN.

Original: The classiﬁer-based evaluation metric might encounter the problem related to adversarial samples, which requires further study.
Revised: The classiﬁer-based evaluation metric might encounter the problem related to adversarial samples, which requires further study.

Original: Combining AM-GAN with Integral Probability Metric based
Revised: Combining AM-GAN with Integral Probability Metric based

Original: GAN models such as Wasserstein GAN (Arjovsky et al., 2017) could also be a promising direction since it is orthogonal to our work.
Revised: GAN models such as Wasserstein GAN (Arjovsky et al., 2017) could also be a promising direction since it is orthogonal to our work.

Original: A G RADIENT V ANISHING & − log( D r ( x )) & L ABEL S MOOTHING A. L ABEL S MOOTHING
Revised: A G RADIENT V ANISHING & − log( D r ( x )) & L ABEL S MOOTHING A. L ABEL S MOOTHING

Original: Label smoothing that avoiding extreme logits value was showed to be a good regularization (Szegedy et al., 2016).
Revised: Label smoothing that avoiding extreme logits value was showed to be a good regularization (Szegedy et al., 2016).

Original: A general version of label smoothing could be: modifying the target probability of discriminator)
Revised: A general version of label smoothing could be: modifying the target probability of discriminator)

Original: [Equation]
Revised: [Equation]

Original: Salimans et al.
Revised: Salimans et al.

Original: (2016) proposed to use only one-side label smoothing.
Revised: (2016) proposed to use only one-side label smoothing.

Original: That is, to only apply label smoothing for real samples: λ 1 = 0 and λ 2 > 0 .
Revised: That is, to only apply label smoothing for real samples: λ 1 = 0 and λ 2 > 0 .

Original: The reasoning of one-side label smoothing is applying label smoothing on fake samples will lead to fake mode on data distribution, which is too obscure.
Revised: The reasoning of one-side label smoothing is applying label smoothing on fake samples will lead to fake mode on data distribution, which is too obscure.

Original: We will next show the exact problems when applying label smoothing to fake samples along with the log(1 − D r ( x )) generator loss, in the view of gradient w.r.t.
Revised: We will next show the exact problems when applying label smoothing to fake samples along with the log(1 − D r ( x )) generator loss, in the view of gradient w.r.t.

Original: class logit, i.e., the class-aware gradient, and we will also show that the problem does not exist when using the − log( D r ( x )) generator loss.
Revised: class logit, i.e., the class-aware gradient, and we will also show that the problem does not exist when using the − log( D r ( x )) generator loss.

Original: A. T HE log(1 − D r ( x )) GENERATOR LOSS
Revised: A. T HE log(1 − D r ( x )) GENERATOR LOSS

Original: The log(1 − D r ( x )) generator loss with label smoothing in terms of cross-entropy is
Revised: The log(1 − D r ( x )) generator loss with label smoothing in terms of cross-entropy is

Original: [Equation]
Revised: [Equation]

Original: [Equation]

Original: the negative gradientof which is
Revised: with lemma 1, its negative gradient is

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: gradient vanishing
Revised: gradient vanishing

Original: D

Original: [Equation]
Revised: [Equation]

Original: D r ( x ) is optimized towards
Revised: D r ( x ) is optimized towards

Original: [Equation]
Revised: [Equation]

Original: is optimizedtowards
Revised: is optimized

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: Gradient vanishing is a well know training problem of GAN.
Revised: Gradient vanishing is a well know training problem of GAN.

Original: Optimizing D r ( x ) towards 0 or 1 is also not what desired, because the discriminator is mapping real samples to the distribution with
Revised: Optimizing D r ( x ) towards 0 or 1 is also not what desired, because the discriminator is mapping real samples to the distribution with

Original: D r ( x ) = 1 − λ 2 .
Revised: D r ( x ) = 1 − λ 2 .

Original: A. T HE − log( D r ( x )) GENERATOR LOSS
Revised: A. T HE − log( D r ( x )) GENERATOR LOSS

Original: The − log( D r ( x )) generator loss with target [1 − λ, λ ] in terms of cross-entropy is
Revised: The − log( D r ( x )) generator loss with target [1 − λ, λ ] in terms of cross-entropy is

Original: [Equation]
Revised: [Equation]

Original: the negative gradient of which is
Revised: the negative gradient of which is

Original: [Equation]
Revised: [Equation]

Original: Without label smooth λ , the − log( D r ( x )) always ∗ preserves the same gradient direction as log(1 − D r ( x )) though giving a difference gradient scale.
Revised: Without label smooth λ , the − log( D r ( x )) always ∗ preserves the same gradient direction as log(1 − D r ( x )) though giving a difference gradient scale.

Original: We must note that non-zero gradient does not mean that the gradient is efﬁcient or valid.
Revised: We must note that non-zero gradient does not mean that the gradient is efﬁcient or valid.

Original: The both-side label smoothed version has a strong connection to Least-Square GAN Mao et al. (2016): with the fake logit ﬁxed to zero, the discriminator maps real to α on the real logit and maps fake to β on the real logit, the generator in contrast tries to map fake sample to α .
Revised: The both-side label smoothed version has a strong connection to Least-Square GAN (Mao et al., 2016): with the fake logit ﬁxed to zero, the discriminator maps real to α on the real logit and maps fake to β on the real logit, the generator in contrast tries to map fake sample to α .

Original: Their gradient on the logit are also similar.
Revised: Their gradient on the logit are also similar.

Original: B C AT GAN
Revised: B C AT GAN

Original: The auxiliary classiﬁer loss of AM-GAN can also be viewed as the cross-entropy version of CatGAN: generator of CatGAN directly optimizes entropy H ( R ( D ( x ))) to make each sample be one class, while AM-GAN achieves this by the ﬁrst term of its decomposed loss H ( R ( v ( x )) , R ( D ( x ))) in terms of cross-entropy with given target distribution.
Revised: The auxiliary classiﬁer loss of AM-GAN can also be viewed as the cross-entropy version of CatGAN: generator of CatGAN directly optimizes entropy H ( R ( D ( x ))) to make each sample be one class, while AM-GAN achieves this by the ﬁrst term of its decomposed loss H ( R ( v ( x )) , R ( D ( x ))) in terms of cross-entropy with given target distribution.

Original: That is, the AM-GAN is the cross-entropy version of CatGAN that is combined with LabelGAN by introducing an additional fake class.
Revised: That is, the AM-GAN is the cross-entropy version of CatGAN that is combined with LabelGAN by introducing an additional fake class.

Original: B. D ISCRIMINATOR LOSS ON FAKE SAMPLE
Revised: B. D ISCRIMINATOR LOSS ON FAKE SAMPLE

Original: The discriminator of CatGAN maximizes the prediction entropy of each fake sample:
Revised: The discriminator of CatGAN maximizes the prediction entropy of each fake sample:

Original: [Equation]
Revised: [Equation]

Original: In AM-GAN, as we have an extra class on fake, we can achieve this in a simpler manner by minimizing the probability on real logits.
Revised: In AM-GAN, as we have an extra class on fake, we can achieve this in a simpler manner by minimizing the probability on real logits.

Original: [Equation]
Revised: [Equation]

Original: If v r
Revised: If v r

Original: ( K +1) is not zero, that is, when we did negative label smoothing Salimans et al.
Revised: ( K +1) is not zero, that is, when we did negative label smoothing Salimans et al.

Original: (2016), we could deﬁne R ( v ( K +1)) to be a uniform distribution.
Revised: (2016), we could deﬁne R ( v ( K +1)) to be a uniform distribution.

Original: [Equation]
Revised: [Equation]

Original: As a result, the label smoothing part probability will be required to be uniformly distributed, similar to CatGAN.
Revised: As a result, the label smoothing part probability will be required to be uniformly distributed, similar to CatGAN.

Original: C U NLABELED D ATA
Revised: C U NLABELED D ATA

Original: In this section, we extend AM-GAN to unlabeled data.
Revised: In this section, we extend AM-GAN to unlabeled data.

Original: Our solution is analogous to CatGAN
Revised: Our solution is analogous to CatGAN

Original: Springenberg (2015).
Revised: Springenberg (2015).

Original: C. S EMI - SUPERVISED SETTING
Revised: C. S EMI - SUPERVISED SETTING

Original: Under semi-supervised setting, we can add the following loss to the original solution to integrate the unlabeled data (with the distribution denoted as p unl ( x ) ):
Revised: Under semi-supervised setting, we can add the following loss to the original solution to integrate the unlabeled data (with the distribution denoted as p unl ( x ) ):

Original: [Equation]
Revised: [Equation]

Original: C. U NSUPERVISED SETTING
Revised: C. U NSUPERVISED SETTING

Original: Under unsupervised setting, we need to introduce one extra loss, analogy to categorical GAN
Revised: Under unsupervised setting, we need to introduce one extra loss, analogy to categorical GAN

Original: Springenberg (2015):
Revised: Springenberg (2015):

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: where the p ref is a reference label distribution for the prediction on unsupervised data.
Revised: where the p ref is a reference label distribution for the prediction on unsupervised data.

Original: For example, p ref could be set as a uniform distribution, which requires the unlabeled data to make use of all the candidate class logits.
Revised: For example, p ref could be set as a uniform distribution, which requires the unlabeled data to make use of all the candidate class logits.

Original: This loss can be optionally added to semi-supervised setting, where the p ref could be deﬁned as the predicted label distribution on the labeled training data E x ∼ p data [ D ( x )] .
Revised: This loss can be optionally added to semi-supervised setting, where the p ref could be deﬁned as the predicted label distribution on the labeled training data E x ∼ p data [ D ( x )] .

Original: D I NCEPTION S CORE
Revised: D I NCEPTION S CORE

Original: As a recently proposed metric for evaluating the performance of the generative models, the InceptionScore has been found well correlated with human evaluation (Salimans et al., 2016), where a pre-trained publicly-available Inception model C is introduced.
Revised: As a recently proposed metric for evaluating the performance of the generative models, the InceptionScore has been found well correlated with human evaluation (Salimans et al., 2016), where a pre-trained publicly-available Inception model C is introduced.

Original: By applying the Inception model to each generated sample x and getting the corresponding class probability distribution C ( x ) , Inception
Revised: By applying the Inception model to each generated sample x and getting the corresponding class probability distribution C ( x ) , Inception

Original: Score is calculated via
Revised: Score is calculated via

Original: [Equation]
Revised: [Equation]

Original: where E x is short of E x ∼ G and ¯ C G
Revised: where E x is short of E x ∼ G and ¯ C G

Original: =
Revised: =

Original: E x
Revised: E x

Original: [ C ( x )] is the overall probability distribution of the generated samples over classes, which is judged by C , and KL denotes the Kullback-Leibler divergence which is deﬁned as
Revised: [ C ( x )] is the overall probability distribution of the generated samples over classes, which is judged by C , and KL denotes the Kullback-Leibler divergence which is deﬁned as

Original: [Equation]
Revised: [Equation]

Original: An extended metric, the Mode Score, is proposed in Che et al.
Revised: An extended metric, the Mode Score, is proposed in Che et al.

Original: (2016) to take the prior distribution of the labels into account, which is calculated via
Revised: (2016) to take the prior distribution of the labels into account, which is calculated via

Original: [Equation]
Revised: [Equation]

Original: where the overall class distribution from the training data ¯ C train has been added as a reference.
Revised: where the overall class distribution from the training data ¯ C train has been added as a reference.

Original: We show in the following that, in fact, Mode Score and Inception Score are equivalent.
Revised: We show in the following that, in fact, Mode Score and Inception Score are equivalent.

Original: Lemma 3.
Revised: Lemma 3.

Original: Let p ( x ) be the class probability distribution of the sample x , and ¯ p denote another probability distribution, then
Revised: Let p ( x ) be the class probability distribution of the sample x , and ¯ p denote another probability distribution, then

Original: [Equation]
Revised: [Equation]

Original: With Lemma
Revised: With Lemma

Original: [Equation]
Revised: [Equation]

Original: we have log
Revised: we have log

Original: [Equation]
Revised: [Equation]

Original: Mode
Revised: Mode

Original: [Equation]
Revised: [Equation]

Original: E T HE L EMMA AND P ROOFS
Revised: E T HE L EMMA AND P ROOFS

Original: Lemma 1.
Revised: Lemma 1.

Original: With l being the logits vector and σ being the softmax function, let σ ( l ) be the current softmax probability distribution and ˆ p denote any target probability distribution, then:
Revised: With l being the logits vector and σ being the softmax function, let σ ( l ) be the current softmax probability distribution and ˆ p denote any target probability distribution, then:

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: [Equation]
Revised: [Equation]

Original: Lemma 2.
Revised: Lemma 2.

Original: Given v =
Revised: Given v =

Original: [ v 1 , . .
Revised: [ v 1 , . .

Original: . , v K +1 ] , v 1: K (cid:44) [ v 1 , . .
Revised: . , v K +1 ] , v 1: K (cid:44) [ v 1 , . .

Original: . , v K ] , v r (cid:44) (cid:80) Kk
Revised: . , v K ] , v r (cid:44) (cid:80) Kk

Original: =1 v k , R ( v ) (cid:44) v 1: K /v r and
Revised: =1 v k , R ( v ) (cid:44) v 1: K /v r and

Original: F ( v ) (cid:44) [ v r , v K +1 ] , let ˆ p =
Revised: F ( v ) (cid:44) [ v r , v K +1 ] , let ˆ p =

Original: [ˆ p 1 , . . .
Revised: [ˆ p 1 , . . .

Original: , ˆ p K +1 ] , p
Revised: , ˆ p K +1 ] , p

Original: =
Revised: =

Original: [ p 1 , . .
Revised: [ p 1 , . .

Original: , p K +1 ] , then we have:
Revised: , p K +1 ] , then we have:

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: [Equation]
Revised: [Equation]

Original: Lemma 3.
Revised: Lemma 3.

Original: Let p ( x ) be the class probability distribution of the sample x that from a certain data distribution
Revised: Let p ( x ) be the class probability distribution of the sample x that from a certain data distribution

Original: [Equation]
Revised: [Equation]

Original: and ¯ p denote the reference probability distribution, then
Revised: and ¯ p denote the reference probability distribution, then

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: Proof.
Revised: Proof.

Original: [Equation]
Revised: [Equation]

Original: = H (cid:0) E x (cid:2) p (
Revised: = H (cid:0) E x (cid:2) p (

Original: ) (cid:3)
Revised: ) (cid:3)

Original: ¯ p (cid:1) .
Revised: ¯ p (cid:1) .

Original: [Figure]
Revised: [Figure]

Original: F N ETWORK S TRUCTURE & H YPER -P ARAMETERS
Revised: F N ETWORK S TRUCTURE & H YPER -P ARAMETERS

Original: Generator:
Revised: Generator:

Original: Operation
Revised: Operation

Original: Kernel
Revised: Kernel

Original: Strides
Revised: Strides

Original: Output Dims
Revised: Output Dims

Original: Output Drop
Revised: Output Drop

Original: Activation
Revised: Activation

Original: BN
Revised: BN

Original: Noise
Revised: Noise

Original: N/A
Revised: N/A

Original: N/A 100 |0.
Revised: N/A 100 |0.

Original: N(0.0, 1.0)
Revised: N(0.0, 1.0)

Original: Linear
Revised: Linear

Original: N/A
Revised: N/A

Original: N/A 4 × 4 ×0.
Revised: N/A 4 × 4 ×0.

Original: Leaky ReLU
Revised: Leaky ReLU

Original: True
Revised: True

Original: Deconvolution 3 ×2 ×8 × 8 ×0.
Revised: Deconvolution 3 ×2 ×8 × 8 ×0.

Original: Leaky ReLU
Revised: Leaky ReLU

Original: True
Revised: True

Original: Deconvolution 3 ×2 ×16 × 16 ×0.
Revised: Deconvolution 3 ×2 ×16 × 16 ×0.

Original: Leaky ReLU
Revised: Leaky ReLU

Original: True
Revised: True

Original: Deconvolution 3 ×2 ×32 × 32 ×0.
Revised: Deconvolution 3 ×2 ×32 × 32 ×0.

Original: Leaky ReLU
Revised: Leaky ReLU

Original: True
Revised: True

Original: Deconvolution 3 ×1 ×32 × 32 ×0.
Revised: Deconvolution 3 ×1 ×32 × 32 ×0.

Original: Tanh
Revised: Tanh

Original: Discriminator:
Revised: Discriminator:

Original: Operation
Revised: Operation

Original: Kernel
Revised: Kernel

Original: Strides
Revised: Strides

Original: Output Dims
Revised: Output Dims

Original: Output Drop
Revised: Output Drop

Original: Activation
Revised: Activation

Original: Add Gaussian Noise
Revised: Add Gaussian Noise

Original: N/A
Revised: N/A

Original: N/A 32 × 32 ×0.
Revised: N/A 32 × 32 ×0.

Original: N(0.0, 0.1)
Revised: N(0.0, 0.1)

Original: Convolution 3 ×1 ×32 × 32 ×0.
Revised: Convolution 3 ×1 ×32 × 32 ×0.

Original: Leaky ReLU
Revised: Leaky ReLU

Original: True
Revised: True

Original: Convolution 3 ×2 ×16 × 16 ×0.
Revised: Convolution 3 ×2 ×16 × 16 ×0.

Original: Leaky ReLU
Revised: Leaky ReLU

Original: True
Revised: True

Original: Convolution 3 ×2 ×8 × 8 ×0.
Revised: Convolution 3 ×2 ×8 × 8 ×0.

Original: Leaky ReLU
Revised: Leaky ReLU

Original: True
Revised: True

Original: Convolution 3 ×2 ×4 × 4 ×0.
Revised: Convolution 3 ×2 ×4 × 4 ×0.

Original: Leaky ReLU
Revised: Leaky ReLU

Original: True
Revised: True

Original: Convolution* 3 ×1 ×4 × 4 ×0.
Revised: Convolution* 3 ×1 ×4 × 4 ×0.

Original: Leaky ReLU
Revised: Leaky ReLU

Original: True
Revised: True

Original: AvgPool
Revised: AvgPool

Original: N/A
Revised: N/A

Original: N/A 1 × 1 ×
Revised: N/A 1 × 1 ×

Original: [Table]
Revised: [Table]

Original: N/A
Revised: N/A

Original: Linear
Revised: Linear

Original: N/A
Revised: N/A

Original: N/
Revised: N/

Original: [Table]
Revised: [Table]

Original: 10 |
Revised: 10 |

Original: [Table]
Revised: [Table]

Original: |
Revised: |

Original: [Table]
Revised: [Table]

Original: Softmax
Revised: Softmax

Original: The *layer was only used for class condition experiments
Revised: The *layer was only used for class condition experiments

Original: Optimizer: Adam with beta1=0.5, beta2=0.999; Batch size=100.
Revised: Optimizer: Adam with beta1=0.5, beta2=0.999; Batch size=100.

Original: Learning rate: Exponential decay with stair, initial learning rate 0.0004.
Revised: Learning rate: Exponential decay with stair, initial learning rate 0.0004.

Original: We use weight normalization for each weight
Revised: We use weight normalization for each weight