Original: I MPROVING DIVERSITY IN G ENERATIVE ADVERSARIAL NETWORKS BY ENCOURAGING DISCRIMINATOR REPRESENTATION ENTROPY
Revised: I MPROVING GAN T RAINING VIA B INARIZED R EPRESENTATION E NTROPY (BRE) R EGULARIZATION

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: We propose a novel regularizer to improve the training of Generative Adversarial
Revised: We propose a novel regularizer to improve the training of Generative Adversarial

Original: Networks (GANs).
Revised: Networks (GANs).

Original: The motivation is that when the discriminator D spreads out its model capacity in the right way, the learning signals given to the generator G are more informative and diverse, which helps G to explore better and discover the real data manifold while avoiding large unstable jumps due to the erroneous extrapolation made by D .
Revised: The motivation is that when the discriminator D spreads out its model capacity in the right way, the learning signals given to the generator G are more informative and diverse, which helps G to explore better and discover the real data manifold while avoiding large unstable jumps due to the erroneous extrapolation made by D .

Original: Our regularizer guides the rectiﬁer discriminator D to better allocate its model capacity, by encouraging the binary activation patterns on selected internal layers of D to have a high joint entropy.
Revised: Our regularizer guides the rectiﬁer discriminator D to better allocate its model capacity, by encouraging the binary activation patterns on selected internal layers of D to have a high joint entropy.

Original: Experimental results on both synthetic data and real datasets demonstrate improvements in stability and convergence speed of the GAN training, as well as higher sample quality.
Revised: Experimental results on both synthetic data and real datasets demonstrate improvements in stability and convergence speed of the GAN training, as well as higher sample quality.

Original: The approach also leads to higher classiﬁcation accuracies in semi-supervised learning
Revised: The approach also leads to higher classiﬁcation accuracies in semi-supervised learning.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Generative Adversarial Network (GAN) (Goodfellow et al., 2014) has been a new promising approach to unsupervised learning of complex high dimensional data in the last two years, with successful applications on image data (Isola et al., 2016;
Revised: Generative Adversarial Network (GAN) (Goodfellow et al., 2014) has been a new promising approach to unsupervised learning of complex high dimensional data in the last two years, with successful applications on image data (Isola et al., 2016;

Original: Shrivastava et al., 2016), and high potential for predictive representation learning (Mathieu et al., 2015) as well as reinforcement learning (Finn et al., 2016; Henderson et al., 2017).
Revised: Shrivastava et al., 2016), and high potential for predictive representation learning (Mathieu et al., 2015) as well as reinforcement learning (Finn et al., 2016; Henderson et al., 2017).

Original: In a nutshell, GANs learn from unlabeled data by engaging the generative model ( G ) in an adversarial game with a discriminator ( D ).
Revised: In a nutshell, GANs learn from unlabeled data by engaging the generative model ( G ) in an adversarial game with a discriminator ( D ).

Original: D learns to tell apart fake data generated by G from real data, while G learns to fool D , having access to D ’s input gradient.
Revised: D learns to tell apart fake data generated by G from real data, while G learns to fool D , having access to D ’s input gradient.

Original: Despite its success in generating high-quality data, such adversarial game setting also raises challenges for the training of GANs.
Revised: Despite its success in generating high-quality data, such adversarial game setting also raises challenges for the training of GANs.

Original: Many architectures and techniques have been proposed (Radford et al., 2015; Salimans et al., 2016; Gulrajani et al., 2017) to reduce extreme failures and improve the realism of generated data.
Revised: Many architectures and techniques have been proposed (Radford et al., 2015; Salimans et al., 2016; Gulrajani et al., 2017) to reduce extreme failures and improve the realism of generated data.

Original: However, many theoretical and practical open problems still remain, which have impeded the ease-of-use of GANs in new problems.
Revised: However, many theoretical and practical open problems still remain, which have impeded the ease-of-use of GANs in new problems.

Original: In particular, often G fails to capture certain variation or modes in the real data distribution, while D fails to exploit this failure to provide better training signal for G , leading to subtle mode collapse.
Revised: In particular, often G fails to capture certain variation or modes in the real data distribution, while D fails to exploit this failure to provide better training signal for G , leading to subtle mode collapse.

Original: Recently Arora et al.
Revised: Recently Arora et al.

Original: (2017) showed that the capacity of D plays an essential role in giving G sufﬁcient learning guidances to model the complex real data distribution.
Revised: (2017) showed that the capacity of D plays an essential role in giving G sufﬁcient learning guidances to model the complex real data distribution.

Original: With insufﬁcient capacity, D could fail to distinguish real and generated data distributions even when their Jensen-Shannon divergence or Wasserstein distance is not small.
Revised: With insufﬁcient capacity, D could fail to distinguish real and generated data distributions even when their Jensen-Shannon divergence or Wasserstein distance is not small.

Original: In this work, we demonstrate that even with sufﬁcient maximum capacity, D might not allocate its capacity in a desirable way that facilitates convergence to a good equilibrium.
Revised: In this work, we demonstrate that even with sufﬁcient maximum capacity, D might not allocate its capacity in a desirable way that facilitates convergence to a good equilibrium.

Original: We then propose a novel regularizer to guide D to have a better model capacity allocation.
Revised: We then propose a novel regularizer to guide D to have a better model capacity allocation.

Original: Our regularizer is constructed to encourage D ’s hidden binary activation patterns to have high joint entropy, based on a connection between the model capacity of a rectiﬁer net and its internal binary activation patterns.
Revised: Our regularizer is constructed to encourage D ’s hidden binary activation patterns to have high joint entropy, based on a connection between the model capacity of a rectiﬁer net and its internal binary activation patterns.

Original: Our experiments show that such high entropy representation leads to faster convergences, improved sample quality, as well as lower errors in semi-supervised learning.
Revised: Our experiments show that such high entropy representation leads to faster convergences, improved sample quality, as well as lower errors in semi-supervised learning.

Original:  C APACITY USAGE OF RECTIFIER NETS AND ITS EFFECTS ON GAN TRAINING
Revised:  C APACITY USAGE OF RECTIFIER NETS AND ITS EFFECTS ON GAN TRAINING

Original: The motivation of our regularizer starts with an observation that during GAN training, the generator
Revised: The motivation of our regularizer starts with an observation that during GAN training, the generator

Original: G receives information about the input space only indirectly through the gradient of D , ∇ x D ( x ) .
Revised: G receives information about the input space only indirectly through the gradient of D , ∇ x D ( x ) .

Original: Typically D is a rectiﬁer net.
Revised: Typically D is a rectiﬁer net.

Original: Absent of the last sigmoid nonlinearity, D computes piecewise linear functions, meaning that the learning signal to G is (almost) piecewise constant.
Revised: Absent of the last sigmoid nonlinearity, D computes piecewise linear functions, meaning that the learning signal to G is (almost) piecewise constant.

Original: The ﬁnal sigmoid nonlinearity does not change the direction of input gradient in each linear region, but merely the scale of gradient vectors.
Revised: The ﬁnal sigmoid nonlinearity does not change the direction of input gradient in each linear region, but merely the scale of gradient vectors.

Original: The learning of G in GANs can be interpreted as the movement of the fake samples generated by G toward the real data distribution, guided by the (almost) piecewise constant vectorial signals according to input space partitioning by D .
Revised: The learning of G in GANs can be interpreted as the movement of the fake samples generated by G toward the real data distribution, guided by the (almost) piecewise constant vectorial signals according to input space partitioning by D .

Original: Hence, the diversity and informativeness of learning signals to G is closely related to how the input space is partitioned, i.e. how D ’s model capacity is allocated.
Revised: Hence, the diversity and informativeness of learning signals to G is closely related to how the input space is partitioned, i.e. how D ’s model capacity is allocated.

Original: In an area of the input space, how much capacity D allocates into it can be approximately measured by the number of linear pieces in that area (Montufar et al., 2014).
Revised: In an area of the input space, how much capacity D allocates into it can be approximately measured by the number of linear pieces in that area (Montufar et al., 2014).

Original: When D spreads out its model capacity in a right way, the evenly dispersed partitioning helps
Revised: When D spreads out its model capacity in a right way, the evenly dispersed partitioning helps

Original: G to explore better and discover the real data manifold, while avoiding large unstable jumps due to overconﬁdent extrapolation made by D .
Revised: G to explore better and discover the real data manifold, while avoiding large unstable jumps due to overconﬁdent extrapolation made by D .

Original: Ideally, when GAN training is stable, the min-max game eventually forces D to represent subtle variations in the real data distribution and pass this information for the learning of G .
Revised: Ideally, when GAN training is stable, the min-max game eventually forces D to represent subtle variations in the real data distribution and pass this information for the learning of G .

Original: However, the discriminator D is solely tasked to separate real samples from the generated fake ones, thus has no incentive to do so, especially when the classiﬁcation task for D is too simple.
Revised: However, the discriminator D is solely tasked to separate real samples from the generated fake ones, thus has no incentive to do so, especially when the classiﬁcation task for D is too simple.

Original: Such is always the case in the early stage of the training and may persist to the later stage if the input space has high dimensionality or if G already collapsed.
Revised: Such is always the case in the early stage of the training and may persist to the later stage if the input space has high dimensionality or if G already collapsed.

Original: In these situations, D could overﬁt, and its internal layers could have degenerate representation whereby large portions of the input space are modelled as linear regions, as pictorially depicted in Fig. 1, and shown in the synthetic experiment in Sec. 4.1.
Revised: In these situations, D could overﬁt, and its internal layers could have degenerate representation whereby large portions of the input space are modelled as linear regions, as pictorially depicted in Fig. 1, and shown in the synthetic experiment in Sec. 4.1.

Original: With such degeneracy, learning signals from D are not diverse and fail to capture the differences among different modes or subtle variations of the real data.
Revised: With such degeneracy, learning signals from D are not diverse and fail to capture the differences among different modes or subtle variations of the real data.

Original: Furthermore, such degeneracy could also cause the learning of G to bluntly extrapolate, resulting in large updates, which in turn drops already discovered real data modes and/or leads to oscillations.
Revised: Furthermore, such degeneracy could also cause the learning of G to bluntly extrapolate, resulting in large updates, which in turn drops already discovered real data modes and/or leads to oscillations.

Original: We observe this phenomenon in the synthetic data problem in Sec. 4.1.
Revised: We observe this phenomenon in the synthetic data problem in Sec. 4.1.

Original: In this paper, we propose a new regularizer for training GANs where D is a rectiﬁer net.
Revised: In this paper, we propose a new regularizer for training GANs where D is a rectiﬁer net.

Original: Our regularizer encourages the discriminator D to cut the input space more ﬁnely around where the current
Revised: Our regularizer encourages the discriminator D to cut the input space more ﬁnely around where the current

Original: G distribution support, as well as where training might transport the generated data distribution to in the short future, as depicted in Fig.
Revised: G distribution support, as well as where training might transport the generated data distribution to in the short future, as depicted in Fig.

Original: 1 (right).
Revised: 1 (right).

Original: In this way, G receives rich guidance for faster exploration and more stable convergence .
Revised: In this way, G receives rich guidance for faster exploration and more stable convergence .

Original: The regularizer facilitates exploration because D tells apart the generated fake samples from the real ones in distinct ways.
Revised: The regularizer facilitates exploration because D tells apart the generated fake samples from the real ones in distinct ways.

Original: Hence a concentrated mass in the fake data distribution has a better chance of been spread apart because if the fake data points x lie in different regions, learning signals ∇ x D ( x ) to G are likely to point to different directions.
Revised: Hence a concentrated mass in the fake data distribution has a better chance of been spread apart because if the fake data points x lie in different regions, learning signals ∇ x D ( x ) to G are likely to point to different directions.

Original: On the other hand, the convergence to equilibrium is more stable because there are less large piecewise linear regions, where G learning constantly receives the same transportation direction, potentially leading to overshoot and oscillation.
Revised: On the other hand, the convergence to equilibrium is more stable because there are less large piecewise linear regions, where G learning constantly receives the same transportation direction, potentially leading to overshoot and oscillation.

Original: Our regularizer are constructed to encourage the activation patterns of the internal representations of points in a mini-batch to be diverse.
Revised: Our regularizer are constructed to encourage the activation patterns of the internal representations of points in a mini-batch to be diverse.

Original: As shown in Raghu et al.
Revised: As shown in Raghu et al.

Original: (2016), the different local linear regions deﬁned by D is closely related to the different activation patterns of D .
Revised: (2016), the different local linear regions deﬁned by D is closely related to the different activation patterns of D .

Original: In particular, two input points into D with different activation patterns on all layers of D are guaranteed to lie on different linear regions.
Revised: In particular, two input points into D with different activation patterns on all layers of D are guaranteed to lie on different linear regions.

Original: More details are presented in Sec.
Revised: More details are presented in Sec.

Original: 3 where the regularizer is deﬁned, along with the analysis of its properties.
Revised: 3 where the regularizer is deﬁned, along with the analysis of its properties.

Original: 2. R ELATED WORKS
Revised: 2. R ELATED WORKS

Original: Other regularization/gradient penalty techniques have also been proposed to stabilize GANs training (Gulrajani et al., 2017; Nagarajan & Kolter, 2017) recently.
Revised: Other regularization/gradient penalty techniques have also been proposed to stabilize GANs training (Gulrajani et al., 2017; Nagarajan & Kolter, 2017) recently.

Original: Gulrajani et al.
Revised: Gulrajani et al.

Original: (2017) adds an input gradient penalty to the update of D , so that the magnitude of signals passed to G is controlled.
Revised: (2017) adds an input gradient penalty to the update of D , so that the magnitude of signals passed to G is controlled.

Original: Nagarajan & Kolter (2017) modiﬁes the update of G to avoid going where the magnitude of ∇ x D ( x ) is large.
Revised: Nagarajan & Kolter (2017) modiﬁes the update of G to avoid going where the magnitude of ∇ x D ( x ) is large.

Original: These methods, as well as other similar works that constrain the input gradient norm or the
Revised: These methods, as well as other similar works that constrain the input gradient norm or the

Original: Lipschitz constant of D , all try to stabilize the training dynamics by regularizing the learning signal magnitude.
Revised: Lipschitz constant of D , all try to stabilize the training dynamics by regularizing the learning signal magnitude.

Original: This is different from our method that diversiﬁes the learning signal directions.
Revised: This is different from our method that diversiﬁes the learning signal directions.

Original: As discussed in the previous section, the diversiﬁed signal directions help both the convergence speed and the stability of the training.
Revised: As discussed in the previous section, the diversiﬁed signal directions help both the convergence speed and the stability of the training.

Original: In Sec. 4.2, we empirically demonstrate that our proposed method achieves better results than Wasserstein GAN with gradient penalty (WGAN-GP)
Revised: In Sec. 4.2, we empirically demonstrate that our proposed method achieves better results than Wasserstein GAN with gradient penalty (WGAN-GP)

Original: (Gulrajani et al., 2017).
Revised: (Gulrajani et al., 2017).

Original: The role of model capacity of the discriminator D in training generative adversarial networks (GANs) has been previously explored by Arora et al.
Revised: The role of model capacity of the discriminator D in training generative adversarial networks (GANs) has been previously explored by Arora et al.

Original: They show that D with a ﬁnite number of parameters has limited capacity in distinguishing real data from the generated ones.
Revised: They show that D with a ﬁnite number of parameters has limited capacity in distinguishing real data from the generated ones.

Original: They suggest increasing the discriminator D ’s capacity among other modiﬁcations.
Revised: They suggest increasing the discriminator D ’s capacity among other modiﬁcations.

Original: Our work can be viewed as a continuation along this direction, except that we treat the model capacity not as a static number, but a varying function in the different parts of the input space during training.
Revised: Our work can be viewed as a continuation along this direction, except that we treat the model capacity not as a static number, but a varying function in the different parts of the input space during training.

Original: Because even with a large number of parameters, D might not use its capacity in a right way to help convergence, as discussed previously.
Revised: Because even with a large number of parameters, D might not use its capacity in a right way to help convergence, as discussed previously.

Original: We explore the question on where and how D can utilize its limited capacity effectively for better training convergence.
Revised: We explore the question on where and how D can utilize its limited capacity effectively for better training convergence.

Original: Encouraging D to use its capacity in a constructive way is non-trivial.
Revised: Encouraging D to use its capacity in a constructive way is non-trivial.

Original: One theoretically sound potential approach to regularize D is to use a Bayesian neural net, whose model capacity away from data is not degenerate.
Revised: One theoretically sound potential approach to regularize D is to use a Bayesian neural net, whose model capacity away from data is not degenerate.

Original: However, computationally scalable deep Bayesian neural networks are still an active area of research (Hern ´ andez-Lobato & Adams, 2015; Hasenclever et al., 2017) and are not easy to use.
Revised: However, computationally scalable deep Bayesian neural networks are still an active area of research (Hern ´ andez-Lobato & Adams, 2015; Hasenclever et al., 2017) and are not easy to use.

Original: Alternatively, we can use auxiliary tasks to regularize D ’s capacity usage.
Revised: Alternatively, we can use auxiliary tasks to regularize D ’s capacity usage.

Original: If given labelled data, semi-supervised learning as an auxiliary task for D , as shown in Salimans et al.
Revised: If given labelled data, semi-supervised learning as an auxiliary task for D , as shown in Salimans et al.

Original: (2016), improves GAN training stability and the resulting generative model.
Revised: (2016), improves GAN training stability and the resulting generative model.

Original: We hypothesize that if the data domain has other structures that can be exploited as supervised learning signal, Exploiting these structures could as well potentially improve the GAN training stability like in Salimans et al.
Revised: We hypothesize that if the data domain has other structures that can be exploited as supervised learning signal, Exploiting these structures could as well potentially improve the GAN training stability like in Salimans et al.

Original: When no supervised task is available, auto-encoding is another potential possibility.
Revised: When no supervised task is available, auto-encoding is another potential possibility.

Original: Energy-Based
Revised: Energy-Based

Original: GAN (Zhao et al., 2016) and Boundary Equilibrium GAN (Berthelot et al., 2017) use auto-encoders as their discriminators.
Revised: GAN (Zhao et al., 2016) and Boundary Equilibrium GAN (Berthelot et al., 2017) use auto-encoders as their discriminators.

Original: However, both EBGAN and BEGAN have different objectives from the vanilla GAN.
Revised: However, both EBGAN and BEGAN have different objectives from the vanilla GAN.

Original: Furthermore, instead of using auto-encoding to simply regularize D , the auto-encoder loss is used to discriminate real data from fake ones.
Revised: Furthermore, instead of using auto-encoding to simply regularize D , the auto-encoder loss is used to discriminate real data from fake ones.

Original: Hence, it is unclear if their beneﬁts stem from the regularization effects or the alternative classiﬁcation approach.
Revised: Hence, it is unclear if their beneﬁts stem from the regularization effects or the alternative classiﬁcation approach.

Original: Another set of works that use auto-encoding as an auxiliary task is in learning an inference network along with GAN (Donahue et al., 2016; Dumoulin et al., 2016).
Revised: Another set of works that use auto-encoding as an auxiliary task is in learning an inference network along with GAN (Donahue et al., 2016; Dumoulin et al., 2016).

Original: However, they both modify the input to D , so that D classify not just the data, but together with the corresponding latent codes from G .
Revised: However, they both modify the input to D , so that D classify not just the data, but together with the corresponding latent codes from G .

Original: Again, in this case, it is unclear if the regularization effect on the model capacity of D is the source of any improvement in learning stability.
Revised: Again, in this case, it is unclear if the regularization effect on the model capacity of D is the source of any improvement in learning stability.

Original: Our preliminary results on using the auxiliary auto-encoding loss on real data show that it does not lead to improvement (see Discussion and Future Work in Sec. 5).
Revised: Our preliminary results on using the auxiliary auto-encoding loss on real data show that it does not lead to improvement (see Discussion and Future Work in Sec. 5).

Original:  B INARIZED R EPRESENTATION E NTROPY
Revised:  B INARIZED R EPRESENTATION E NTROPY

Original: We now introduce our regularizer, the binarized representation entropy regularizer (BRE).
Revised: We now introduce our regularizer, the binarized representation entropy regularizer (BRE).

Original: Recall that we would like to encourage diverse activation patterns in D across different samples.
Revised: Recall that we would like to encourage diverse activation patterns in D across different samples.

Original: For a sample x , its activation pattern on a particular internal layer of D, can be represented by a binary vector, as shown in Figure 2.
Revised: For a sample x , its activation pattern on a particular internal layer of D can be represented by a binary vector, as shown in Figure 2.

Original: In particular, let h ∈ R
Revised: In particular, let h ∈ R

Original: d be the immediate pre-nonlinearity activity of a sample x on a particular layer of d hidden units.
Revised: d be the immediate pre-nonlinearity activity of a sample x on a particular layer of d hidden units.

Original: The activation pattern of x on this layer can be represented by the sign vector of h , deﬁned as s = sign ( h ) :
Revised: The activation pattern of x on this layer can be represented by the sign vector of h , deﬁned as s = sign ( h ) :

Original: = h | h | ∈ {± 1 } d where | · | is entry-wise absolute value.
Revised: = h | h | ∈ {± 1 } d where | · | is entry-wise absolute value.

Original: is the binary activation vector of on layer L
Revised: is the binary activation vector of on layer L

Original: We call this binary vector s ∈ {± 1 } d the activation vector of the sample x on this particular layer.
Revised: We call this binary vector s ∈ {± 1 } d the activation vector of the sample x on this particular layer.

Original: In this work, we model the activation vector of each sample in a particular layer of D as a random binary vector.
Revised: In this work, we model the activation vector of each sample in a particular layer of D as a random binary vector.

Original: Given a mini-batch, { x 1 , . . .
Revised: Given a mini-batch, { x 1 , . . .

Original: , x K } of size K , assume that each binary activation vector s k of x k , k = 1 , . .
Revised: , x K } of size K , assume that each binary activation vector s k of x k , k = 1 , . .

Original: , K , on a particular layer with d hidden units is an independent sample of a random binary vector U =
Revised: , K , on a particular layer with d hidden units is an independent sample of a random binary vector U =

Original: ( U 1 , . . .
Revised: ( U 1 , . . .

Original: , U d ) , where U i denotes a
Revised: , U d ) , where U i denotes a

Original: Bernoulli random variable with parameters p i and distribution function  P i for i = 1 , . . .
Revised: Bernoulli random variable with parameters p i and distribution function 2 P i for i = 1 , . . .

Original: , d2 .
Revised: , d .

Original: Also denotes the joint distribution function of ( U 1 , . .
Revised: Also denotes the joint distribution function of ( U 1 , . .

Original: . , U d ) by P .
Revised: . , U d ) by P .

Original: To have diverse activation patterns, we would like to construct a regularizer that encourages P to have a large joint entropy.
Revised: To have diverse activation patterns, we would like to construct a regularizer that encourages P to have a large joint entropy.

Original: Ideally, one could use an empirical estimate of the entropy function as a desired regularizer.
Revised: Ideally, one could use an empirical estimate of the entropy function as a desired regularizer.

Original: However, sample-based estimation of the entropy of a high-dimensional distribution has been well known to be difﬁcult, especially with a small mini-batch size (Darbellay & Vajda, 1999; Miller, 2003; Kybic, 2007; Kybic & Vnu ˇ cko, 2012; Scott, 2015).
Revised: However, sample-based estimation of the entropy of a high-dimensional distribution has been well known to be difﬁcult, especially with a small mini-batch size (Darbellay & Vajda, 1999; Miller, 2003; Kybic, 2007; Kybic & Vnu ˇ cko, 2012; Scott, 2015).

Original: We instead propose a simple binarized representation entropy (BRE) regularizer, which encourages the large entropy of P in a weak manner.
Revised: We instead propose a simple binarized representation entropy (BRE) regularizer, which encourages the large entropy of P in a weak manner.

Original: For a particular layer in D , our BRE regularizer R BRE is computed over a mini-batch of { x 1 , . . .
Revised: For a particular layer in D , our BRE regularizer R BRE is computed over a mini-batch of { x 1 , . . .

Original: , x K } , and consists of two terms, marginal entropy R ME and activation correlation R AC ,
Revised: , x K } , and consists of two terms, marginal entropy R ME and activation correlation R AC ,

Original: both acting on the binarized activation vectors of the hidden units 3 :
Revised: both acting on the binarized activation vectors of the hidden units 3 :

Original: R BRE =
Revised: R BRE =

Original: R ME + R AC , where
Revised: R ME + R AC , where

Original: [Equation]
Revised: [Equation]

Original: and
Revised: and

Revised: [Equation]

Revised: |

Original: [Equation]
Revised: [Equation]

Original: | d .
Revised: | d .

Original: [Equation]
Revised: [Equation]

Original: + + - - - + + - +
Revised: + + - - - + + - +

Original: U
Revised: U

Original: i
Revised: i

Original: U d U
Revised: U d U

Original: Here ¯ s ( i ) =
Revised: Here ¯ s ( i ) =

Original: 1 K (cid:80) Kk
Revised: 1 K (cid:80) Kk

Original: =1 s k,i is the average of the i th element (corresponding to the i th hidden unit) of the activation vectors s k across the mini-batch, where s k is the activation vector of x k for k = 1 , . . .
Revised: =1 s k,i is the average of the i th element (corresponding to the i th hidden unit) of the activation vectors s k across the mini-batch, where s k is the activation vector of x k for k = 1 , . . .

Original: , K , as shown in Figure 3.
Revised: , K , as shown in Figure 3.

Original: Thus R ME can be interpreted as an empirical estimate of 1 d (cid:80) di
Revised: Thus R ME can be interpreted as an empirical estimate of 1 d (cid:80) di

Original: =1 E
Revised: =1 E

Original: [ U i ] 2 , and R AC as an empirical estimate of
Revised: [ U i ] 2 , and R AC as an empirical estimate of

Original: [Equation]
Revised: [Equation]

Original: , where U , ˜ U are two i.i.d.
Revised: , where U , ˜ U are two i.i.d.

Original: random vectors with probability function P .
Revised: random vectors with probability function P .

Original: As shown in Section 3.1, our regularizer encourages a large joint entropy of this random binary vector.
Revised: As shown in Section 3.1, our regularizer encourages a large joint entropy of this random binary vector.

Original: In particular, we show that the ﬁrst term, R ME , encourages individual hidden units to be active half of the time on average, to have high marginal entropy ; the second term, R AC , encourages low activation correlation between each pair of the hidden units.
Revised: In particular, we show that the ﬁrst term, R ME , encourages individual hidden units to be active half of the time on average, to have high marginal entropy ; the second term, R AC , encourages low activation correlation between each pair of the hidden units.

Original: We further show in Section 3.2 that having the regularizer being close to 0 is a necessary condition for ( U 1 , . . .
Revised: We further show in Section 3.2 that having the regularizer being close to 0 is a necessary condition for ( U 1 , . . .

Original: , U d ) to achieve its maximum entropy.
Revised: , U d ) to achieve its maximum entropy.

Original: Details in the practical implementation of our regularizer are discussed in Section 3.3.
Revised: Details in the practical implementation of our regularizer are discussed in Section 3.3.

Original: 3. BRE ENCOURAGES HIGH JOINT ENTROPY OF THE ACTIVATION PATTERNS
Revised: 3. BRE ENCOURAGES HIGH JOINT ENTROPY OF THE ACTIVATION PATTERNS

Original: Note that each summand ¯ s ( i ) in R ME is an empirical estimate of 2 p i − 1 , the mean of the marginal distribution P i .
Revised: Note that each summand ¯ s ( i ) in R ME is an empirical estimate of 2 p i − 1 , the mean of the marginal distribution P i .

Original: Thus minimizing ¯ s 2( i ) leads to p i
Revised: Thus minimizing ¯ s 2( i ) leads to p i

Original: = 12 , i.e. U i is zero-mean for i = 1 , . . .
Revised: = 12 , i.e. U i is zero-mean for i = 1 , . . .

Original: , d .
Revised: , d .

Original: In other words, R ME is 0 when there are equal number of ± 1 in U .
Revised: In other words, R ME is 0 when there are equal number of ± 1 in U .

Original: Moreover, for j, k = 1 , . .
Revised: Moreover, for j, k = 1 , . .

Original: , K where j (cid:54) =
Revised: , K where j (cid:54) =

Original: k , minimizing | s (cid:62)
Revised: k , minimizing | s (cid:62)

Original: j s k | in the second term R AC is essentially equivalent to minimizing (cid:0) s (cid:62)
Revised: j s k | in the second term R AC is essentially equivalent to minimizing (cid:0) s (cid:62)

Original: j s k (cid:1) 2 .
Revised: j s k (cid:1) 2 .

Original: Thus, minimizing R AC can be seen as minimizing
Revised: Thus, minimizing R AC can be seen as minimizing

Original: E (cid:20) (cid:16) U (cid:62) ˜
Revised: E (cid:20) (cid:16) U (cid:62) ˜

Original: U (cid:17) 2 (cid:21) where U, ˜
Revised: U (cid:17) 2 (cid:21) where U, ˜

Original: U are i.i.d.
Revised: U are i.i.d.

Original: from P .
Revised: from P .

Original: Since R ME is enforcing U i to be zero-mean for i = 1 , . . .
Revised: Since R ME is enforcing U i to be zero-mean for i = 1 , . . .

Original: , d , as shown in Proposition 3.1, minimizing R AC enforces the pairwise independence of the U i ’s.
Revised: , d , as shown in Proposition 3.1, minimizing R AC enforces the pairwise independence of the U i ’s.

Original: Lastly, Assuming the hidden units U i ’s are zero-mean and pairwise independent, by Corollary 3. of Gavinsky & Pudl ´ ak (2015) (which we restate in Appendix C for completeness), we have that the entropy of P satisﬁes
Revised: Lastly, Assuming the hidden units U i ’s are zero-mean and pairwise independent, by Corollary 3. of Gavinsky & Pudl ´ ak (2015) (which we restate in Appendix C for completeness), we have that the entropy of P satisﬁes

Original: [Equation]
Revised: [Equation]

Original: Proposition 3.1.
Revised: Proposition 3.1.

Original: Let U
Revised: Let U

Original: =
Revised: =

Original: ( U 1 , . . .
Revised: ( U 1 , . . .

Original: , U d ) be a zero-mean multivariate Bernoulli vector of P , and
Revised: , U d ) be a zero-mean multivariate Bernoulli vector of P , and

Original: ˜ U
Revised: ˜ U

Original: =
Revised: =

Original: ( ˜ U 1 , . . .
Revised: ( ˜ U 1 , . . .

Original: , ˜ U d ) denotes another random vector of P that is independent to U .
Revised: , ˜ U d ) denotes another random vector of P that is independent to U .

Original: Then
Revised: Then

Original: [Equation]
Revised: [Equation]

Original: We defer the proof of this proposition to Appendix B.
Revised: We defer the proof of this proposition to Appendix B.

Original: 3. M AXIMUM ENTROPY REPRESENTATION HAS R BRE ≈ 0 .
Revised: 3. M AXIMUM ENTROPY REPRESENTATION HAS R BRE ≈ 0 .

Original: We further show that R BRE ≈ 0 is a necessary condition for P to achieve the maximum entropy.
Revised: We further show that R BRE ≈ 0 is a necessary condition for P to achieve the maximum entropy.

Original: It is straightforward that the maximum entropy of P is achieved if and only if E
Revised: It is straightforward that the maximum entropy of P is achieved if and only if E

Original: [ U i ] = 

Revised: [ U i ] =

Original: ( p i = 1 / 2 ) for all i ∈ { 1 , ..., d } , i.e. each hidden unit is activated half of the time, and ( U 1 , . .
Revised: ( p i = 1 / 2 ) for all i ∈ { 1 , ..., d } , i.e. each hidden unit is activated half of the time, and ( U 1 , . .

Original: , U d ) are mutually independent.
Revised: , U d ) are mutually independent.

Original: Therefore, the i th element of the average activation vector
Revised: Therefore, the i th element of the average activation vector

Original: ¯ s ( i ) is approximately zero for i ∈ { 1 , ..., d } , and so is R ME .
Revised: ¯ s ( i ) is approximately zero for i ∈ { 1 , ..., d } , and so is R ME .

Original: Further, note that R AC is an empirical estimate of E (cid:104) (cid:12)(cid:12) (cid:12) (cid:80) di
Revised: Further, note that R AC is an empirical estimate of E (cid:104) (cid:12)(cid:12) (cid:12) (cid:80) di

Original: =1 M i /d (cid:12)(cid:12) (cid:12) (cid:105) where M i
Revised: =1 M i /d (cid:12)(cid:12) (cid:12) (cid:105) where M i

Original: =
Revised: =

Original: U
Revised: U

Original: i ˜
Revised: i ˜

Original: U
Revised: U

Original: i for i = 1 , . . .
Revised: i for i = 1 , . . .

Original: , d .
Revised: , d .

Original: Note that given p i = 1 / 

Revised: Note that given p i = 1 /

Original: and U
Revised: and U

Original: i ’s are mutually independent, one can show that M i ’s are mutually independent and have the distribution of Bernoulli( 0 . 5 ) as well.
Revised: i ’s are mutually independent, one can show that M i ’s are mutually independent and have the distribution of Bernoulli( 0 . 5 ) as well.

Original: Therefore by the Central
Revised: Therefore by the Central

Original: Limit Theorem, the distribution of (cid:80) di =1 M i converges in distribution to the Gaussian distribution
Revised: Limit Theorem, the distribution of (cid:80) di =1 M i converges in distribution to the Gaussian distribution

Original: N (0 , d ) .
Revised: N (0 , d ) .

Original: Given sufﬁciently large d , the distribution of (cid:80) di =1 M i d is approximately N (0 , 1 /d ) , and thus R AC is approximately zero 4 .
Revised: Given sufﬁciently large d , the distribution of (cid:80) di =1 M i d is approximately N (0 , 1 /d ) , and thus R AC is approximately zero 4 .

Original: 3. GAN TRAINING WITH BRE REGULARIZER P RACTICAL IMPLEMENTATIONS OF R BRE
Revised: 3. GAN TRAINING WITH BRE REGULARIZER P RACTICAL IMPLEMENTATIONS OF R BRE

Original: In practice, due to the degenerate gradient of the sign function, we replace s in R ME by its smooth approximation a = softsign ( h ) :
Revised: In practice, due to the degenerate gradient of the sign function, we replace s in R ME by its smooth approximation a = softsign ( h ) :

Original: =
Revised: =

Original: h | h | + (cid:15) , where (cid:15) is a hyperparameter to be chosen.
Revised: h | h | + (cid:15) , where (cid:15) is a hyperparameter to be chosen.

Original: If (cid:15) is too small, the nonlinearity becomes too non-smooth for stochastic gradient descent; if it is too large, it fails to be a good approximation to the sign function.
Revised: If (cid:15) is too small, the nonlinearity becomes too non-smooth for stochastic gradient descent; if it is too large, it fails to be a good approximation to the sign function.

Original: Furthermore, not only different layers could have different scales of h , hence requiring different (cid:15) , during training the scale of h could change too.
Revised: Furthermore, not only different layers could have different scales of h , hence requiring different (cid:15) , during training the scale of h could change too.

Original: Therefore, instead of setting a ﬁxed (cid:15) , we set (cid:15) =
Revised: Therefore, instead of setting a ﬁxed (cid:15) , we set (cid:15) =

Original: c avg ( | h | ) , where c is some small constant and avg ( | h | ) is a scalar, where the average runs over samples in the minibatch and the d dimensions of the layer.
Revised: c avg ( | h | ) , where c is some small constant and avg ( | h | ) is a scalar, where the average runs over samples in the minibatch and the d dimensions of the layer.

Original: In this way, softsign ( · ) is invariant with respect to any multiplicative scaling of h in the forward pass of the computation; in the backward pass for the gradient computation, we do not backpropagate through (cid:15) .
Revised: In this way, softsign ( · ) is invariant with respect to any multiplicative scaling of h in the forward pass of the computation; in the backward pass for the gradient computation, we do not backpropagate through (cid:15) .

Original: We choose c = 0 .
Revised: We choose c = 0 .

Original: 001 , as we observe empirically that this usually makes 90% to 99% of units to have absolute value at least .
Revised: 001 , as we observe empirically that this usually makes 90% to 99% of units to have absolute value at least .

Original: An alternative to this softsign is the tanh nonlinearity.
Revised: An alternative to this softsign is the tanh nonlinearity.

Original: However, tanh lacks the scale invariance of our proposed softsign with varying (cid:15) , hence potentially less effective in capturing the nature of input space partitioning.
Revised: However, tanh lacks the scale invariance of our proposed softsign with varying (cid:15) , hence potentially less effective in capturing the nature of input space partitioning.

Original: In Sec.
Revised: In Sec.

Original: 4.2 (Table.
Revised: 4.2 (Table.

Original: 1), we conﬁrm empirically that using tanh instead of softsign decreases the effectiveness of BRE.
Revised: 1), we conﬁrm empirically that using tanh instead of softsign decreases the effectiveness of BRE.

Original: We also relax R AC by allowing a soft margin term, as R AC =
Revised: We also relax R AC by allowing a soft margin term, as R AC =

Original: avg j (cid:54) =
Revised: avg j (cid:54) =

Original: k max (cid:0) 0 , | a (cid:62) j a k | /d − η (cid:1) .
Revised: k max (cid:0) 0 , | a (cid:62) j a k | /d − η (cid:1) .

Original: Recall that a (cid:62) j a k /d has an approximate distribution of  a good choice for the margin threshold is η = c (cid:112) 1 /d , where we adopt the “ 3 σ rule” and choose c = 3 to leave 99 .
Revised: Recall that a (cid:62) j a k /d has an approximate distribution of N (0 , 1 /d ) , so a good choice for the margin threshold is η = c (cid:112) 1 /d , where we adopt the “ 3 σ rule” and choose c = 3 to leave 99 .

Original: 7% of i, j pairs unpenalized in the maximum entropy case.
Revised: 7% of i, j pairs unpenalized in the maximum entropy case.

Original: To regularize GAN training, R BRE is applied to the immediate pre-nonlinearity activities on selected layers of D .
Revised: To regularize GAN training, R BRE is applied to the immediate pre-nonlinearity activities on selected layers of D .

Original: Therefore, if there is any normalization layer before nonlinearity, R BRE needs to be applied after the normalization.
Revised: Therefore, if there is any normalization layer before nonlinearity, R BRE needs to be applied after the normalization.

Original: We emphasize that we use softsign for the regularizer only, we do not modify the nonlinearity or any other structure of the neural net.
Revised: We emphasize that we use softsign for the regularizer only, we do not modify the nonlinearity or any other structure of the neural net.

Original: W HICH LAYERS SHOULD R BRE BE APPLIED ON ?
Revised: W HICH LAYERS SHOULD R BRE BE APPLIED ON ?

Original: Technically R BRE can be applied on any rectiﬁer layer before the nonlinearity.
Revised: Technically R BRE can be applied on any rectiﬁer layer before the nonlinearity.

Original: However, having it on the last hidden rectiﬁer layer before classiﬁcation might hinder D ’s ability to separate real from fake, as the high entropy representation encouraged by R BRE might not be compatible with linear classiﬁcation.
Revised: However, having it on the last hidden rectiﬁer layer before classiﬁcation might hinder D ’s ability to separate real from fake, as the high entropy representation encouraged by R BRE might not be compatible with linear classiﬁcation.

Original: Therefore, for unsupervised learning, we apply R BRE on all except the last rectiﬁer nonlinearity before the ﬁnal classiﬁcation; for semi-supervised tasks using the augmented class setup from Salimans et al.
Revised: Therefore, for unsupervised learning, we apply R BRE on all except the last rectiﬁer nonlinearity before the ﬁnal classiﬁcation; for semi-supervised tasks using the augmented class setup from Salimans et al.

Original: , we apply R BRE only on 2 nd, 4 th and 6 th convolutional layer, and leave the three nonlinear layers before the ﬁnal softmax untouched.
Revised: , we apply R BRE only on 2 nd, 4 th and 6 th convolutional layer, and leave the three nonlinear layers before the ﬁnal softmax untouched.

Original: W HICH PART OF THE DATA SHOULD R BRE BE APPLIED ON ?
Revised: W HICH PART OF THE DATA SHOULD R BRE BE APPLIED ON ?

Original: Recall from Sec.
Revised: Recall from Sec.

Original: 2 that we want D to spend enough capacity on both the real data manifold, and the current generated data manifold by G , as well as having adequate capacity in region where we do not currently observe real or fake points but might in future iterations.
Revised: 2 that we want D to spend enough capacity on both the real data manifold, and the current generated data manifold by G , as well as having adequate capacity in region where we do not currently observe real or fake points but might in future iterations.

Original: To enforce this, we apply R BRE on generated data minibatch, as well as random interpolation inbetween real and generated data.
Revised: To enforce this, we apply R BRE on generated data minibatch, as well as random interpolation inbetween real and generated data.

Original: Speciﬁcally, let x k and ˜x k be a real and a fake data points respectively, we sample α k ∼ U (0 , 1) and let ˆx k
Revised: Speciﬁcally, let x k and ˜x k be a real and a fake data points respectively, we sample α k ∼ U (0 , 1) and let ˆx k

Original: = α k x k
Revised: = α k x k

Original: + (1 − α k )
Revised: + (1 − α k )

Original: ˜x k , and apply R BRE on selected layer representation computed on interpolated data points { ˆx k | k = 1 . . .
Revised: ˜x k , and apply R BRE on selected layer representation computed on interpolated data points { ˆx k | k = 1 . . .

Original: , K } as well.
Revised: , K } as well.

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: Using a 2D synthetic dataset and CIFAR10 dataset (Krizhevsky, 2009), we show that our BRE improves unsupervised GAN training in two ways: (a) when GAN training is unstable (for e.g. due to architectures that are less well tuned than DCGAN (Radford et al., 2015)), BRE stabilizes the training and achieves much-improved results, often surpassing tuned conﬁgurations.
Revised: Using a 2D synthetic dataset and CIFAR10 dataset (Krizhevsky, 2009), we show that our BRE improves unsupervised GAN training in two ways: (a) when GAN training is unstable (for e.g. due to architectures that are less well tuned than DCGAN (Radford et al., 2015)), BRE stabilizes the training and achieves much-improved results, often surpassing tuned conﬁgurations.

Original: (b) with architecture and hyperparameters settings that are engineered to be stable already, BRE makes GAN learning converges faster.
Revised: (b) with architecture and hyperparameters settings that are engineered to be stable already, BRE makes GAN learning converges faster.

Original: We then demonstrate that BRE regularization improves semi-supervised classiﬁcation accuracy on CIFAR10 and SVHN dataset (Netzer et al., 2011).
Revised: We then demonstrate that BRE regularization improves semi-supervised classiﬁcation accuracy on CIFAR10 and SVHN dataset (Netzer et al., 2011).

Original: Additional results on imbalanced 2D mixture as well as CelebA dataset are presented in the Appendix D.
Revised: Additional results on imbalanced 2D mixture as well as CelebA dataset are presented in the Appendix D.

Original: 4. S YNTHETIC DATASET : M IXTURE OF G AUSSIANS
Revised: 4. S YNTHETIC DATASET : M IXTURE OF G AUSSIANS

Original: We ﬁrst demonstrate BRE regularizer’s effect on ﬁtting a 2D mixture of Gaussian.
Revised: We ﬁrst demonstrate BRE regularizer’s effect on ﬁtting a 2D mixture of Gaussian.

Original: In Figure 4, the top and bottom three rows correspond respectively to experiments without the BRE regularizer (control) and with the regularizer (treat).
Revised: In Figure 4, the top and bottom three rows correspond respectively to experiments without the BRE regularizer (control) and with the regularizer (treat).

Original: Within each setting, each row represents one iteration during GAN training, selected to be at the beginning, middle, and the end of the training process.
Revised: Within each setting, each row represents one iteration during GAN training, selected to be at the beginning, middle, and the end of the training process.

Original: The ﬁrst column shows real data points (blue) and generated data points (red).
Revised: The ﬁrst column shows real data points (blue) and generated data points (red).

Original: The second to ﬁfth columns show hidden layers 1 to 4 of D , where contiguous pixels with the same colour have the same binary activation pattern on that particular layer.
Revised: The second to ﬁfth columns show hidden layers 1 to 4 of D , where contiguous pixels with the same colour have the same binary activation pattern on that particular layer.

Original: The last column shows the probability of real data according to D .
Revised: The last column shows the probability of real data according to D .

Original: The BRE regularizer is added on h 2 and h 3 .
Revised: The BRE regularizer is added on h 2 and h 3 .

Original: More results in Appendix D. c on t r o l it e r :c on t r o l it e r : c on t r o l it e r : t r ea t it e r :t r ea t it e r : t r ea t it e r :
Revised: More results in Appendix D. c on t r o l it e r :c on t r o l it e r : c on t r o l it e r : t r ea t it e r :t r ea t it e r : t r ea t it e r :

Original: MoG samples h
Revised: MoG samples h

Original: h h h
Revised: h h h

Original: D
Revised: D

Original: By adding BRE, the input domain is partitioned more ﬁnely as reﬂected by visualization for h 2 , h 3 , and h 4 .
Revised: By adding BRE, the input domain is partitioned more ﬁnely as reﬂected by visualization for h 2 , h 3 , and h 4 .

Original: The richer D representation allows more effective exploration of different input regions because the gradient signals provided by D to G are more diverse than the degenerate baseline case where D is linear in large regions of the input.
Revised: The richer D representation allows more effective exploration of different input regions because the gradient signals provided by D to G are more diverse than the degenerate baseline case where D is linear in large regions of the input.

Original: Once a real data mode is discovered, G locks onto it without oscillation.
Revised: Once a real data mode is discovered, G locks onto it without oscillation.

Original: This shows that with better D capacity usage, the GAN optimisation converges faster and is more stable, while the resulting equilibrium suffers much less from mode dropping.
Revised: This shows that with better D capacity usage, the GAN optimisation converges faster and is more stable, while the resulting equilibrium suffers much less from mode dropping.

Original: 4. F ASTER AND BETTER CONVERGENCE IN UNSUPERVISED LEARNING
Revised: 4. F ASTER AND BETTER CONVERGENCE IN UNSUPERVISED LEARNING

Original: We quantitatively measure the resulting G using the Inception score (Salimans et al., 2016).
Revised: We quantitatively measure the resulting G using the Inception score (Salimans et al., 2016).

Original: Table 1 shows improved ﬁnal Inception scores on DCGAN, as well as the following non-standard architectures (only mentioning difference from the standard DCGAN): densely connected convnet
Revised: Table 1 shows improved ﬁnal Inception scores on DCGAN, as well as the following non-standard architectures (only mentioning difference from the standard DCGAN): densely connected convnet

Original: Huang et al.
Revised: Huang et al.

Original: (2016) for D ; G and D with an equal number of ﬁlters on each layer; D with ReLU nonlinearity.
Revised: (2016) for D ; G and D with an equal number of ﬁlters on each layer; D with ReLU nonlinearity.

Original: In all cases, the models regularized by BRE improve over the baseline counter-parts without regularization (no-BRE).
Revised: In all cases, the models regularized by BRE improve over the baseline counter-parts without regularization (no-BRE).

Original: Furthermore, vanilla GAN’s with BRE applied on multiple D layers (BRE multi) always outperform WGAN-GP (Gulrajani et al., 2017).
Revised: Furthermore, vanilla GAN’s with BRE applied on multiple D layers (BRE multi) always outperform WGAN-GP (Gulrajani et al., 2017).

Original: Fig.
Revised: Fig.

Original: 6 shows some generated samples from a DCGAN-ReLU model without and with BRE regularization.
Revised: 6 shows some generated samples from a DCGAN-ReLU model without and with BRE regularization.

Original: Fig.
Revised: Fig.

Original: 9 of
Revised: 9 of

Original: Appendix D.1 show more samples laid out by t-SNE visualization to better illustrate mode collapsing.
Revised: Appendix D.1 show more samples laid out by t-SNE visualization to better illustrate mode collapsing.

Original: [Table]
Revised: [Table]

Original: Fig.
Revised: Fig.

Original: 5 shows that with BRE, DCGAN training converges faster, as measured by Inception score.
Revised: 5 shows that with BRE, DCGAN training converges faster, as measured by Inception score.

Original: The 1 σ error bars are estimated from ten different random runs.
Revised: The 1 σ error bars are estimated from ten different random runs.

Original: Because DCGAN architecture is engineered to be stable, in the end, baseline DCGAN can still achieve comparable Inception score with the regularized version on average.
Revised: Because DCGAN architecture is engineered to be stable, in the end, baseline DCGAN can still achieve comparable Inception score with the regularized version on average.

Original: But clearly the convergence is much faster with BRE during the initial transient phase, conﬁrming our intuition that BRE improves exploration.
Revised: But clearly the convergence is much faster with BRE during the initial transient phase, conﬁrming our intuition that BRE improves exploration.

Original: Fig.shows Inception score and (thresholded) activation correlation values ( R AC ) during one particular set of runs with default DCGAN optimization settings, and a more aggressive optimization setting.
Revised: Fig.shows Inception score and (thresholded) activation correlation values ( R AC ) during one particular set of runs with default DCGAN optimization settings, and a more aggressive optimization setting.

Original: In both cases, BRE regularization results in similarly faster convergence, and higher ﬁnal Inception score in the unstable case with more aggressive optimization.
Revised: In both cases, BRE regularization results in similarly faster convergence, and higher ﬁnal Inception score in the unstable case with more aggressive optimization.

Original: The bottom row in Fig.
Revised: The bottom row in Fig.

Original: 7 shows that
Revised: 7 shows that

Original: BRE regularization is indeed making a qualitative difference to the activation correlation ( R AC ) by keeping it low during training in both cases.
Revised: BRE regularization is indeed making a qualitative difference to the activation correlation ( R AC ) by keeping it low during training in both cases.

Original: BRE makes convergence faster.
Revised: BRE makes convergence faster.

Original: 4. I MPROVED S EMI - SUPERVISED L EARNING ON CIFAR10 AND SVHN
Revised: 4. I MPROVED S EMI - SUPERVISED L EARNING ON CIFAR10 AND SVHN

Original: BRE regularization is not only compatible with semi-supervised learning using GAN’s, but also improves classiﬁcation accuracy.
Revised: BRE regularization is not only compatible with semi-supervised learning using GAN’s, but also improves classiﬁcation accuracy.

Original: Table.
Revised: Table.

Original: 2 shows results on CIFAR10 with feature matching semi-supervised learning GAN.
Revised: 2 shows results on CIFAR10 with feature matching semi-supervised learning GAN.

Original: BRE allows the learning process to discover a better solution during training that also generalizes better, indicated by a lower training classiﬁcation loss as well as lower test classiﬁcation error rates.
Revised: BRE allows the learning process to discover a better solution during training that also generalizes better, indicated by a lower training classiﬁcation loss as well as lower test classiﬁcation error rates.

Original: We
Revised: We

Original: [Figure]
Revised: [Figure]

Original: used the same code and hyperparameters 5 from Salimans et al.
Revised: used the same code and hyperparameters 5 from Salimans et al.

Original: Details on BRE hyperparameters are in Appendix A, learning curve plots in Appendix D.
Revised: Details on BRE hyperparameters are in Appendix A, learning curve plots in Appendix D.

Original: [Table]
Revised: [Table]

Original: On Street View House Numbers (SVHN) dataset (Netzer et al., 2011)
Revised: On Street View House Numbers (SVHN) dataset (Netzer et al., 2011)

Original: [Table]
Revised: [Table]

Original: with the same setup from
Revised: with the same setup from

Original: Salimans et al
Revised: Salimans et al

Original: [Table] [Table]
Revised: [Table] [Table]

Original: learning is not always stable when trained for a long time.
Revised: learning is not always stable when trained for a long time.

Original: Fig.
Revised: Fig.

Original: 8 (top row) shows that without BRE regularization
Revised: 8 (top row) shows that without BRE regularization

Original: [Table]
Revised: [Table]

Original: when trained for a very long time, sometimes learning diverges.
Revised: when trained for a very long time, sometimes learning diverges.

Original: Such failure is dramatically reduced by BRE (bottom row of Fig.
Revised: Such failure is dramatically reduced by BRE (bottom row of Fig.

Original: [Table]
Revised: [Table]

Original:  D ISCUSSION AND FUTURE WORK
Revised:  D ISCUSSION AND FUTURE WORK

Original: There are still many unexplored avenues along this line of research.
Revised: There are still many unexplored avenues along this line of research.

Original: For example, how can our new regularizer collaborate with other GANs training techniques to further improve the training GANs?
Revised: For example, how can our new regularizer collaborate with other GANs training techniques to further improve the training GANs?

Original: We leave such further explorations for future works.
Revised: We leave such further explorations for future works.

Original: Meanwhile, there are two interesting questions related to the central theme of this work.
Revised: Meanwhile, there are two interesting questions related to the central theme of this work.

Original: D OES DIRECTLY REGULARIZING THE DIVERSITY OF ∇ x D ( x ) WORK ? I F NOT , WHY NOT ?
Revised: D OES DIRECTLY REGULARIZING THE DIVERSITY OF ∇ x D ( x ) WORK ? I F NOT , WHY NOT ?

Original: To diversify G ’s learning signals, it might be tempting to enforce gradient directions ∇ x D ( x k ) to be diverse.
Revised: To diversify G ’s learning signals, it might be tempting to enforce gradient directions ∇ x D ( x k ) to be diverse.

Original: However, in rectiﬁer networks, if two inputs share the same activation pattern, the input gradients located at the two points are co-linear; hence any gradient-based learning with such diversity regularizer would have difﬁculty pulling them apart.
Revised: However, in rectiﬁer networks, if two inputs share the same activation pattern, the input gradients located at the two points are co-linear; hence any gradient-based learning with such diversity regularizer would have difﬁculty pulling them apart.

Original: In general, unlike BRE which operates directly on both activated and non-activated portions of D ’s internal units, an input gradient regularizer can only access information on the activated path in the network, so that it can only encourage existing non-shared activated path, but cannot directly create any new non-shared activated path.
Revised: In general, unlike BRE which operates directly on both activated and non-activated portions of D ’s internal units, an input gradient regularizer can only access information on the activated path in the network, so that it can only encourage existing non-shared activated path, but cannot directly create any new non-shared activated path.

Original: In theory, tanh nonlinearities as activations for D could avoid this problem, but such network is hard to train in the ﬁrst place.
Revised: In theory, tanh nonlinearities as activations for D could avoid this problem, but such network is hard to train in the ﬁrst place.

Original: In our preliminary studies, on networks with tanh, input gradient diversity regularizer with either cosine similarity or a soft-sign based regularizer like BRE does not work.
Revised: In our preliminary studies, on networks with tanh, input gradient diversity regularizer with either cosine similarity or a soft-sign based regularizer like BRE does not work.

Original: C OULD AUXILIARY TASKS HELP REGULARIZE D ?
Revised: C OULD AUXILIARY TASKS HELP REGULARIZE D ?

Original: As discussed in Sec. 2.1, auxilary tasks could potentially regularize D and stabilize training.
Revised: As discussed in Sec. 2.1, auxilary tasks could potentially regularize D and stabilize training.

Original: One possible auxilary task is reconstruction loss.
Revised: One possible auxilary task is reconstruction loss.

Original: We performed some preliminary experiments, and found that reconstructing real data as auxilary tasks worsens the resulting learned G .
Revised: We performed some preliminary experiments, and found that reconstructing real data as auxilary tasks worsens the resulting learned G .

Original: See Appendix.
Revised: See Appendix.

Original: D.3 for results.
Revised: D.3 for results.

Original: Further study is needed, and is beyond the scope of this work.
Revised: Further study is needed, and is beyond the scope of this work.

Original:  C ONCLUSIONS
Revised:  C ONCLUSIONS

Original: We proposed a novel regularizer in this paper to guide the discriminator in GANs to better allocate its model capacity.
Revised: We proposed a novel regularizer in this paper to guide the discriminator in GANs to better allocate its model capacity.

Original: Based on the relation between the model capacity and the activation pattern of the network, we constructed our regularizer to encourage a high joint entropy of the activation pattern on the hidden layers of the discriminator D .
Revised: Based on the relation between the model capacity and the activation pattern of the network, we constructed our regularizer to encourage a high joint entropy of the activation pattern on the hidden layers of the discriminator D .

Original: Experimental results demonstrated the beneﬁts of our new regularizer: faster progress in the initial phase of learning thanks to improved exploration, more stable convergence, and better ﬁnal results in both unsupervised and semi-supervised learning
Revised: Experimental results demonstrated the beneﬁts of our new regularizer: faster progress in the initial phase of learning thanks to improved exploration, more stable convergence, and better ﬁnal results in both unsupervised and semi-supervised learning

Original: A M ODEL AND HYPERPARAMETER DETAILS A. 2D EXAMPLE
Revised: A M ODEL AND HYPERPARAMETER DETAILS A. 2D EXAMPLE

Original: G is 4-layer (excluding noise layer)
Revised: G is 4-layer (excluding noise layer)

Original: MLP with ReLU hidden activation function, and tanh visible activation; D is 5-layer MLP with LeakyRelu( . 2 ).
Revised: MLP with ReLU hidden activation function, and tanh visible activation; D is 5-layer MLP with LeakyRelu( . 2 ).

Original: Both D and G has 10 units on each hidden layer, no batch or other normalization is used in D or G , no other stabilization techniques are used;
Revised: Both D and G has 10 units on each hidden layer, no batch or other normalization is used in D or G , no other stabilization techniques are used;

Original: For
Revised: For

Original: Fig.
Revised: Fig.

Original: 4, Fig.
Revised: 4, Fig.

Original: 12, and Fig.
Revised: 12, and Fig.

Original: 13, lr=.001 with adam(.0, .999), and BRE regularizer weight 1 . , applied on h2 and h3; both lr and BRE weight linearly decay to over iterations to 1 e − 6 and 0 respectively.
Revised: 13, lr=.001 with adam(.0, .999), and BRE regularizer weight 1 . , applied on h2 and h3; both lr and BRE weight linearly decay to over iterations to 1 e − 6 and 0 respectively.

Original: For
Revised: For

Original: Fig.
Revised: Fig.

Original: 12 and Fig. 13, lr=.002 with adam(.5, .999), and BRE regularizer weight 1 . , applied on h2.
Revised: 12 and Fig. 13, lr=.002 with adam(.5, .999), and BRE regularizer weight 1 . , applied on h2.

Original: A. U NSUPERVISED L EARNING CIFAR 
Revised: A. U NSUPERVISED L EARNING CIFAR

Original: Table.
Revised: Table.

Original: “single” all
Revised: “single” all

Original: “multi”
Revised: “multi”

Original: For Fig.
Revised: For Fig.

Original: 7, the default optimization setting (left column, i.e. (a) and (c)) is lr = 2e − 4 and one D update per G update, lr for both D and G annealled to 1 e − 6 over 90 K G updates; while the aggressive setting (right column, i.e. (b) and (d)) is lr = 2e − 3 and three D update for every G update, lr for both D and G annealed to 1 e − 6 over 10 K G updates.
Revised: 7, the default optimization setting (left column, i.e. (a) and (c)) is lr = 2e − 4 and one D update per G update, lr for both D and G annealled to 1 e − 6 over 90 K G updates; while the aggressive setting (right column, i.e. (b) and (d)) is lr = 2e − 3 and three D update for every G update, lr for both D and G annealed to 1 e − 6 over 10 K G updates.

Original: A. S EMI -S UPERVISED L EARNING
Revised: A. S EMI -S UPERVISED L EARNING

Original: We used exactly the same code and GAN hyperparameters 6 from Salimans et al.
Revised: We used exactly the same code and GAN hyperparameters 6 from Salimans et al.

Original: R BRE regularization is applied on every other second layer, starting from the 2nd until 4 layers before the classiﬁcation layer (applied on three layers in total).
Revised: R BRE regularization is applied on every other second layer, starting from the 2nd until 4 layers before the classiﬁcation layer (applied on three layers in total).

Original: On CIFAR10, we used a regularizer weight of . 01 , and on SVHN we used 0 . 1 .
Revised: On CIFAR10, we used a regularizer weight of . 01 , and on SVHN we used 0 . 1 .

Original: BRE is applied on real, fake and interp data.
Revised: BRE is applied on real, fake and interp data.

Original: B P ROOF OF P ROPOSITION 3. 
Revised: B P ROOF OF P ROPOSITION 3.

Original: Proof.
Revised: Proof.

Original: Let M i
Revised: Let M i

Original: =
Revised: =

Original: U
Revised: U

Original: i ˜
Revised: i ˜

Original: U
Revised: U

Original: i .
Revised: i .

Original: Then
Revised: Then

Original: [Equation]
Revised: [Equation]

Original: where Equation (1) is due to the independence of U and ˜ U , Equation (2) is due to that
Revised: where Equation (1) is due to the independence of U and ˜ U , Equation (2) is due to that

Original: U 2 i = 1 with probability 1 , and Equation (3) is because E
Revised: U 2 i = 1 with probability 1 , and Equation (3) is because E

Original: [ U i ] = 0 .
Revised: [ U i ] = 0 .

Original: C C OROLLARY 3.3 OF G AVINSKY & P UDL ´ AK (2015)
Revised: C C OROLLARY 3.3 OF G AVINSKY & P UDL ´ AK (2015)

Original: Theorem
Revised: Theorem

Original: C.
Revised: C.

Original: (Corollary 3. of
Revised: (Corollary 3. of

Original: Gavinsky &
Revised: Gavinsky &

Original: Pudl´ak (2015)) .
Revised: Pudl´ak (2015)) .

Original: Let
Revised: Let

Original: H min ( P ) = − log (max x P ( X = x )) .
Revised: H min ( P ) = − log (max x P ( X = x )) .

Original: Also let ( U 1 , . .
Revised: Also let ( U 1 , . .

Original: , U d ) be pairwise independent random variable of
Revised: , U d ) be pairwise independent random variable of

Original: Bernoulli( 0 . 5 ).
Revised: Bernoulli( 0 . 5 ).

Original: Then,
Revised: Then,

Original: H ( P ) ≥ H min ( P ) ≥ log( d + 1) .
Revised: H ( P ) ≥ H min ( P ) ≥ log( d + 1) .

Original: D A DDITIONAL E XPERIMENTAL R ESULTS D. M ORE SAMPLES FROM DCGAN-R E LU
Revised: D A DDITIONAL E XPERIMENTAL R ESULTS D. M ORE SAMPLES FROM DCGAN-R E LU

Original: We show more samples generated from the DCGAN-ReLU model, mentioned in Sec.
Revised: We show more samples generated from the DCGAN-ReLU model, mentioned in Sec.

Original: 4.2, without the BRE regularizer (with red frames) and with the BRE regularizer (with blue frames) in Fig.
Revised: 4.2, without the BRE regularizer (with red frames) and with the BRE regularizer (with blue frames) in Fig.

Original: In Fig.
Revised: In Fig.

Original: 9 images are arranged based on L 2 distances in the original pixel value space.
Revised: 9 images are arranged based on L 2 distances in the original pixel value space.

Original: Images that are similar in pixel values are roughly grouped together.
Revised: Images that are similar in pixel values are roughly grouped together.

Original: To achieve this, we apply t-SNE (Maaten & Hinton, 2008) to reduce the dimensionality of these images into 2D points.
Revised: To achieve this, we apply t-SNE (Maaten & Hinton, 2008) to reduce the dimensionality of these images into 2D points.

Original: These 2D coordinates are then transformed to 2D grids RasterFairy 7 .
Revised: These 2D coordinates are then transformed to 2D grids RasterFairy 7 .

Original: At the same time, the neighborhood relations of the rastered 2D points are preserved to a certain degree.
Revised: At the same time, the neighborhood relations of the rastered 2D points are preserved to a certain degree.

Original: We then use these rastered 2D points to arrange the location of these images.
Revised: We then use these rastered 2D points to arrange the location of these images.

Original: D. S EMI - SUPERVISED LEARNING CURVES
Revised: D. S EMI - SUPERVISED LEARNING CURVES

Original: Fig.
Revised: Fig.

Original: 10 shows the Learning curves for semi-supervised learning on CIFAR
Revised: 10 shows the Learning curves for semi-supervised learning on CIFAR

Original: . D. R ECONSTRUCTION AS AUXILIARY TASK TO REGULARIZE D WORSENS RESULTS
Revised: . D. R ECONSTRUCTION AS AUXILIARY TASK TO REGULARIZE D WORSENS RESULTS

Original: D Recon, no BRE ln, λ recon =6 . 1958 ± 0 .ln, λ recon =6 .
Revised: D Recon, no BRE ln, λ recon =6 . 1958 ± 0 .ln, λ recon =6 .

Original: 2218 ± 0 .ln, λ recon = .6 .
Revised: 2218 ± 0 .ln, λ recon = .6 .

Original: 2437 ± 0 .ln, λ recon =6 .
Revised: 2437 ± 0 .ln, λ recon =6 .

Original: 4025 ± 0 .bn, λ recon =6 .
Revised: 4025 ± 0 .bn, λ recon =6 .

Original: 5356 ± 0 .bn, λ recon = .6 .
Revised: 5356 ± 0 .bn, λ recon = .6 .

Original: [Table]
Revised: [Table]

Original: D. C ELEB A
Revised: D. C ELEB A

Original: We compare stable and unstable runs of DCGAN on CelebA dataset (Liu et al., 2015), as well as the effect of the BRE regularizer.
Revised: We compare stable and unstable runs of DCGAN on CelebA dataset (Liu et al., 2015), as well as the effect of the BRE regularizer.

Original: Fig.
Revised: Fig.

Original: 11(a) shows thresholded R AC term (deﬁned in Sec. 3.3) through training.
Revised: 11(a) shows thresholded R AC term (deﬁned in Sec. 3.3) through training.

Original: The model being investigated is a 4-layer DCGAN for both G and D , with batch normalization.
Revised: The model being investigated is a 4-layer DCGAN for both G and D , with batch normalization.

Original: The unstable run (Fig. 11(b)) uses a large initial learning rate of . 01 and 3 D update steps for each G update, whereas the stable run (Fig. 11(d)) uses initial lr = 2 e − 3 and 1 D update for each G update.
Revised: The unstable run (Fig. 11(b)) uses a large initial learning rate of . 01 and 3 D update steps for each G update, whereas the stable run (Fig. 11(d)) uses initial lr = 2 e − 3 and 1 D update for each G update.

Original: Even without the BRE regularizer, we can see that when GAN training is stable,
Revised: Even without the BRE regularizer, we can see that when GAN training is stable,

Original: D uses more capacity around fake and real data as well as inbetween, as measured by R AC values in Fig.
Revised: D uses more capacity around fake and real data as well as inbetween, as measured by R AC values in Fig.

Original: 11(a).
Revised: 11(a).

Original: When BRE regularizer is applied, the usage of D ’s capacity is more improved, and resulted in more diversity in the learned distribution by G (Fig.
Revised: When BRE regularizer is applied, the usage of D ’s capacity is more improved, and resulted in more diversity in the learned distribution by G (Fig.

Original: 11(c)).
Revised: 11(c)).

Original: D. A DDITIONAL 2D M O G R ESULTS
Revised: D. A DDITIONAL 2D M O G R ESULTS

Original: We show more 2D mixture of Gaussian results in Fig.
Revised: We show more 2D mixture of Gaussian results in Fig.

Original: 12, 13, 14, and 15.
Revised: 12, 13, 14, and 15.

Original: c on t r o l it e r :c on t r o l it e r : c on t r o l it e r : c on t r o l it e r : c on t r o l it e r : c on t r o l it e r : c on t r o l it e r : c on t r o l it e r : c on t r o l it e r : c on t r o l it e r : c on t r o l it e r :
Revised: c on t r o l it e r :c on t r o l it e r : c on t r o l it e r : c on t r o l it e r : c on t r o l it e r : c on t r o l it e r : c on t r o l it e r : c on t r o l it e r : c on t r o l it e r : c on t r o l it e r : c on t r o l it e r :

Original: MoG samples h
Revised: MoG samples h

Original: h h h
Revised: h h h

Original: D t r ea t it e r :t r ea t it e r : t r ea t it e r : t r ea t it e r : t r ea t it e r : t r ea t it e r : t r ea t it e r : t r ea t it e r : t r e a t it e r : t r ea t it e r : t r ea t it e r :
Revised: D t r ea t it e r :t r ea t it e r : t r ea t it e r : t r ea t it e r : t r ea t it e r : t r ea t it e r : t r ea t it e r : t r ea t it e r : t r e a t it e r : t r ea t it e r : t r ea t it e r :

Original: MoG samples h
Revised: MoG samples h

Original: h h h
Revised: h h h

Original: D c on t r o l it e r : c on t r o l it e r : c on t r o l it e r : c on t r o l it e r : c on t r o l it e r : c on t r o l it e r : c on t r o l it e r : c on t r o l it e r : c on t r o l it e r : c on t r o l it e r : c on t r o l it e r :
Revised: D c on t r o l it e r : c on t r o l it e r : c on t r o l it e r : c on t r o l it e r : c on t r o l it e r : c on t r o l it e r : c on t r o l it e r : c on t r o l it e r : c on t r o l it e r : c on t r o l it e r : c on t r o l it e r :

Original: MoG samples h
Revised: MoG samples h

Original: h h h
Revised: h h h

Original: D t r ea t it e r : t r ea t it e r : t r ea t it e r : t r ea t it e r : t r ea t it e r : t r ea t it e r : t r ea t it e r : t r ea t it e r : t r e a t it e r : t r ea t it e r : t r ea t it e r :
Revised: D t r ea t it e r : t r ea t it e r : t r ea t it e r : t r ea t it e r : t r ea t it e r : t r ea t it e r : t r ea t it e r : t r ea t it e r : t r e a t it e r : t r ea t it e r : t r ea t it e r :

Original: MoG samples h
Revised: MoG samples h

Original: h h h
Revised: h h h

Original: D c on t r o l it e r :c on t r o l it e r : c on t r o l it e r : c on t r o l it e r : c on t r o l it e r : c on t r o l it e r : c on t r o l it e r : c on t r o l it e r : c on t r o l it e r : c on t r o l it e r : c on t r o l it e r :
Revised: D c on t r o l it e r :c on t r o l it e r : c on t r o l it e r : c on t r o l it e r : c on t r o l it e r : c on t r o l it e r : c on t r o l it e r : c on t r o l it e r : c on t r o l it e r : c on t r o l it e r : c on t r o l it e r :

Original: MoG samples h
Revised: MoG samples h

Original: h h h
Revised: h h h

Original: D t r ea t it e r :t r ea t it e r : t r ea t it e r : t r ea t it e r : t r ea t it e r : t r ea t it e r : t r ea t it e r : t r ea t it e r : t r e a t it e r : t r ea t it e r : t r ea t it e r :
Revised: D t r ea t it e r :t r ea t it e r : t r ea t it e r : t r ea t it e r : t r ea t it e r : t r ea t it e r : t r ea t it e r : t r ea t it e r : t r e a t it e r : t r ea t it e r : t r ea t it e r :

Original: MoG samples h
Revised: MoG samples h

Original: h h h
Revised: h h h

Original: D
Revised: D