Original: I MPROVING DIVERSITY IN G ENERATIVE ADVERSARIAL NETWORKS BY ENCOURAGING DISCRIMINATOR REPRESENTATION ENTROPY
Revised: I MPROVING DIVERSITY IN G ENERATIVE ADVERSARIAL NETWORKS BY ENCOURAGING DISCRIMINATOR REPRESENTATION ENTROPY

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: We propose a novel regularizer for the training of Generative Adversarial Networks (GANs) so that the generator can better capture the variations within the real data distribution, thus helps to avoid subtle model collapse and improve the performance of GANs. The idea is to encourage the discriminator D to provides more informative signals for the learning of the generator G by allocating the model capacity of D in a more desirable way.
Revised: We propose a novel regularizer to improve the training of Generative Adversarial Networks (GANs). The motivation is that when the discriminator D spreads out its model capacity in the right way, the learning signals given to the generator G are more informative and diverse, which helps G to explore better and discover the real data manifold while avoiding large unstable jumps due to the erroneous extrapolation made by D .

Original: In particular, we measure the model capacity of Dby its activation patterns and our new regularizer is constructed to encourage a high joint entropy of the  activation patterns of the hidden layers of D .
Revised: Our regularizer guides the rectiﬁer discriminator D to better allocate its model capacity, by encouraging the binary activation patterns on selected internal layers of D to have a high joint entropy.

Original: Experimental results on both synthetic data and real datasets show that our regularizer helps to improve the sample quality in the unsupervised learning setting, and also the  classiﬁcation accuracy inthe semi-supervised learningsetting
Revised: Experimental results on both synthetic data and real datasets demonstrate improvements in stability and convergence speed of the GAN training, as well as higher sample quality. The approach also leads to higher classiﬁcation accuracies in semi-supervised learning

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Generative Adversarial Networks (GANs) (Goodfellow et al., 2014) learn from unlabeled databy engaging the generative model (G ) in an adversarial game with a discriminator (D ).
Revised: Generative Adversarial Network (GAN) (Goodfellow et al., 2014) has been a new promising approach to unsupervised learning of complex high dimensional data in the last two years, with successful applications on image data (Isola et al., 2016;

Original: GANs provide a promising new approach to unsupervised learning of complex high dimensional data, with successful applications on image data (Isola et al., 2016; Shrivastava et al., 2016), and great potential for predictive representation learning (Mathieu et al., 2015) as well as reinforcement learning (Finn et al., 2016; Henderson et al., 2017). 
Revised: Shrivastava et al., 2016), and high potential for predictive representation learning (Mathieu et al., 2015) as well as reinforcement learning (Finn et al., 2016; Henderson et al., 2017). In a nutshell, GANs learn from unlabeled data by engaging the generative model ( G ) in an adversarial game with a discriminator ( D ). D learns to tell apart fake data generated by G from real data, while G learns to fool D , having access to D ’s input gradient.

Original: However, challenges in GANs training impede their wider applications in new data domain and as building blocks in other models.
Revised: Despite its success in generating high-quality data, such adversarial game setting also raises challenges for the training of GANs.

Original: Many architectures and techniques have been proposed (Radford et al., 2015; Salimans et al., 2016; Gulrajani et al., 2017) to reduce extreme failures and improve  realism of generated data.
Revised: Many architectures and techniques have been proposed (Radford et al., 2015; Salimans et al., 2016; Gulrajani et al., 2017) to reduce extreme failures and improve the realism of generated data.

Original: With the various techniques, practitioners can often train GAN without extreme collapse and heavily oscillation nowadays on standard datasets.

Original: However, a number of theoretical and practical open problems still remain.
Revised: However, many theoretical and practical open problems still remain, which have impeded the ease-of-use of GANs in new problems.

Original: In particular,G often  fails to capture certain variationin data or modes of real data distribution, while D fails to exploit the failure to provide better training signal to G , leading to subtle mode collapse.
Revised: In particular, often G fails to capture certain variation or modes in the real data distribution, while D fails to exploit this failure to provide better training signal for G , leading to subtle mode collapse.

Original:  Arora et al.
Revised: Recently Arora et al.

Original: (2017) shows that with insufﬁcient capacity, D could fail to distinguish real and generated data distributions even whenthey are their Jensen-Shannon divergence or Wasserstein distance is not small.
Revised: (2017) showed that the capacity of D plays an essential role in giving G sufﬁcient learning guidances to model the complex real data distribution. With insufﬁcient capacity, D could fail to distinguish real and generated data distributions even when their Jensen-Shannon divergence or Wasserstein distance is not small.

Original: However Arora et al.

Original: (2017)’s capacity notion is a static one. During training D ’s ﬁnite capacity allocation in input space is changing. A more reﬁned and more dynamic analysis of capacity is desired. In this work, we demonstrate that even with sufﬁcient maximum capacity, D might not allocate it in a  way that helps GAN training reaching equilibrium.
Revised: In this work, we demonstrate that even with sufﬁcient maximum capacity, D might not allocate its capacity in a desirable way that facilitates convergence to a good equilibrium.

Original: We then propose a novel regularizer that to encourage D ’s hidden binary activation  to have high joint entropy.
Revised: We then propose a novel regularizer to guide D to have a better model capacity allocation. Our regularizer is constructed to encourage D ’s hidden binary activation patterns to have high joint entropy, based on a connection between the model capacity of a rectiﬁer net and its internal binary activation patterns.

Original: Our experiments show that  high entropy representation leads to  improved sample quality and semi-supervise learning results.
Revised: Our experiments show that such high entropy representation leads to faster convergences, improved sample quality, as well as lower errors in semi-supervised learning.

Original: Several regularization/gradient penalty techniques have recently been proposed to stablize GANs training (Gulrajani et al., 2017; Mescheder et al., 2017). Since we are motivated from the capacity angle, our method differs from them the most in that we regularize D ’s hidden layer acitivities, neither on input gradients nor parameters directly. This in turn will constraint the parameters of
Revised: The motivation of our regularizer starts with an observation that during GAN training, the generator

Original: D and hence G . In terms of functional forms, (Gulrajani et al., 2017) adds gradient penality to thediscriminator D ’s input gradient, ∇ x D ( x ) , to indirectly regularize D and G ’s weights.
Revised: G receives information about the input space only indirectly through the gradient of D , ∇ x D ( x ) . Typically D is a rectiﬁer net. Absent of the last sigmoid nonlinearity, D computes piecewise linear functions, meaning that the learning signal to G is (almost) piecewise constant.

Original: Both (Mescheder et al., 2017) regularize the training dynamics by constraining the parameters directly via adding other vector ﬁelds to the  gradient updates.  R ECTIFIER NET CAPACITY USAGE AND EFFECTS ON GAN
Revised: The ﬁnal sigmoid nonlinearity does not change the direction of input gradient in each linear region, but merely the scale of gradient vectors.

Original: The motivation of our proposed regularizer starts with an observation that during the training ofGANs, the generator G only receives information about real data distribution and its implicit manifold indirectly through the gradient of D , ∇ x D( x ) . 
Revised: The learning of G in GANs can be interpreted as the movement of the fake samples generated by G toward the real data distribution, guided by the (almost) piecewise constant vectorial signals according to input space partitioning by D . Hence, the diversity and informativeness of learning signals to G is closely related to how the input space is partitioned, i.e. how D ’s model capacity is allocated.

Original: However, the task of the discriminator D is solely to separate real samples from the generated fake ones.
Revised: In an area of the input space, how much capacity D allocates into it can be approximately measured by the number of linear pieces in that area (Montufar et al., 2014). When D spreads out its model capacity in a right way, the evenly dispersed partitioning helps G to explore better and discover the real data manifold, while avoiding large unstable jumps due to overconﬁdent extrapolation made by D .

Original: When realand fake data are far apart in the early stages ofthe training or when the generator G collapses into some particular modes, the discriminator D  has no incentive to allocate its capacity in the most desired way, andthus it may only provides very limited information for G , incapable toguide the generator G to capture different variations within the real data distribution.
Revised: Ideally, when GAN training is stable, the min-max game eventually forces D to represent subtle variations in the real data distribution and pass this information for the learning of G . However, the discriminator D is solely tasked to separate real samples from the generated fake ones, thus has no incentive to do so, especially when the classiﬁcation task for D is too simple. Such is always the case in the early stage of the training and may persist to the later stage if the input space has high dimensionality or if G already collapsed.

Original: See Figure 1 for a pictorial illustration.
Revised: In these situations, D could overﬁt, and its internal layers could have degenerate representation whereby large portions of the input space are modelled as linear regions, as pictorially depicted in Fig. 1, and shown in the synthetic experiment in Sec. 4.1.

Original: In this paper we propose a new regularizer to encourage the discriminator D  to allocate its model capacity in a more desirable way to provides extra guidance for the learning of G. In particular, we would like the discriminator D to allocate its model capacity to tell apart the generated fake samples from the real ones in distinct ways, and not waste the capacity on the regions where there is no data.
Revised: With such degeneracy, learning signals from D are not diverse and fail to capture the differences among different modes or subtle variations of the real data. Furthermore, such degeneracy could also cause the learning of G to bluntly extrapolate, resulting in large updates, which in turn drops already discovered real data modes and/or leads to oscillations.

Original: Typically, the discriminator D uses rectiﬁer as its activation function. As illustrated in Figure 1, the discriminator D essentially cuts the input space into different local linear regions.
Revised: We observe this phenomenon in the synthetic data problem in Sec. 4.1. In this paper, we propose a new regularizer for training GANs where D is a rectiﬁer net. Our regularizer encourages the discriminator D to cut the input space more ﬁnely around where the current

Original: Intuitively, for a ﬁxed region in the input space, how much capacity the discriminator D put into can be approximately measured by the number of local linear pieces in the region(Montufar et al., 2014).
Revised: G distribution support, as well as where training might transport the generated data distribution to in the short future, as depicted in Fig. 1 (right).

Original: The feedback information for the generator G is essentially the slope of the corresponding local linear region deﬁned by D .
Revised: In this way, G receives rich guidance for faster exploration and more stable convergence . The regularizer facilitates exploration because D tells apart the generated fake samples from the real ones in distinct ways.

Original: As the generated data points x lie in different regions,  ∇ x D ( x )  are likely to be diverse in different directions.
Revised: Hence a concentrated mass in the fake data distribution has a better chance of been spread apart because if the fake data points x lie in different regions, learning signals ∇ x D ( x ) to G are likely to point to different directions.

Original: As such the generator Gin turn can receive diverse learning signals from ∇ x D ( x ) .
Revised: On the other hand, the convergence to equilibrium is more stable because there are less large piecewise linear regions, where G learning constantly receives the same transportation direction, potentially leading to overshoot and oscillation. Our regularizer are constructed to encourage the activation patterns of the internal representations of points in a mini-batch to be diverse.

Original: Further note that the different local linear regions deﬁned by D is closely related to the different activation patterns of D (Raghu et al., 2016).
Revised: As shown in Raghu et al. (2016), the different local linear regions deﬁned by D is closely related to the different activation patterns of D .

Original: More speciﬁcally, two generated data points  with different activation patterns in D are guaranteed to lie on different linear regions.
Revised: In particular, two input points into D with different activation patterns on all layers of D are guaranteed to lie on different linear regions.

Original: In this paper we propose a regularizer to encourage the diversity of the activation patterns of D , in order to guide the discriminator D to allocate its model capacity in such way that G can receive more informative signals. The rest of the paper is organized as follows. Related works in the literature are discussed in Section 2.1.
Revised: More details are presented in Sec.

Original: We formally present our regularizer in Section 3. The proposed regularizer will be applied on fake data (blue in Fig. 1) and region inbetween real and fake data (purple in 1), together withauto-encoding or other auxiliary tasks on the real data forms our proposed strategy for improving GAN training.
Revised: 3 where the regularizer is deﬁned, along with the analysis of its properties.

Original: 2. R ELATED WORKS
Revised: 2. R ELATED WORKS

Original: The role of model capacity of the discriminator D in training generative adversarial networks (GANs) has been previously explored by Arora et al.
Revised: Other regularization/gradient penalty techniques have also been proposed to stabilize GANs training (Gulrajani et al., 2017; Nagarajan & Kolter, 2017) recently. Gulrajani et al.

Original: While they show that a discriminator D with ﬁnite number of parameters has a limited capacity in distinguishing real data and the generator G’s samples (and suggest to increase the discriminator D ’s capacity), the question on where and how D can utilize its limited capacity effectively is left unanswered.
Revised: (2017) adds an input gradient penalty to the update of D , so that the magnitude of signals passed to G is controlled. Nagarajan & Kolter (2017) modiﬁes the update of G to avoid going where the magnitude of ∇ x D ( x ) is large. These methods, as well as other similar works that constrain the input gradient norm or the Lipschitz constant of D , all try to stabilize the training dynamics by regularizing the learning signal magnitude.

Original: Our paper can be seen as a work continuing along this line, providing guidance for better usage of the model capacity.
Revised: This is different from our method that diversiﬁes the learning signal directions. As discussed in the previous section, the diversiﬁed signal directions help both the convergence speed and the stability of the training. In Sec. 4.2, we empirically demonstrate that our proposed method achieves better results than Wasserstein GAN with gradient penalty (WGAN-GP) (Gulrajani et al., 2017).

Original: Encouraging D to use its capacity  in a constructive way is non-trivial. Auto-encoding methods appear promising, and have been  explored in Energy Based GAN (Zhao et al., 2016) and Boundary
Revised: The role of model capacity of the discriminator D in training generative adversarial networks (GANs) has been previously explored by Arora et al. They show that D with a ﬁnite number of parameters has limited capacity in distinguishing real data from the generated ones.

Original: Equilibrium GAN (Berthelot et al., 2017).
Revised: They suggest increasing the discriminator D ’s capacity among other modiﬁcations.

Original: In their work, energy is pushed down on real data manifolds while energy is pushed up for generated data. Because their framework involves auto-encoding tasks, they naturally require D  to learn almost all essential features (v.s. plain classiﬁcation tasks).
Revised: Our work can be viewed as a continuation along this direction, except that we treat the model capacity not as a static number, but a varying function in the different parts of the input space during training. Because even with a large number of parameters, D might not use its capacity in a right way to help convergence, as discussed previously.

Original: However, when fake data is closer to real ones, there appears to be less-understood tradeoffs in sample quality and sample diversity where Berthelot et al.
Revised: We explore the question on where and how D can utilize its limited capacity effectively for better training convergence.

Original: empirically observe. Some mechanisms potentially hinder how well D can separate real from generated data while capturing different variations within the real data distribution.
Revised: Encouraging D to use its capacity in a constructive way is non-trivial.

Original: Another potential way to mitigate the problem is to use a Bayesian neural net, whose model capacity away from data is not degenerate.
Revised: One theoretically sound potential approach to regularize D is to use a Bayesian neural net, whose model capacity away from data is not degenerate.

Original: However, computationally scalable deep Bayesian neural networks are still an active area of research ? and are not easy to use.
Revised: However, computationally scalable deep Bayesian neural networks are still an active area of research (Hern ´ andez-Lobato & Adams, 2015; Hasenclever et al., 2017) and are not easy to use.

Original: Alternatively, we can use auxiliary tasks to regularize D ’s capacity usage.
Revised: Alternatively, we can use auxiliary tasks to regularize D ’s capacity usage.

Original: On real data manifold, this is generally easy.

Original: If the data has label, augmenting D with semi-supervised learning task, as shown in ? , improves GAN training stability and the resulting generative model.
Revised: If given labelled data, semi-supervised learning as an auxiliary task for D , as shown in Salimans et al. (2016), improves GAN training stability and the resulting generative model.

Original: If the data domain has other structures, for instance order or context structure in many sequence data, they can be exploited to regularize the model capacity usage as well and should in principle improve  GAN training.
Revised: We hypothesize that if the data domain has other structures that can be exploited as supervised learning signal, Exploiting these structures could as well potentially improve the GAN training stability like in Salimans et al.

Original: Finally, if there is no label, we can use auto-encoding as an auxiliary task on the real data distribution.
Revised: When no supervised task is available, auto-encoding is another potential possibility.

Original: B INARIZED R EPRESENTATION E NTROPY
Revised: Energy-Based GAN (Zhao et al., 2016) and Boundary Equilibrium GAN (Berthelot et al., 2017) use auto-encoders as their discriminators. However, both EBGAN and BEGAN have different objectives from the vanilla GAN.

Original: We now introduce a regularizer that encourages the hidden representation of different data points to be diverse. The regularizer acts on the binarized activation pattern, and has two terms: the ﬁrst term, R ME ,encourages individual hidden units to be on half of the time on average, to have high marginal entropy ; the second term, R AC , encourages pairs of hidden units to have low activation correlation .
Revised: Furthermore, instead of using auto-encoding to simply regularize D , the auto-encoder loss is used to discriminate real data from fake ones. Hence, it is unclear if their beneﬁts stem from the regularization effects or the alternative classiﬁcation approach. Another set of works that use auto-encoding as an auxiliary task is in learning an inference network along with GAN (Donahue et al., 2016; Dumoulin et al., 2016).

Original: After deﬁning the terms mathematically below, we will ﬁrst motivate their forms as a necessary condition for maximizing the representation entropy in Sec. 3.2. Then we will show that minimizing the proposed regularizer leads to a lower bound on the joint entropy of  the binarized representation in Sec. 3.3.
Revised: However, they both modify the input to D , so that D classify not just the data, but together with the corresponding latent codes from G . Again, in this case, it is unclear if the regularization effect on the model capacity of D is the source of any improvement in learning stability. Our preliminary results on using the auxiliary auto-encoding loss on real data show that it does not lead to improvement (see Discussion and Future Work in Sec. 5).

Original:  3. T HE BINARY REPRESENTATION ENTROPY REGULARIZER
Revised:  B INARIZED R EPRESENTATION E NTROPY

Original: We present the formal deﬁnition of the binary representation entropy regularizer in this section.
Revised: We now introduce our regularizer, the binarized representation entropy regularizer (BRE).

Original: Given a mini-batch, { x1 , . . .
Revised: Recall that we would like to encourage diverse activation patterns in D across different samples. For a sample x , its activation pattern on a particular internal layer of D , can be represented by a binary vector, as shown in Figure 2.

Original: , x K } of size K , let hi ∈ R
Revised: In particular, let h ∈ R

Original: d be the immediate pre-nonlinearity activity ofdata point x i in a mini-batch on a particular layer of d hidden units.
Revised: d be the immediate pre-nonlinearity activity of a sample x on a particular layer of d hidden units.

Original: h i is represented  asa column vector. Let si = sign ( hi ) :
Revised: The activation pattern of x on this layer can be represented by the sign vector of h , deﬁned as s = sign ( h ) :

Original: = hi | hi | ∈ {± 1 } dbe the sign function, where | · | is entry-wise absolute value. 
Revised: = h | h | ∈ {± 1 } d where | · | is entry-wise absolute value. is the binary activation vector of on layer L

Revised: We call this binary vector s ∈ {± 1 } d the activation vector of the sample x on this particular layer.

Original: Assume that the sign function s i of each data point x i is an independent sample of ( U 1 , . .
Revised: In this work, we model the activation vector of each sample in a particular layer of D as a random binary vector. Given a mini-batch, { x 1 , . . . , x K } of size K , assume that each binary activation vector s k of x k , k = 1 , . . , K , on a particular layer with d hidden units is an independent sample of a random binary vector U =

Original:  . , U d ) , where( U 1 , . .
Revised: ( U 1 , . . . , U d ) , where U i denotes a

Original: , U d ) denote a multivariate Bernourlli random vector with parameters( p  1 , . . 
Revised: Bernoulli random variable with parameters p i and distribution function P i for i = 1 , . . .

Original: . ,p d ) .
Revised: , d 2 .

Original: Also denotes its joint distribution function  by Pand its k th marginal Bernoulli distribution function by P k .
Revised: Also denotes the joint distribution function of ( U 1 , . . . , U d ) by P .

Original: We would like to construct a regularizer that encourages P to have larger entropy.
Revised: To have diverse activation patterns, we would like to construct a regularizer that encourages P to have a large joint entropy.

Original: Ideally, one could use an empirical estimate of the entropy function as a desired regularizer.
Revised: Ideally, one could use an empirical estimate of the entropy function as a desired regularizer.

Original: However, estimating the entropy of a high-dimensional distributionbased a small number of samples has been well known to be difﬁcult, especially with a small mini-batch size (Darbellay & Vajda, 1999; Miller, 2003; Kybic, 2007; Kybic & Vnuˇcko, 2012; Scott, 2015).
Revised: However, sample-based estimation of the entropy of a high-dimensional distribution has been well known to be difﬁcult, especially with a small mini-batch size (Darbellay & Vajda, 1999; Miller, 2003; Kybic, 2007; Kybic & Vnu ˇ cko, 2012; Scott, 2015).

Original: In this paper, we propose a simple regularizer R BRE which, as shown below, encourages the large entropy of P in a weak manner.
Revised: We instead propose a simple binarized representation entropy (BRE) regularizer, which encourages the large entropy of P in a weak manner.

Original: Our proposed Binary Representation Entropy (BRE) regularizer R BRE applied on a given layer consists oftwo terms deﬁned on { s 1 , . . . , s K } : R BRE
Revised: For a particular layer in D , our BRE regularizer R BRE is computed over a mini-batch of { x 1 , . . . , x K } , and consists of two terms, marginal entropy R ME and activation correlation R AC ,

Original: =
Revised: both acting on the binarized activation vectors of the hidden units 3 :

Original: R  ME + R AC , where
Revised: R BRE = R ME + R AC , where

Original: [Equation]
Revised: [Equation]

Original: and
Revised: and

Original: [Equation]
Revised: [Equation]

Original: Here ¯ s ( k ) is the average of the sign function on the k th hidden unit across the mini-batch, and avg i (cid:54) = j denotes the average taken over all possible pairs of ( i, j ) such that i (cid:54) = j . Thus R ME can be interpreted as an empirical estimate of 1 d (cid:80) dk
Revised: | d .

Original: =1 E [ U k ] 2 , and R AC as an empirical estimate of E (cid:104) | (cid:80) dk
Revised: [Equation]

Original: =1 U k ˜ Uk
Revised: + + - - - + + - + U

Original: | (cid:105) , where ˜ U k is a random sample of P k that is independent to U k . It is straightforward that the maximum entropy of P is achieved when p k = 1 / 2 and ( U 1 , . . . , U d ) are mutually independent.
Revised: i

Original: Intuitively, R ME encourages p k to be close to 1 / 2 , while R AC , as shown in Section 3.3, encourages ( U 1 , . . . , U d ) to be pairwise independent. Although pairwise independence is weaker than mutual independence, we will see in Section 3.3 that pairwise independence guarantees a lower bound for the entropy of the joint distribution P . In practice, due to the degenerate gradient of the sign function, we replace s i in R ME by its smooth approximation
Revised: U d U

Original: a i = softsign (h i ): =h i |
Revised: Here ¯ s ( i ) =

Original: h i | + (cid:15) . We also relax R AC by allowing a soft margin term, as R AC =
Revised: 1 K (cid:80) Kk

Original: avg i (cid:54) = j max (cid:0) 0 , | a (cid:62) i a j | /d − η (cid:1) , where η =3 (cid:112) 1 /d .We defer discussing the choice of (cid:15) to the end of this section. 3. M AXIMUM ENTROPY REPRESENTATION HAS R BRE ≈ 0 .
Revised: =1 s k,i is the average of the i th element (corresponding to the i th hidden unit) of the activation vectors s k across the mini-batch, where s k is the activation vector of x k for k = 1 , . . .

Original: In this section we show that R BRE ≈ 0 is a necessary condition for P to achieve maximum entropy. The binarized representation on a layer attain its maximal entropy, when U k ’s are mutually independent and E [ U k ] = 0 ,for all k ∈ { 1 , ..., K } . Therefore, the average signed activation ¯ s k is approximately zero, and so is R ME .
Revised: , K , as shown in Figure 3.

Original: For R AC , lets consider (cid:80) dk =1 M k , where M k = U k ˜
Revised: Thus R ME can be interpreted as an empirical estimate of 1 d (cid:80) di

Original: U k . Note that given P k being Bernoulli( 0 . 5 ) and U k ’s are mutually independent, M k ’s are mutually independent and have the distribution of Bernoulli( 0 . 5 ) as well.
Revised: =1 E

Original: Therefore by the Central Limit Theorem, the distribution of (cid:80) dk =1 M k converges in distribution to the Gaussian distribution N (0 , d ) . Therefore, given sufﬁciently large d , the distribution of(cid:80) dk =1 M k d =
Revised: [ U i ] 2 , and R AC as an empirical estimate of

Original: [Equation]
Revised: [Equation]

Original: s [Equation] d is approximately N (0 , 1 /d ) .
Revised: , where U , ˜ U are two i.i.d. random vectors with probability function P .

Original: A good choice for the margin threshold is η = c (cid:112) 1/d , where we choose c = 3 to leave 99 .
Revised: As shown in Section 3.1, our regularizer encourages a large joint entropy of this random binary vector. In particular, we show that the ﬁrst term, R ME , encourages individual hidden units to be active half of the time on average, to have high marginal entropy ; the second term, R AC , encourages low activation correlation between each pair of the hidden units. We further show in Section 3.2 that having the regularizer being close to 0 is a necessary condition for ( U 1 , . . .

Original: 7% of i, j pairs unpenalized in the maximum entropy case.
Revised: , U d ) to achieve its maximum entropy.

Original:  3.M INIMIZING THE BRE REGULARIZER TO THE ENTROPY OF MULTIVARIATE B ERNOULLI DISTRIBUTION
Revised: Details in the practical implementation of our regularizer are discussed in Section 3.3. 3. BRE ENCOURAGES HIGH JOINT ENTROPY OF THE ACTIVATION PATTERNS

Original: We further show that minimizing R BRE is equivalent to enforcing the maximum entropy of each P k and the pairwise independence of ( U  1 , . . 
Revised: Note that each summand ¯ s ( i ) in R ME is an empirical estimate of 2 p i − 1 , the mean of the marginal distribution P i . Thus minimizing ¯ s 2( i ) leads to p i = 12 , i.e. U i is zero-mean for i = 1 , . . .

Original: . ,U d) .
Revised: , d .

Original: To see that, ﬁrst note that each summand ¯ s k in R ME is the empirical estimate of 2 p k − 1 , the mean of the marginal distribution P k .
Revised: In other words, R ME is 0 when there are equal number of ± 1 in U . Moreover, for j, k = 1 , . .

Original: Thus minimizing ¯ s 2 k leads to p k = 12 , i.e. maximizing the entropy of P k .
Revised: , K where j (cid:54) =

Original: Moreover, the regularizer R AC is essentially equivalent to minimizing (cid:0) s (cid:62)
Revised: k , minimizing | s (cid:62) j s k | in the second term R AC is essentially equivalent to minimizing (cid:0) s (cid:62)

Original: i s j  (cid:1) 2 .
Revised: j s k (cid:1) 2 .

Original: The following proposition shows that given U k being zero-mean, R AC tends to enforce Cov ( U k , U t )
Revised: Thus, minimizing R AC can be seen as minimizing E (cid:20) (cid:16) U (cid:62) ˜ U (cid:17) 2 (cid:21) where U, ˜

Original: = 0 , i.e. Uk ’s are pairwise independent.
Revised: U are i.i.d. from P . Since R ME is enforcing U i to be zero-mean for i = 1 , . . .

Revised: , d , as shown in Proposition 3.1, minimizing R AC enforces the pairwise independence of the U i ’s.

Revised: Lastly, Assuming the hidden units U i ’s are zero-mean and pairwise independent, by Corollary 3. of Gavinsky & Pudl ´ ak (2015) (which we restate in Appendix C for completeness), we have that the entropy of P satisﬁes

Original:  Proposition 3.1.
Revised: [Equation] Proposition 3.1.

Original: Let U
Revised: Let U

Original: =
Revised: =

Original: ( U 1 , . . .
Revised: ( U 1 , . . .

Original: , U d ) be a zero-mean multivariate Bernoulli vector of P , and
Revised: , U d ) be a zero-mean multivariate Bernoulli vector of P , and

Original: ˜ U
Revised: ˜ U

Original: =
Revised: =

Original: ( ˜ U 1 , . . .
Revised: ( ˜ U 1 , . . .

Original: , ˜ U d ) denotes another random vector of P that is independent to U .
Revised: , ˜ U d ) denotes another random vector of P that is independent to U .

Original: Then
Revised: Then

Original: [Equation]
Revised: [Equation]

Original: We prove this proposition in Appendix A.
Revised: We defer the proof of this proposition to Appendix B.

Original: Lastly, Assuming the hidden units U k ’s are zero-mean and pairwise independent, by Corollary 3. of Gavinsky & Pudl´ak (2015) (which we restate here for completeness), the  entropy of P satisﬁes
Revised: We further show that R BRE ≈ 0 is a necessary condition for P to achieve the maximum entropy. It is straightforward that the maximum entropy of P is achieved if and only if E

Original: [Equation]{+[ U i ] = 
+}
Revised: {+[ U i ] = 
+}

Original: Theorem 3. (Corollary 3. of Gavinsky &
Revised: ( p i = 1 / 2 ) for all i ∈ { 1 , ..., d } , i.e. each hidden unit is activated half of the time, and ( U 1 , . .

Original: Pudl´ak (2015)) .
Revised: , U d ) are mutually independent.

Original: Let
Revised: Therefore, the i th element of the average activation vector

Original: H min ( P ) = − log (max x P ( X = x )) .
Revised: ¯ s ( i ) is approximately zero for i ∈ { 1 , ..., d } , and so is R ME .

Original: Also let ( U 1 , . . , U d ) be pairwise independent random variable of 
Revised: Further, note that R AC is an empirical estimate of E (cid:104) (cid:12)(cid:12) (cid:12) (cid:80) di =1 M i /d (cid:12)(cid:12) (cid:12) (cid:105) where M i =

Original: Bernoulli( 0 .5 ).
Revised: U i ˜ U i for i = 1 , . . .

Original: Then,{+, d . Note that given p i = 1 / 
 and U i ’s are mutually independent, one can show that M i ’s are mutually independent and have the distribution of Bernoulli( 0 . 5 ) as well.+}
Revised: {+, d . Note that given p i = 1 / 
 and U i ’s are mutually independent, one can show that M i ’s are mutually independent and have the distribution of Bernoulli( 0 . 5 ) as well.+}

Original: H ( P ) ≥ H min ( P ) ≥ log( n + 1) .
Revised: Therefore by the Central Limit Theorem, the distribution of (cid:80) di =1 M i converges in distribution to the Gaussian distribution N (0 , d ) . Given sufﬁciently large d , the distribution of (cid:80) di =1 M i d is approximately N (0 , 1 /d ) , and thus R AC is approximately zero 4 .

Original: 3. C ONTROLLING THE SOFTNESS OF SOFT - SIGN APPROXIMATION
Revised: 3. GAN TRAINING WITH BRE REGULARIZER P RACTICAL IMPLEMENTATIONS OF R BRE

Original:  R BRE is deﬁned on ak = softsign ( hk ) :
Revised: In practice, due to the degenerate gradient of the sign function, we replace s in R ME by its smooth approximation a = softsign ( h ) :

Original: =
Revised: =

Original: hk | hk | + (cid:15) , so (cid:15) is a hyperparameter to be chosen.
Revised: h | h | + (cid:15) , where (cid:15) is a hyperparameter to be chosen.

Original: If (cid:15) is too small, the nonlinearity becomes too non-smooth for stochastic gradient descent; if it is too large, it fails to be a good approximation to the sign function for R BRE to be useful.
Revised: If (cid:15) is too small, the nonlinearity becomes too non-smooth for stochastic gradient descent; if it is too large, it fails to be a good approximation to the sign function.

Original: Furthermore, not only different layers could have different scales of hk , hence requiring different (cid:15) , during training the scale of h could change too.
Revised: Furthermore, not only different layers could have different scales of h , hence requiring different (cid:15) , during training the scale of h could change too.

Original: Therefore, instead of setting a ﬁxed (cid:15) , we set (cid:15) = s | hk |  , where  | hk | averages over  the d dimensions of the layer and s is a constant with value 0 .
Revised: Therefore, instead of setting a ﬁxed (cid:15) , we set (cid:15) = c avg ( | h | ) , where c is some small constant and avg ( | h | ) is a scalar, where the average runs over samples in the minibatch and the d dimensions of the layer.

Original: In this way, softsign ( · ) is invariant with respect to any multiplicative scaling of hk in the forward pass of the computation; in the backward pass for the gradient computation, we do not backpropagate through (cid:15) .
Revised: In this way, softsign ( · ) is invariant with respect to any multiplicative scaling of h in the forward pass of the computation; in the backward pass for the gradient computation, we do not backpropagate through (cid:15) .

Original: We choose s = 0 .
Revised: We choose c = 0 .

Original: 001 , as we observe empirically that this usually makes 90% to 99% of units to have absolute value at least .
Revised: 001 , as we observe empirically that this usually makes 90% to 99% of units to have absolute value at least .

Original: Because G receives training signal from D , so it might be tempting to add a regularizer to enforce gradient directions ∇ x D ( x i ) i to be diverse.
Revised: An alternative to this softsign is the tanh nonlinearity. However, tanh lacks the scale invariance of our proposed softsign with varying (cid:15) , hence potentially less effective in capturing the nature of input space partitioning.

Original: However, in rectiﬁer networks, if two inputs share the same activation pattern, the input gradients located at the two points are co-linear, hence any gradient based learning with such diversity regularizer would have difﬁculty pulling them apart. In general, unlike R BRE which operates directly on both activated and non-activated path, input gradient regularizer can only access information on the activated path in the network, so that it can only encourage exisiting non-shared activated path, but cannot directly create new non-shared activated path. In theory, other nonlinearities such as tanh might allow input gradient regularizer to work, but they are very hard to train at the the ﬁrst place.
Revised: In Sec. 4.2 (Table.

Original: In our preliminary experiments, on tanh nonlinearity networks, input gradient diversity regularizer with either cosine similarity or soft-sign based regularizer like R BRE does not work.
Revised: 1), we conﬁrm empirically that using tanh instead of softsign decreases the effectiveness of BRE. We also relax R AC by allowing a soft margin term, as R AC = avg j (cid:54) = k max (cid:0) 0 , | a (cid:62) j a k | /d − η (cid:1) .

Original: 3. GAN TRAINING WITH BRE REGULARIZER W HERE IS R BRE APPLIED ?
Revised: Recall that a (cid:62) j a k /d has an approximate distribution of a good choice for the margin threshold is η = c (cid:112) 1 /d , where we adopt the “ 3 σ rule” and choose c = 3 to leave 99 . 7% of i, j pairs unpenalized in the maximum entropy case.

Original: To regularize GAN training, R BRE is applied to the immediate pre-nonlinearity activities on selected layers of D .
Revised: To regularize GAN training, R BRE is applied to the immediate pre-nonlinearity activities on selected layers of D .

Original: Therefore, if there is any normalization layer before nonlinearity, R BRE needs to be applied after the normalization.
Revised: Therefore, if there is any normalization layer before nonlinearity, R BRE needs to be applied after the normalization.

Original: We emphasize that we use soft-sign for the regularizer only, we do not modify the nonlinearity or any other structure of the neural net.
Revised: We emphasize that we use softsign for the regularizer only, we do not modify the nonlinearity or any other structure of the neural net.

Original: W HICH LAYERS SHOULD R BRE BE APPLIED ON ?
Revised: W HICH LAYERS SHOULD R BRE BE APPLIED ON ?

Original: Technically R BRE can be applied on any rectiﬁer layer before the nonlinearity.
Revised: Technically R BRE can be applied on any rectiﬁer layer before the nonlinearity.

Original: However, having it on the last hidden rectiﬁer layer before classiﬁcation might hinder D ’s ability to separate real from fake, as the high entropy representation encouraged by R BRE might not be compatible with linear classiﬁcation.
Revised: However, having it on the last hidden rectiﬁer layer before classiﬁcation might hinder D ’s ability to separate real from fake, as the high entropy representation encouraged by R BRE might not be compatible with linear classiﬁcation.

Original: Therefore, for unsupervised learning, we apply R BRE on all except the last rectiﬁer nonlinearity before the ﬁnal classiﬁcation; for semi-supervised tasks using the K + 1 -way classiﬁcation setup from Salimans et al.
Revised: Therefore, for unsupervised learning, we apply R BRE on all except the last rectiﬁer nonlinearity before the ﬁnal classiﬁcation; for semi-supervised tasks using the augmented class setup from Salimans et al.

Original: , we apply R BRE only on 2 nd, 4 th and 6 th convolutional layer, leaving 3 nonlinear layersun-regularized before the ﬁnal softmax 
Revised: , we apply R BRE only on 2 nd, 4 th and 6 th convolutional layer, and leave the three nonlinear layers before the ﬁnal softmax untouched.

Original: . W HICH PART OF THE DATA SHOULD R BRE BE APPLIED ON ?
Revised: W HICH PART OF THE DATA SHOULD R BRE BE APPLIED ON ?

Original: Recall from Sec.
Revised: Recall from Sec.

Original: 2 that we want D to spend enough capacity on both the real data manifold, and the current generated data manifold by G , as well as having adequate capacity on regions where we do not currently observe real or fake points but might in future iterations.
Revised: 2 that we want D to spend enough capacity on both the real data manifold, and the current generated data manifold by G , as well as having adequate capacity in region where we do not currently observe real or fake points but might in future iterations.

Original: To enforce this, we apply R BRE onselected layers on generated data minibatch, as well as random interpolation inbetween real and generated data.
Revised: To enforce this, we apply R BRE on generated data minibatch, as well as random interpolation inbetween real and generated data.

Original: Speciﬁcally, let x i and ˜x i be a real and a fake data points respectively, we sample α i ∼ U (0 , 1) and let ˆx i
Revised: Speciﬁcally, let x k and ˜x k be a real and a fake data points respectively, we sample α k ∼ U (0 , 1) and let ˆx k

Original: = α 
Revised: = α k x k

Original: i x i + (1 − α i )
Revised: + (1 − α k )

Original: ˜x i , and apply R BRE on selected layer representation computed on interpolated data points { ˆx i } as well.
Revised: ˜x k , and apply R BRE on selected layer representation computed on interpolated data points { ˆx k | k = 1 . . . , K } as well.

Original:  E XPERIMENTS4. S YNTHETIC DATASET : M IXTURE OF G AUSSIANS
Revised:  E XPERIMENTS

Original: c on t r o l it e r : c o n t r
Revised: Using a 2D synthetic dataset and CIFAR10 dataset (Krizhevsky, 2009), we show that our BRE improves unsupervised GAN training in two ways: (a) when GAN training is unstable (for e.g. due to architectures that are less well tuned than DCGAN (Radford et al., 2015)), BRE stabilizes the training and achieves much-improved results, often surpassing tuned conﬁgurations.

Original: o
Revised: (b) with architecture and hyperparameters settings that are engineered to be stable already, BRE makes GAN learning converges faster. We then demonstrate that BRE regularization improves semi-supervised classiﬁcation accuracy on CIFAR10 and SVHN dataset (Netzer et al., 2011).

Original: l it e r : c on t r o l it e r : t r ea t it e r : t r ea t it e r : t r ea t it e r : [Figure]
Revised: Additional results on imbalanced 2D mixture as well as CelebA dataset are presented in the Appendix D. 4. S YNTHETIC DATASET : M IXTURE OF G AUSSIANS

Original: We ﬁrst demonstrate BRE regularizer’s effect on ﬁtting  2D mixture of Gaussian.
Revised: We ﬁrst demonstrate BRE regularizer’s effect on ﬁtting a 2D mixture of Gaussian.

Original: The top half of Figure 2 contains results from a control experiment (no BRE regularizer). The bottom half of the ﬁgure contains results regularized by BRE.
Revised: In Figure 4, the top and bottom three rows correspond respectively to experiments without the BRE regularizer (control) and with the regularizer (treat).

Original: Within each setting, each row represents one iteration. We show the beginning, middle, and the end of the training process.
Revised: Within each setting, each row represents one iteration during GAN training, selected to be at the beginning, middle, and the end of the training process.

Original: The ﬁrst column shows real data points (blue) and generated data points (red).
Revised: The ﬁrst column shows real data points (blue) and generated data points (red).

Original: The second to ﬁfth columns show hidden layersto 4 of D . Pixels with the same color has the same binary activation pattern on that particular layer. 
Revised: The second to ﬁfth columns show hidden layers 1 to 4 of D , where contiguous pixels with the same colour have the same binary activation pattern on that particular layer. The last column shows the probability of real data according to D .

Revised: The BRE regularizer is added on h 2 and h 3 .

Revised: More results in Appendix D. c on t r o l it e r :c on t r o l it e r : c on t r o l it e r : t r ea t it e r :t r ea t it e r : t r ea t it e r :

Original: We can see that by adding BRE,in h2, h3, and h4, the input domain is divided into more pieces.
Revised: MoG samples h h h h D By adding BRE, the input domain is partitioned more ﬁnely as reﬂected by visualization for h 2 , h 3 , and h 4 .

Revised: The richer D representation allows more effective exploration of different input regions because the gradient signals provided by D to G are more diverse than the degenerate baseline case where D is linear in large regions of the input.

Revised: Once a real data mode is discovered, G locks onto it without oscillation.

Original: This indicates the better usage of D’s model capacity, and resulted in better exploration of the space and  more stable training.
Revised: This shows that with better D capacity usage, the GAN optimisation converges faster and is more stable, while the resulting equilibrium suffers much less from mode dropping.

Original: 4. C ELEB A
Revised: 4. F ASTER AND BETTER CONVERGENCE IN UNSUPERVISED LEARNING

Original: We compare stable and unstable runs of DCGAN on CelebA dataset (Liu et al., 2015), as well as effect of BRE regularizer.
Revised: We quantitatively measure the resulting G using the Inception score (Salimans et al., 2016). Table 1 shows improved ﬁnal Inception scores on DCGAN, as well as the following non-standard architectures (only mentioning difference from the standard DCGAN): densely connected convnet

Original: Fig.
Revised: Huang et al.

Original: 3(a) shows thresholded R AC term through training. The model being investigated is a 4-layer DCGAN for both G and D, with batch normalization.
Revised: (2016) for D ; G and D with an equal number of ﬁlters on each layer; D with ReLU nonlinearity.

Original: The unstable run (Fig. 3(b)) uses a large initial learning rate of . 01 and 3 D update steps for each G update, whereas the stable run (Fig. 3(d)) uses initial lr = 2 e − 3 and 1 D update for each G update. Even without the BRE regularizer, we can see that when GAN training is stable, D uses more capacity around fake and real data as well as inbetween, as measured byR AC values in Fig. 3(a). When BRE regularizer is applied, the usage of D ’s capacity is more improved, and resulted in more diversity in the learned distribution by G (Fig. 3(c)). 4. Q UANTITATIVE RESULTS ON CIFAR10: IMPROVED I NCEPTION SCORE IN UNSUPERVISED SETTING AND HIGHER SEMI - SUPERVISED CLASSIFICATION ACCURACY
Revised: In all cases, the models regularized by BRE improve over the baseline counter-parts without regularization (no-BRE).

Original: First, we demonstrate quantitatively that BRE regularization improves learned distribution as measured by Inception score (Salimans et al., 2016) on CIFAR10 dataset (Krizhevsky, 2009).
Revised: Furthermore, vanilla GAN’s with BRE applied on multiple D layers (BRE multi) always outperform WGAN-GP (Gulrajani et al., 2017).

Original: Fig.shows inception score and (thresholded) activation correlation values ( R AC ) during training iteration, for two different experiments with a normal optimization setting ( 4 left column) and a more aggressive optimization setting ( 4 right column).
Revised: Fig. 6 shows some generated samples from a DCGAN-ReLU model without and with BRE regularization. Fig. 9 of

Original: In both cases, BRE regularization keeps activation correlation low, leading to higher representation entropy by D , and the resulting in higher inception scores.
Revised: Appendix D.1 show more samples laid out by t-SNE visualization to better illustrate mode collapsing. [Table] Fig. 5 shows that with BRE, DCGAN training converges faster, as measured by Inception score.

Original: In Table.
Revised: The 1 σ error bars are estimated from ten different random runs. Because DCGAN architecture is engineered to be stable, in the end, baseline DCGAN can still achieve comparable Inception score with the regularized version on average. But clearly the convergence is much faster with BRE during the initial transient phase, conﬁrming our intuition that BRE improves exploration. Fig.shows Inception score and (thresholded) activation correlation values ( R AC ) during one particular set of runs with default DCGAN optimization settings, and a more aggressive optimization setting.

Original: 1 in Appendix, we show more quantitative results as vary the size of D , and demonstrate that BRE regularization consistently improves inception score.
Revised: In both cases, BRE regularization results in similarly faster convergence, and higher ﬁnal Inception score in the unstable case with more aggressive optimization. The bottom row in Fig. 7 shows that BRE regularization is indeed making a qualitative difference to the activation correlation ( R AC ) by keeping it low during training in both cases.

Original: We also show that the proposed BRE regularizer is not only compatible with semi-supervised learning using GANs, but also improves classiﬁcation accuracy.
Revised: BRE makes convergence faster. 4. I MPROVED S EMI - SUPERVISED L EARNING ON CIFAR10 AND SVHN BRE regularization is not only compatible with semi-supervised learning using GAN’s, but also improves classiﬁcation accuracy.

Original: Fig.
Revised: Table.

Revised: 2 shows results on CIFAR10 with feature matching semi-supervised learning GAN.

Original: 5 shows that BRE allows  learning  to discover a better solution during training, as shown by both training losses as well as  test classiﬁcation error rate.
Revised: BRE allows the learning process to discover a better solution during training that also generalizes better, indicated by a lower training classiﬁcation loss as well as lower test classiﬁcation error rates.

Original: We  usedexactly the same code and hyperparameter from Salimans et al.
Revised: We [Figure] used the same code and hyperparameters 5 from Salimans et al.

Original: (2016), and add R BRE regularization on every other second layer, starting from the 2nd until 4 layers before the classiﬁcation layer, with a regularizer weight of . 01 and not decayed.
Revised: Details on BRE hyperparameters are in Appendix A, learning curve plots in Appendix D. [Table] On Street View House Numbers (SVHN) dataset (Netzer et al., 2011) [Table]

Revised: with the same setup from

Revised: Salimans et al

Revised: [Table]

Revised: [Table]

Revised: learning is not always stable when trained for a long time.

Revised: Fig.

Revised: 8 (top row) shows that without BRE regularization

Revised: [Table]

Revised: when trained for a very long time, sometimes learning diverges.

Revised: Such failure is dramatically reduced by BRE (bottom row of Fig.

Revised: [Table]

Revised:  D ISCUSSION AND FUTURE WORK

Revised: There are still many unexplored avenues along this line of research.

Revised: For example, how can our new regularizer collaborate with other GANs training techniques to further improve the training GANs?

Revised: We leave such further explorations for future works.

Revised: Meanwhile, there are two interesting questions related to the central theme of this work.

Revised: D OES DIRECTLY REGULARIZING THE DIVERSITY OF ∇ x D ( x ) WORK ? I F NOT , WHY NOT ?

Revised: To diversify G ’s learning signals, it might be tempting to enforce gradient directions ∇ x D ( x k ) to be diverse.

Revised: However, in rectiﬁer networks, if two inputs share the same activation pattern, the input gradients located at the two points are co-linear; hence any gradient-based learning with such diversity regularizer would have difﬁculty pulling them apart.

Revised: In general, unlike BRE which operates directly on both activated and non-activated portions of D ’s internal units, an input gradient regularizer can only access information on the activated path in the network, so that it can only encourage existing non-shared activated path, but cannot directly create any new non-shared activated path.

Revised: In theory, tanh nonlinearities as activations for D could avoid this problem, but such network is hard to train in the ﬁrst place.

Revised: In our preliminary studies, on networks with tanh, input gradient diversity regularizer with either cosine similarity or a soft-sign based regularizer like BRE does not work.

Revised: C OULD AUXILIARY TASKS HELP REGULARIZE D ?

Revised: As discussed in Sec. 2.1, auxilary tasks could potentially regularize D and stabilize training.

Revised: One possible auxilary task is reconstruction loss.

Revised: We performed some preliminary experiments, and found that reconstructing real data as auxilary tasks worsens the resulting learned G .

Revised: See Appendix.

Revised: D.3 for results.

Revised: Further study is needed, and is beyond the scope of this work.

Original:  C ONCLUSIONS
Revised:  C ONCLUSIONS

Original: We proposed a novel regularizer in this paper to guide the discriminator in GANs to better allocate its model capacity.
Revised: We proposed a novel regularizer in this paper to guide the discriminator in GANs to better allocate its model capacity.

Original: Based on the relation between the model capacity and the activation pattern of the network, we construct our regularizer to encourage a high joint entropy of the activation pattern  of the discriminator D in a weak manner.
Revised: Based on the relation between the model capacity and the activation pattern of the network, we constructed our regularizer to encourage a high joint entropy of the activation pattern on the hidden layers of the discriminator D .

Original: Experimental results demonstrate the beneﬁts of our new regularizer.
Revised: Experimental results demonstrated the beneﬁts of our new regularizer: faster progress in the initial phase of learning thanks to improved exploration, more stable convergence, and better ﬁnal results in both unsupervised and semi-supervised learning A M ODEL AND HYPERPARAMETER DETAILS A. 2D EXAMPLE G is 4-layer (excluding noise layer) MLP with ReLU hidden activation function, and tanh visible activation; D is 5-layer MLP with LeakyRelu( . 2 ).

Original: There are still many unexplored avenues alone this line of research.
Revised: Both D and G has 10 units on each hidden layer, no batch or other normalization is used in D or G , no other stabilization techniques are used; For Fig. 4, Fig.

Original: For example, how can our new regularizer collaborate with other GANs training techniques to further improve the training GANs?
Revised: 12, and Fig. 13, lr=.001 with adam(.0, .999), and BRE regularizer weight 1 . , applied on h2 and h3; both lr and BRE weight linearly decay to over iterations to 1 e − 6 and 0 respectively. For Fig.

Revised: 12 and Fig. 13, lr=.002 with adam(.5, .999), and BRE regularizer weight 1 . , applied on h2.

Revised: A. U NSUPERVISED L EARNING CIFAR

Original: We leave such further explorations for future works.
Revised: Table. “single” all “multi” For Fig.

Revised: 7, the default optimization setting (left column, i.e. (a) and (c)) is lr = 2e − 4 and one D update per G update, lr for both D and G annealled to 1 e − 6 over 90 K G updates; while the aggressive setting (right column, i.e. (b) and (d)) is lr = 2e − 3 and three D update for every G update, lr for both D and G annealed to 1 e − 6 over 10 K G updates.

Revised: A. S EMI -S UPERVISED L EARNING

Revised: We used exactly the same code and GAN hyperparameters 6 from Salimans et al.

Revised: R BRE regularization is applied on every other second layer, starting from the 2nd until 4 layers before the classiﬁcation layer (applied on three layers in total).

Revised: On CIFAR10, we used a regularizer weight of . 01 , and on SVHN we used 0 . 1 .

Revised: BRE is applied on real, fake and interp data.

Original:  ROOF OF P ROPOSITION 3. 
Revised: B P ROOF OF P ROPOSITION 3.

Original: Proof.
Revised: Proof.

Original: Let M k
Revised: Let M i

Original: = U
Revised: = U

Original: k ˜
Revised: i ˜

Original: U k .
Revised: U i .

Original: Then
Revised: Then

Original: [Equation]
Revised: [Equation]

Original: where Equation (1) is due  the independence of U and ˜ U , Equation (2) is due to that U 2 k = 1 with probability 1 , and Equation (3) is because E [ U k ] = 0 . 
Revised: where Equation (1) is due to the independence of U and ˜ U , Equation (2) is due to that U 2 i = 1 with probability 1 , and Equation (3) is because E [ U i ] = 0 . C C OROLLARY 3.3 OF G AVINSKY & P UDL ´ AK (2015)

Original: B A DDITIONAL E XPERIMENTAL R ESULTS
Revised: Theorem C. (Corollary 3. of Gavinsky &

Revised: Pudl´ak (2015)) .

Revised: Let

Revised: H min ( P ) = − log (max x P ( X = x )) .

Revised: Also let ( U 1 , . .

Revised: , U d ) be pairwise independent random variable of

Revised: Bernoulli( 0 . 5 ).

Revised: Then,

Revised: H ( P ) ≥ H min ( P ) ≥ log( d + 1) .

Revised: D A DDITIONAL E XPERIMENTAL R ESULTS D. M ORE SAMPLES FROM DCGAN-R E LU

Revised: We show more samples generated from the DCGAN-ReLU model, mentioned in Sec.

Revised: 4.2, without the BRE regularizer (with red frames) and with the BRE regularizer (with blue frames) in Fig.

Revised: In Fig.

Revised: 9 images are arranged based on L 2 distances in the original pixel value space.

Revised: Images that are similar in pixel values are roughly grouped together.

Revised: To achieve this, we apply t-SNE (Maaten & Hinton, 2008) to reduce the dimensionality of these images into 2D points.

Revised: These 2D coordinates are then transformed to 2D grids RasterFairy 7 .

Revised: At the same time, the neighborhood relations of the rastered 2D points are preserved to a certain degree.

Revised: We then use these rastered 2D points to arrange the location of these images.

Revised: D. S EMI - SUPERVISED LEARNING CURVES

Revised: Fig.

Revised: 10 shows the Learning curves for semi-supervised learning on CIFAR

Revised: . D. R ECONSTRUCTION AS AUXILIARY TASK TO REGULARIZE D WORSENS RESULTS

Revised: D Recon, no BRE ln, λ recon =6 . 1958 ± 0 .ln, λ recon =6 .

Revised: 2218 ± 0 .ln, λ recon = .6 .

Revised: 2437 ± 0 .ln, λ recon =6 .

Revised: 4025 ± 0 .bn, λ recon =6 .

Revised: 5356 ± 0 .bn, λ recon = .6 .

Revised: [Table]

Revised: D. C ELEB A

Revised: We compare stable and unstable runs of DCGAN on CelebA dataset (Liu et al., 2015), as well as the effect of the BRE regularizer.

Revised: Fig.

Revised: 11(a) shows thresholded R AC term (deﬁned in Sec. 3.3) through training.

Revised: The model being investigated is a 4-layer DCGAN for both G and D , with batch normalization.

Revised: The unstable run (Fig. 11(b)) uses a large initial learning rate of . 01 and 3 D update steps for each G update, whereas the stable run (Fig. 11(d)) uses initial lr = 2 e − 3 and 1 D update for each G update.

Revised: Even without the BRE regularizer, we can see that when GAN training is stable,

Revised: D uses more capacity around fake and real data as well as inbetween, as measured by R AC values in Fig.

Revised: 11(a).

Revised: When BRE regularizer is applied, the usage of D ’s capacity is more improved, and resulted in more diversity in the learned distribution by G (Fig.

Original:  We show more 2D mixture of Gaussian results in Figure 6 and 7.
Revised: 11(c)). D. A DDITIONAL 2D M O G R ESULTS We show more 2D mixture of Gaussian results in Fig. 12, 13, 14, and 15.

Original: c on t r o l it e r : c on t r o l it e r : c on t r o l it e r : c on t r o l it e r : c on t r o l it e r : c on t r o l it e r : c on t r o l it e r : c on t r o l it e r : c on t r o l it e r : c on t r o l it e r : c on t r o l it e r :
Revised: c on t r o l it e r :c on t r o l it e r : c on t r o l it e r : c on t r o l it e r : c on t r o l it e r : c on t r o l it e r : c on t r o l it e r : c on t r o l it e r : c on t r o l it e r : c on t r o l it e r : c on t r o l it e r :

Original: MoG samples h 
Revised: MoG samples h h h h

Revised: D t r ea t it e r :t r ea t it e r : t r ea t it e r : t r ea t it e r : t r ea t it e r : t r ea t it e r : t r ea t it e r : t r ea t it e r : t r e a t it e r : t r ea t it e r : t r ea t it e r :

Revised: MoG samples h

Original: h h h
Revised: h h h

Original: D  t r ea t it e r :  t r ea t it e r :  t r ea t it e r :  t r ea t it e r :  t r ea t it e r :  t r ea t it e r :  t r ea t it e r :  t r ea t it e r :  t r e a t it e r :  t r ea t it e r :  t r ea t it e r :
Revised: D c on t r o l it e r : c on t r o l it e r : c on t r o l it e r : c on t r o l it e r : c on t r o l it e r : c on t r o l it e r : c on t r o l it e r : c on t r o l it e r : c on t r o l it e r : c on t r o l it e r : c on t r o l it e r :

Original: MoG samples h 
Revised: MoG samples h h h h

Revised: D t r ea t it e r : t r ea t it e r : t r ea t it e r : t r ea t it e r : t r ea t it e r : t r ea t it e r : t r ea t it e r : t r ea t it e r : t r e a t it e r : t r ea t it e r : t r ea t it e r :

Revised: MoG samples h

Original: h h h
Revised: h h h

Original: D 
Revised: D c on t r o l it e r :c on t r o l it e r : c on t r o l it e r : c on t r o l it e r : c on t r o l it e r : c on t r o l it e r : c on t r o l it e r : c on t r o l it e r : c on t r o l it e r : c on t r o l it e r : c on t r o l it e r : MoG samples h h h h D t r ea t it e r :t r ea t it e r : t r ea t it e r : t r ea t it e r : t r ea t it e r : t r ea t it e r : t r ea t it e r : t r ea t it e r : t r e a t it e r : t r ea t it e r : t r ea t it e r :

Revised: MoG samples h

Revised: h h h

Revised: D