Paper:
CALIBRATING ENERGY-BASED GENERATIVE ADVER-
In this paper we propose equipping Generative Adversarial Networks with the ability to produce direct energy estimates for samples. Specifically, we develop a flexible adversarial training framework, and prove this framework not only ensures the generator converges to the true data distribution, but also enables the discriminator to retain the density information at the global optimum. We derive the analytic form of the induced solution, and analyze its properties. In order to make the proposed framework trainable in practice, we introduce two effective approximation techniques. Empirically, the experiment results closely match our theoretical analysis, verifying that the discriminator is able to recover the energy of data distribution.
1 INTRODUCTION
Generative Adversarial Networks (GANs) (Goodfellow et al., 2014) represent an important milestone on the path towards more effective generative models. GANs cast generative model training as a minimax game between a generative network (generator), which maps a random vector into the data space, and a discriminative network (discriminator), whose objective is to distinguish generated samples from real samples. Multiple researchers Radford et al. (2015); Salimans et al. (2016); Zhao et al. (2016) have shown that the adversarial interaction with the discriminator can result in a generator that produces compelling samples. The empirical successes of the GAN framework were also supported by the theoretical analysis of Goodfellow et al., who showed that, under certain conditions, the distribution produced by the generator converges to the true data distribution, while the discriminator converges to a degenerate uniform solution.
While GANs have excelled as compelling sample generators, their use as general purpose probabilistic generative models has been limited by the difficulty in using them to provide density estimates or even unnormalized energy values for sample evaluation.
It is tempting to consider the GAN discriminator as a candidate for providing this sort of scoring function. Conceptually, it is a trainable sample evaluation mechanism that – owing to GAN training paradigm – could be closely calibrated to the distribution modeled by the generator. If the discriminator could retain fine-grained information of the relative quality of samples, measured for instance by probability density or unnormalized energy, it could be used as an evaluation metric. Such data-driven evaluators would be highly desirable for problems where it is difficult to define evaluation criteria that correlate well with human judgment. Indeed, the real-valued discriminator of the recently introduced energy-based GANs Zhao et al. (2016) might seem like an ideal candidate energy function. Unfortunately, as we will show, the degenerate fate of the GAN discriminator at the optimum equally afflicts the energy-based GAN of Zhao et al..
In this paper we consider the questions: (i) does there exists an adversarial framework that induces a non-degenerate discriminator, and (ii) if so, what form will the resulting discriminator take? We introduce a novel adversarial learning formulation, which leads to a non-degenerate discriminator while ensuring the generator distribution matches the data distribution at the global optimum. We derive a general analytic form of the optimal discriminator, and discuss its properties and their
∗Part of this work was completed while author was at Maluuba Research
relationship to the specific form of the training objective. We also discuss the connection between the proposed formulation and existing alternatives such as the approach of Kim & Bengio (2016). Finally, for a specific instantiation of the general formulation, we investigate two approximation techniques to optimize the training objective, and verify our results empirically.
2 RELATED WORK
Following a similar motivation, the field of Inverse Reinforcement Learning (IRL) (Ng & Russell, 2000) has been exploring ways to recover the “intrinsic” reward function (analogous to the discriminator) from observed expert trajectories (real samples). Taking this idea one step further, apprenticeship learning or imitation learning (Abbeel & Ng, 2004; Ziebart et al., 2008) aims at learning a policy (analogous to the generator) using the reward signals recovered by IRL. Notably, Ho & Ermon draw a connection between imitation learning and GAN by showing that the GAN formulation can be derived by imposing a specific regularization on the reward function. Also, under a special case of their formulation, Ho & Ermon provide a duality-based interpretation of the problem, which inspires our theoretical analysis. However, as the focus of (Ho & Ermon, 2016) is only on the policy, the authors explicitly propose to bypass the intermediate IRL step, and thus provide no analysis of the learned reward function.
The GAN models most closely related to our proposed framework are energy-based GAN models of Zhao et al. (2016) and Kim & Bengio (2016). In the next section, We show how one can derive both of these approaches from different assumptions regarding regularization of the generative model.
3 ALTERNATIVE FORMULATION OF ADVERSARIAL TRAINING
3.1 BACKGROUND
Before presenting the proposed formulation, we first state some basic assumptions required by the analysis, and introduce notations used throughout the paper.
Following the original work on GANs (Goodfellow et al., 2014), our analysis focuses on the nonparametric case, where all models are assumed to have infinite capacities. While many of the nonparametric intuitions can directly transfer to the parametric case, we will point out cases where this transfer fails. We assume a finite data space throughout the analysis, to avoid technical machinery out of the scope of this paper. Our results, however, can be extended to continuous data spaces, and our experiments are indeed performed on continuous data.
Let X be the data space under consideration, and P = {p | p(x) ≥ 0,∀x ∈ X , ∑ x∈X p(x) = 1} be the set of all proper distributions defined on X . Then, pdata ∈ P : X 7→ R and pgen ∈ P : X 7→ R will denote the true data distribution and the generator distribution. Ex∼p f(x) denotes the expectation of the quantity f(x) w.r.t. x drawn from p. Finally, the term “discriminator” will refer to any structure that provides training signals to the generator based on some measure of difference between the generator distribution and the real data distribution, which which includes but is not limited to f -divergence.
3.2 PROPOSED FORMULATION
In order to understand the motivation of the proposed approach, it is helpful to analyze the optimization dynamics near convergence in GANs first.
When the generator distribution matches the data distribution, the training signal (gradient) w.r.t. the discriminator vanishes. At this point, assume the discriminator still retains density information, and views some samples as more real and others as less. This discriminator will produce a training signal (gradient) w.r.t. the generator, pushing the generator to generate samples that appear more real to the discriminator. Critically, this training signal is the sole driver of the generator’s training. Hence, the generator distribution will diverge from the data distribution. In other words, as long as the discriminator retains relative density information, the generator distribution cannot stably match the data distribution. Thus, in order to keep the generator stationary as the data distribution, the discriminator must assign flat (exactly the same) density to all samples at the optimal.
From the analysis above, the fundamental difficulty is that the generator only receives a single training signal (gradient) from the discriminator, which it has to follow. To keep the generator stationary, this single training signal (gradient) must vanish, which requires a degenerate discriminator. In this work, we propose to tackle this single training signal constraint directly. Specifically, we introduce a novel adversarial learning formulation which incorporates an additional training signal to the generator, such that this additional signal can
• balance (cancel out) the discriminator signal at the optimum, so that the generator can stay stationary even if the discriminator assigns non-flat density to samples
• cooperate with the discriminator signal to make sure the generator converges to the data distribution, and the discriminator retains the correct relative density information
The proposed formulation can be written as the following minimax training objective,
max c min pgen∈P E x∼pgen
[ c(x) ] − E x∼pdata [ c(x) ] +K(pgen), (1)
where c(x) : X 7→ R is the discriminator that assigns each data point an unbounded scalar cost, and K(pgen) : P 7→ R is some (functionally) differentiable, convex function of pgen. Compared to the original GAN, despite the similar minimax surface form, the proposed fomulation has two crucial distinctions.
Firstly, while the GAN discriminator tries to distinguish “fake” samples from real ones using binary classification, the proposed discriminator achieves that by assigning lower cost to real samples and higher cost to “fake” one. This distinction can be seen from the first two terms of Eqn. (1), where the discriminator c(x) is trained to widen the expected cost gap between “fake” and real samples, while the generator is adversarially trained to minimize it. In addition to the different adversarial mechanism, a calibrating term K(pgen) is introduced to provide a countervailing source of training signal for pgen as we motivated above. For now, the form of K(pgen) has not been specified. But as we will see later, its choice will directly decide the form of the optimal discriminator c∗(x).
With the specific optimization objective, we next provide theoretical characterization of both the generator and the discriminator at the global optimum.
Define L(pgen, c) = Ex∼pgen [ c(x) ] −Ex∼pdata [ c(x) ] +K(pgen), then L(pgen, c) is the Lagrange dual function of the following optimization problem
min pgen∈P K(pgen)
s.t. pgen(x)− pdata(x) = 0,∀x ∈ X (2)
where c(x),∀x appears in L(pgen, c) as the dual variables introduced for the equality constraints. This duality relationship has been observed previously in (Ho & Ermon, 2016, equation (7)) under the adversarial imitation learning setting. However, in their case, the focus was fully on the generator side (induced policy), and no analysis was provided for the discriminator (reward function).
In order to characterize c∗, we first expand the set constraint on pgen into explicit equality and inequality constraints:
min pgen K(pgen)
s.t. pgen(x)− pdata(x) = 0,∀x − pgen(x) ≤ 0,∀x∑ x∈X pgen(x)− 1 = 0.
(3)
Notice thatK(pgen) is a convex function of pgen(x) by definition, and both the equality and inequality constraints are affine functions of pgen(x). Thus, problem (2) is a convex optimization problem. What’s more, since (i) domK is open, and (ii) there exists a feasible solution pgen = pdata to (3), by the refined Slater’s condition (Boyd & Vandenberghe, 2004, page 226), we can further verify that strong duality holds for (3). With strong duality, a typical approach to characterizing the optimal solution is to apply the Karush-Kuhn-Tucker (KKT) conditions, which gives rise to this theorem:
Proposition 3.1. By the KKT conditions of the convex problem (3), at the global optimum, the optimal generator distribution p∗gen matches the true data distribution pdata, and the optimal discriminator c∗(x) has the following form:
c∗(x) = − ∂K(pgen)
∂pgen(x) ∣∣∣∣ pgen=pdata − λ∗ + µ∗(x),∀x ∈ X ,
where µ∗(x) = { 0, pdata(x) > 0
ux, pdata(x) = 0 ,
λ∗ ∈ R, is an under-determined real number independent of x, ux ∈ R+, is an under-determined non-negative real number.
(4)
The detailed proof of proposition 3.1 is provided in appendix A.1. From (4), we can see the exact form of the optimal discriminator depends on the term K(pgen), or more specifically its gradient. But, before we instantiateK(pgen) with specific choices and show the corresponding forms of c∗(x), we first discuss some general properties of c∗(x) that do not depend on the choice of K.
Weak Support Discriminator. As part of the optimal discriminator function, the term µ∗(x) plays the role of support discriminator. That is, it tries to distinguish the support of the data distribution, i.e. SUPP(pdata) = {x ∈ X | pdata(x) > 0}, from its complement set with zeroprobability, i.e. SUPP(pdata){ = {x ∈ X | pdata(x) = 0}. Specifically, for any x ∈ SUPP(pdata) and x′ ∈ SUPP(pdata){, it is guaranteed that µ∗(x) ≤ µ∗(x′). However, because µ∗(·) is underdetermined, there is nothing preventing the inequality from degenerating into an equality. Therefore, we name it the weak support discriminator. But, in all cases, µ∗(·) assigns zero cost to all data points within the support. As a result, it does not possess any fine-grained density information inside of the data support. It is worth pointing out that, in the parametric case, because of the smoothness and the generalization properties of the parametric model, the learned discriminator may generalize beyond the data support.
Global Bias. In (4), the term λ∗ is a scalar value shared for all x. As a result, it does not affect the relative cost among data points, and only serves as a global bias for the discriminator function.
Having discussed general properties, we now consider some specific cases of the convex function K, and analyze the resulting optimal discriminator c∗(x) in detail.
1. First, let us consider the case where K is the negative entropy of the generator distribution, i.e. K(pgen) = −H(pgen). Taking the derivative of the negative entropy w.r.t. pgen(x), we have
c∗ent(x) = − log pdata(x)− 1− λ∗ + µ∗(x),∀x ∈ X , (5) where µ∗(x) and λ∗ have the same definitions as in (4). Up to a constant, this form of c∗ent(x) is exactly the energy function of the data distribution pdata(x). This elegant result has deep connections to several existing formulations, which include max-entropy imitation learning (Ziebart et al., 2008) and the directed-generator-trained energybased model (Kim & Bengio, 2016). The core difference is that these previous formulations are originally derived from maximum-likelihood estimation, and thus the minimax optimization is only implicit. In contrast, with an explicit minimax formulation we can develop a better understanding of the induced solution. For example, the global bias λ∗ suggests that there exists more than one stable equilibrium the optimal discriminator can actually reach. Further, µ∗(x) can be understood as a support discriminator that poses extra cost on generator samples which fall in zero-probability regions of data space.
2. When K(pgen) = 12 ∑ x∈X pgen(x) 2 = 12‖pgen‖ 2 2, which can be understood as posing `2 regular-
ization on pgen, we have ∂K(pgen) ∂pgen(x) ∣∣ pgen=pdata = pdata(x), and it follows
c∗`2(x) = −pdata(x)− λ ∗ + µ∗(x),∀x ∈ X , (6)
with µ∗(x), λ∗ similarly defined as in (4). Surprisingly, the result suggests that the optimal discriminator c∗`2(x) directly recovers the negative probability −pdata(x), shifted by a constant. Thus, similar to the entropy solution (5), it fully retains the relative density information of data points within the support.
However, because of the under-determined term µ∗(x), we cannot recover the distribution density pdata exactly from either c∗`2 or c ∗ ent if the data support is finite. Whether this ambiguity can
be resolved is beyond the scope of this paper, but poses an interesting research problem. 3. Finally, let’s consider consider a degenerate case, where K(pgen) is a constant. That is, we dont
provide any additional training signal for pgen at all. With K(pgen) = const, we simply have
c∗cst(x) = λ ∗ + µ∗(x),∀x ∈ X , (7)
whose discriminative power is fully controlled by the weak support discriminator µ∗(x). Thus, it follows that c∗cst(x) won’t be able to discriminate data points within the support of pdata, and its power to distinguish data from SUPP(pdata) and SUPP(pdata){ is weak. This closely matches the intuitive argument in the beginning of this section. Note that when K(pgen) is a constant, the objective function (1) simplifies to:
max c min pgen∈P E x∼pgen
[ c(x) ] − E x∼pdata [ c(x) ] , (8)
which is very similar to the EBGAN objective (Zhao et al., 2016, equation (2) and (4)). As we show in appendix A.2, compared to the objective in (8), the EBGAN objective puts extra constraints on the allowed discriminator function. In spite of that, the EBGAN objective suffers from the single-training-signal problem and does not guarantee that the discriminator will recover the real energy function (see appendix A.2 for detailed analysis).
As we finish the theoretical analysis of the proposed formulation, we want to point out that simply adding the same term K(pgen) to the original GAN formulation will not lead to both a generator that matches the data distribution, and a discriminator that retains the density information (see appendix A.3 for detailed analysis).
4 PARAMETRIC INSTANTIATION WITH ENTROPY APPROXIMATION
While the discussion in previous sections focused on the non-parametric case, in practice we are limited to a finite amount of data, and the actual problem involves high dimensional continuous spaces. Thus, we resort to parametric representations for both the generator and the discriminator. In order to train the generator using standard back-propagation, we do not parametrize the generator distribution directly. Instead, we parametrize a directed generator network that transforms random noise z ∼ pz(z) to samples from a continuous data space Rn. Consequently, we don’t have analytical access to the generator distribution, which is defined implicitly by the generator network’s noise→data mapping. However, the regularization term K(pgen) in the training objective (1) requires the generator distribution. Faced with this problem, we focus on the max-entropy formulation, and exploit two different approximations of the regularization term K(pgen) = −H(pgen).
4.1 NEAREST-NEIGHBOR ENTROPY GRADIENT APPROXIMATION
The first proposed solution is built upon an intuitive interpretation of the entropy gradient. Firstly, since we construct pgen by applying a deterministic, differentiable transform gθ to samples z from a fixed distribution pz , we can write the gradient of H(pgen) with respect to the generator parameters θ as follows:
−∇θH(pgen) = Ez∼pz [∇θ log pgen(gθ(z))] = Ez∼pz [ ∂gθ(z)
∂θ
∂ log pgen(gθ(z))
∂gθ(z)
] , (9)
where the first equality relies on the “reparametrization trick”. Equation 9 implies that, if we can compute the gradient of the generator log-density log pgen(x) w.r.t. any x = gθ(z), then we can directly construct the Monte-Carlo estimation of the entropy gradient ∇θH(pgen) using samples from the generator.
Intuitively, for any generated data x = gθ(z), the term ∂ log pgen(x)
∂x essentially describes the direction of local change in the sample space that will increase the log-density. Motivated by this intuition, we propose to form a local Gaussian approximation pigen of pgen around each point xi in a batch of samples {x1, ..., xn} from the generator, and then compute the gradient ∂ log pgen(xi)∂xi based on the
Gaussian approximation. Specifically, each local Gaussian approximation pigen is formed by finding the k nearest neighbors of xi in the batch {x1, ..., xn}, and then placing an isotropic Gaussian distribution at their mean (i.e. maximimum likelihood). Based on the isotropic Gaussian approximation, the resulting gradient has the following form
∂ log pgen(xi)
∂xi ≈ µi − xi, where µi =
1
k ∑ x′∈KNN(xi) x′ is the mean of the Gaussian (10)
Finally, note the scale of this gradient approximation may not be reliable. To fix this problem, we normalize the approximated gradient into unit norm, and use a single hyper-parameter to model the scale for all x, leading to the following entropy gradient approximation
−∇θH(pgen) ≈ α 1
k ∑ xi=gθ(zi) µi − xi ‖µi − xi‖2
(11)
where α is the hyper-parameter and µi is defined as in equation (10).
An obvious weakness of this approximation is that it relies on Euclidean distance to find the k nearest neighbors. However, Euclidean distance is usually not the proper metric to use when the effective dimension is very high. As the problem is highly challenging, we leave it for future work.
4.2 VARIATIONAL LOWER BOUND ON THE ENTROPY
Another approach we consider relies on defining and maximizing a variational lower bound on the entropy H(pgen(x)) of the generator distribution. We can define the joint distribution over observed data and the noise variables as pgen(x, z) = pgen(x | z)pgen(z), where simply pgen(z) = pz(z) is a fixed prior. Using the joint, we can also define the marginal pgen(x) and the posterior pgen(z | x). We can also write the mutual information between the observed data and noise variables as:
I(pgen(x); pgen(z)) = H(pgen(x))−H(pgen(x | z)) = H(pgen(z))−H(pgen(z | x)),
(12)
where H(pgen(. | .)) denotes the conditional entropy. By reorganizing terms in this definition, we can write the entropy H(pgen(x)) as:
H(pgen(x)) = H(pgen(z))−H(pgen(z | x)) +H(pgen(x | z)) (13)
We can think of pgen(x | z) as a peaked Gaussian with a fixed, diagonal covariance, and hence its conditional entropy is constant and can be dropped. Furthermore, H(pgen(z)) is also assumed to be fixed a priori. Hence, we can maximize H(pgen(x)) by minimizing the conditional entropy:
H(pgen(z | x)) = Ex∼pgen(x) [ Ez∼pgen(z|x) [− log pgen(z | x)] ] (14)
Optimizing this term is still problematic, because (i) we do not have access to the posterior pgen(z | x), and (ii) we cannot sample from it. Therefore, we instead minimize a variational upper bound defined by an approximate posterior qgen(z | x):
H(pgen(z | x)) = Ex∼pgen(x) [ Ez∼pgen(z|x) [− log qgen(z | x)]− KL(pgen(z | x)‖qgen(z | x)) ] ≤ Ex∼pgen(x) [ Ez∼pgen(z|x) [− log qgen(z | x)]
] = U(qgen).
(15)
We can also rewrite the variational upper bound as: U(qgen) = Ex,z∼pgen(x,z) [− log qgen(z | x)] = Ez∼pgen(z) [ Ex∼pgen(x|z) [− log qgen(z | x)] ] , (16)
which can be optimized efficiently with standard back-propagation and Monte Carlo integration of the relevant expectations based on independent samples drawn from the joint pgen(x, z). By minimizing this upper bound on the conditional entropy H(pgen(z | x)), we are effectively maximizing a variational lower bound on the entropy H(pgen(x)).
5 EXPERIMENTS
In this section, we verify our theoretical results empirically on several synthetic and real datasets. In particular, we evaluate whether the discriminator obtained from the entropy-regularized adversarial training can capture the density information (in the form of energy), while making sure the generator distribution matches the data distribution. For convenience, we refer to the obtained models as EGAN-Ent. Our experimental setting follows closely recommendations from Radford et al. (2015), except in Sec. 5.1 where we use fully-connected models (see appendix B.1 for details). 1
5.1 SYNTHETIC LOW-DIMENSIONAL DATA
First, we consider three synthetic datasets in 2-dimensional space, which are drawn from the following distributions: (i) Mixture of 4 Gaussians with equal mixture weights, (ii) Mixture of 200 Gaussians arranged as two spirals (100 components each spiral), and (iii) Mixture of 2 Gaussians with highly biased mixture weights, P (c1) = 0.9, P (c2) = 0.1. We visualize the ground-truth energy of these distributions along with 100K training samples in Figure 1. Since the data lies in 2-dimensional space, we can easily visualize both the learned generator (by drawing samples) and the discriminator for direct comparison and evaluation. We evaluate here our EGAN-Ent model using both approximations: the nearest-neighbor based approximation (EGAN-Ent-NN) and the variational-inference based approximation (EGAN-Ent-VI), and compare them with two baselines: the original GAN and the energy based GAN with no regularization (EGAN-Const).
Experiment results are summarized in Figure 2 for baseline models, and Figure 3 for the proposed models. As we can see, all four models can generate perfect samples. However, for the discriminator, both GAN and EGAN-Const lead to degenerate solution, assigning flat energy inside the empirical data support. In comparison, EGAN-Ent-VI and EGAN-Ent-NN clearly capture the density information, though to different degrees. Specifically, on the equally weighted Gaussian mixture and the two-spiral mixture datasets, EGAN-Ent-NN tends to give more accurate and fine-grained solutions compared to EGAN-Ent-VI. However, on the biased weighted Gaussian mixture dataset, EGAN-Ent-VI actually fails to captures the correct mixture weights of the two modes, incorrectly assigning lower energy to the mode with lower probability (smaller weight). In contrast, EGANEnt-NN perfectly captures the bias in mixture weight, and obtains a contour very close to the ground truth.
To better quantify these differences, we present detailed comparison based on KL divergence in appendix B.2. What’s more, the performance difference between EGAN-Ent-VI and EGAN-Ent-NN on biased Gaussian mixture reveals the limitations of the variational inference based approximation, i.e. providing inaccurate gradients. Due to space consideratiosn, we refer interested readers to the appendix B.3 for a detailed discussion.
5.2 RANKING NIST DIGITS
In this experiment, we verify that the results in synthetic datasets can translate into data with higher dimensions. While visualizing the learned energy function is not feasible in high-dimensional space, we can verify whether the learned energy function learns relative densities by inspecting the ranking of samples according to their assigned energies. We train on 28× 28 images of a single handwritten
1For more details, please refer to https://github.com/zihangdai/cegan_iclr2017.
digit from the NIST dataset. 2 We compare the ability of EGAN-Ent-NN with both EGAN-Const and GAN on ranking a set of 1,000 images, half of which are generated samples and the rest are real test images. Figures 4 and 5 show the top-100 and bottom-100 ranked images respectively for each model, after training them on digit 1. We also show in Figure 7 the mean of all training samples, so we can get a sense of what is the most common style (highest density) of digit 1 in NIST. We can notice that all of the top-ranked images by EGAN-Ent-NN look similar to the mean sample. In addition, the lowest-ranked images are clearly different from the mean image, with either high (clockwise or counter-clockwise) rotation degrees from the mean, or an extreme thickness level. We do not see such clear distinction in other models. We provide in the appendix B.4 the ranking of the full set of images.
5.3 SAMPLE QUALITY ON NATURAL IMAGE DATASETS
In this last set of experiments, we evaluate the visual quality of samples generated by our model in two datasets of natural images, namely CIFAR-10 and CelebA. We employ here the variationalbased approximation for entropy regularization, which can scale well to high-dimensional data. Figure 6 shows samples generated by EGAN-Ent-VI. We can see that despite the noisy gradients provided by the variational approximation, our model is able to generate high-quality samples.
2https://www.nist.gov/srd/nist-special-database-19, which is an extended version of MNIST with an average of over 74K examples per digit.
We futher validate the quality of our model’s samples on CIFAR-10 using the Inception score proposed by (Salimans et al., 2016) 3. Table 1 shows the scores of our EGAN-Ent-VI, the best GAN model from Salimans et al. (2016) which uses only unlabeled data, and an EGAN-Const model which has the same architecture as our model. We notice that even without employing suggested techniques in Salimans et al. (2016), energy-based models perform quite similarly to the GAN model. Furthermore, the fact that our model scores higher than EGAN-Const highlights the importance of entropy regularization in obtaining good quality samples.
6 CONCLUSION
In this paper we have addressed a fundamental limitation in adversarial learning approaches, which is their inability of providing sensible energy estimates for samples. We proposed a novel adversarial learning formulation which results in a discriminator function that recovers the true data energy. We provided a rigorous characterization of the learned discriminator in the non-parametric setting, and proposed two methods for instantiating it in the typical parametric setting. Our experimental results verify our theoretical analysis about the discriminator properties, and show that we can also obtain samples of state-of-the-art quality.
7 ACKNOWLEDGEMENTS
We would like to thank the developers of Theano (Theano Development Team, 2016) for developing such a powerful tool for scientific computing. Amjad Almahairi was supported by funding from Maluuba Research.
3Using the evaluation script released in https://github.com/openai/improved-gan/
Figure 7: mean digit
A SUPPLEMENTARY MATERIALS FOR SECTION 3
A.1 OPTIMAL DISCRIMINATOR FORM UNDER THE PROPOSED FORMULATION
Proof of proposition 3.1. Refining the Lagrange L(pgen, c) by introducing additional dual variables for the probability constraints (the second and third), the new Lagrange function has the form
L(pgen, c, µ, λ) = K(pgen)+ ∑ x∈X c(x) ( pgen(x)−pdata(x) ) − ∑ x∈X µ(x)pgen(x)+λ( ∑ x∈X pgen(x)−1) (17) where c(x) ∈ R,∀x, µ(x) ∈ R+,∀x, and λ ∈ R are the dual variables. The KKT conditions for the optimal primal and dual variables are as follows
∂K(pgen)
∂pgen(x) ∣∣∣∣ pgen=pdata + c∗(x)− µ∗(x) + λ∗ = 0, ∀x (stationarity)
µ∗(x)p∗gen(x) = 0, ∀x (complement slackness) µ∗(x) ≥ 0, ∀x (dual feasibility)
p∗gen(x) ≥ 0, p∗gen(x) = pdata(x), ∀x (primal feasibility)∑ x∈X pgen(x) = 1 (primal feasibility)
(18)
Rearranging the conditions above, we get p∗gen(x) = pdata(x),∀x ∈ X as well as equation (4), which concludes the proof.
A.2 OPTIMAL CONDITIONS OF EBGAN
In (Zhao et al., 2016), the training objectives of the generator and the discriminator cannot be written as a single minimax optimization problem since the margin structure is only applied to the objective of the discriminator. In addition, the discriminator is designed to produce the mean squared reconstruction error of an auto-encoder structure. This restricted the range of the discriminator output to be non-negative, which is equivalent to posing a set constraint on the discriminator under the non-parametric setting.
Thus, to characterize the optimal generator and discriminator, we adapt the same analyzing logic used in the proof sketch of the original GAN (Goodfellow et al., 2014). Specifically, given a specific generator distribution pgen, the optimal discriminator function given the generator distribution c∗(x; pgen) can be derived by examining the objective of the discriminator. Then, the conditional optimal discriminator function is substituted into the training objective of pgen, simplifying the “adversarial” training as a minimizing problem only w.r.t. pgen, which can be well analyzed.
Firstly, given any generator distribution pgen, the EBGAN training objective for the discriminator can be written as the following form
c∗(x; pgen) = argmax c∈C − E pgen max(0,m− c(x))− E pdata c(x)
= argmax c∈C E pgen min(0, c(x)−m)− E pdata
c(x) (19)
where C = {c : c(x) ≥ 0,∀x ∈ X} is the set of allowed non-negative discriminator functions. Note this set constraint comes from the fact the mean squared reconstruction error as discussed above.
Since the problem (19) is independent w.r.t. each x, the optimal solution can be easily derived as
c∗(x; pgen) =  0, pgen(x) < pdata(x) m, pgen(x) > pdata(x) αx, pgen(x) = pdata(x) > 0
βx, pgen(x) = pdata(x) = 0
(20)
where αx ∈ [0,m] is an under-determined number, a βx ∈ [0,∞) is another under-determined nonnegative real number, and the subscripts in m,αx, βx reflect that fact that these under-determined values can be distinct for different x.
This way, the overall training objective can be cast into a minimization problem w.r.t. pgen,
p∗gen = argmin pgen∈P E x∼pgen c∗(x; pgen)− E x∼pdata c∗(x; pgen)
= argmin pgen∈P ∑ x∈X [ pgen(x)− pdata(x) ] c∗(x; pgen)
(21)
where the second term of the first line is implicitly defined as the problem is an adversarial game between pgen and c. Proposition A.1. The global optimal of the EBGAN training objective is achieved if and only if pgen = pdata. At that point, c∗(x) is fully under-determined.
Proof. The proof is established by showing contradiction.
Firstly, assume the optimal p∗gen 6= pdata. Thus, there must exist a non-equal setX6= = {x | pdata(x) 6= p∗gen(x)}, which can be further splitted into two subsets, the greater-than set X> = {x | p∗gen(x) > pdata(x)}, and the less-than set X< = {x | p∗gen(x) < pdata(x)}. Similarly, we define the equal set X= = {x : p∗gen(x) = pdata(x)}. Obviously, X> ⋃ X< ⋃ X= = X .
Let L(pgen) = ∑ x∈X [ pgen(x) − pdata(x) ] c∗(x; pgen), substituting the results from equation (20) into (21), the L(pgen)∗ can be written as
L(p∗gen) = ∑
x∈X< ⋃ X< ⋃ X=
[ p∗gen(x)− pdata(x) ] c∗(x; p∗gen)
= ∑ x∈X< [ p∗gen(x)− pdata(x) ] c∗(x; p∗gen) + ∑ x∈X> [ p∗gen(x)− pdata(x) ] c∗(x; p∗gen)
= m ∑ x∈X> p∗gen(x)− pdata(x)
> 0
(22)
However, when p′gen = pdata, we have
L(p′gen) = 0 < L(p ∗ gen) (23)
which contradicts the optimal (miminum) assumption of p∗gen. Hence, the contradiction concludes that at the global optimal, p∗gen = pdata. By equation (20), it directly follows that c
∗(x; p∗gen) = αx, which completes the proof.
A.3 ANALYSIS OF ADDING ADDITIONAL TRAINING SIGNAL TO GAN FORMULATION
To show that simply adding the same training signal to GAN will not lead to the same result, it is more convenient to directly work with the formulation of f -GAN (Nowozin et al., 2016, equation (6)) family, which include the original GAN formulation as a special case.
Specifically, the general f -GAN formulation takes the following form
max c min pgen∈P E x∼pgen
[ f?(c(x)) ] − E x∼pdata [ c(x) ] , (24)
where the f?(·) denotes the convex conjugate (Boyd & Vandenberghe, 2004) of the f -divergence function. The optimal condition of the discriminator can be found by taking the variation w.r.t. c, which gives the optimal discriminator
c∗(x) = f ′( pdata(x)
pgen(x) ) (25)
where f ′(·) is the first-order derivative of f(·). Note that, even when we add an extra term L(pgen) to equation (24), since the term K(pgen) is a constant w.r.t. the discriminator, it does not change the result given by equation (25) about the optimal discriminator. As a consequence, for the optimal
discriminator to retain the density information, it effectively means pgen 6= pdata. Hence, there will be a contradiction if both c∗(x) retains the density information, and the generator matches the data distribution.
Intuitively, this problem roots in the fact that f -divergence is quite “rigid” in the sense that given the pgen(x) it only allows one fixed point for the discriminator. In comparison, the divergence used in our proposed formulation, which is the expected cost gap, is much more flexible. By the expected cost gap itself, i.e. without the K(pgen) term, the optimal discriminator is actually under-determined.
B SUPPLEMENTARY MATERIALS FOR SECTION 5
B.1 EXPERIMENT SETTING
Here, we specify the neural architectures used for experiements presented in Section 5.
Firstly, for the Egan-Ent-VI model, we parameterize the approximate posterior distribution qgen(z | x) with a diagonal Gaussian distribution, whose mean and covariance matrix are the output of a trainable inference network, i.e.
qgen(z | x) = N (µ, Iσ2) µ, log σ = f infer(x)
(26)
where f infer denotes the inference network, and I is the identity matrix. Note that the Inference Network only appears in the Egan-Ent-VI model.
For experiments with the synthetic datasets, the following fully-connected feed forward neural networks are employed
• Generator: FC(4,128)-BN-ReLU-FC(128,128)-BN-ReLU-FC(128,2) • Discriminator: FC(2,128)-ReLU-FC(128,128)-ReLU-FC(128,1) • Inference Net: FC(2,128)-ReLU-FC(128,128)-ReLU-FC(128,4*2)
where FC and BN denote fully-connected layer and batch normalization layer respectively. Note that since the input noise to the generator has dimension 4, the Inference Net output has dimension 4*2, where the first 4 elements correspond the inferred mean, and the last 4 elements correspond to the inferred diagonal covariance matrix in log scale.
For the handwritten digit experiment, we closely follow the DCGAN (Radford et al., 2015) architecture with the following configuration
• Generator: FC(10,512*7*7)-BN-ReLU-DC(512,256;4c2s)-BN-ReLU -DC(256,128;4c2s)-BN-ReLU-DC(128,1;3c1s)-Sigmoid
• Discriminator: CV(1,64;3c1s)-BN-LRec-CV(64,128;4c2s)-BN-LRec -CV(128,256;4c2s)-BN-LRec-FC(256*7*7,1)
• Inference Net: CV(1,64;3c1s)-BN-LRec-CV(64,128;4c2s)-BN-LRec -CV(128,256;4c2s)-BN-LRec-FC(256*7*7,10*2)
Here, LRec is the leaky rectified non-linearity recommended by Radford et al. (2015). In addition, CV(128,256,4c2s) denotes a convolutional layer with 128 input channels, 256 output channels, and kernel size 4 with stride 2. Similarly, DC(256,128,4c2s) denotes a corresponding transposed convolutional operation. Compared to the original DCGAN architecture, the discriminator under our formulation does not have the last sigmoid layer which squashes a scalar value into a probability in [0, 1].
For celebA experiment with 64× 64 color images, we use the following architecture
• Generator: FC(10,512*4*4)-BN-ReLU-DC(512,256;4c2s)-BN-ReLU-DC(256,128;4c2s) -BN-ReLU-DC(256,128;4c2s)-BN-ReLU-DC(128,3;4c2s)-Tanh
• Discriminator: CV(3,64;4c2s)-BN-LRec-CV(64,128;4c2s)-BN-LRec-CV(128,256;4c2s) -BN-LRec-CV(256,256;4c2s)-BN-LRec-FC(256*4*4,1)
• Inference Net: CV(3,64;4c2s)-BN-LRec-CV(64,128;4c2s)-BN-LRec-CV(128,256;4c2s) -BN-LRec-CV(256,256;4c2s)-BN-LRec-FC(256*4*4,10*2)
For Cifar10 experiment, where the image size is 32× 32, similar architecture is used
• Generator: FC(10,512*4*4)-BN-ReLU-DC(512,256;4c2s)-BN-ReLU-DC(256,128;3c1s) -BN-ReLU-DC(256,128;4c2s)-BN-ReLU-DC(128,3;4c2s)-Tanh
• Discriminator: CV(3,64;3c1s)-BN-LRec-CV(64,128;4c2s)-BN-LRec-CV(128,256;4c2s) -BN-LRec-CV(256,256;4c2s)-BN-LRec-FC(256*4*4,1)
• Inference Net: CV(3,64;3c1s)-BN-LRec-CV(64,128;4c2s)-BN-LRec-CV(128,256;4c2s) -BN-LRec-CV(256,256;4c2s)-BN-LRec-FC(256*4*4,10*2)
Given the chosen architectures, we follow Radford et al. (2015) and use Adam as the optimization algorithm. For more detailed hyper-parameters, please refer to the code.
B.2 QUANTITATIVE COMPARISON OF DIFFERENT MODELS
In order to quantify the quality of recovered distributions, we compute the pairwise KL divergence of the following four distributions:
• The real data distribution with analytic form, denoted as pdata • The empirical data distribution approximated from the 100K training data, denoted as pemp • The generator distribution approximated from 100K generated data, denoted as pgen • The discriminator distribution re-normalized from the learned energy, denoted as pdisc
Since the synthetic datasets are two dimensional, we approximate both the empirical data distribution and the generator distribution using the simple histogram estimation. Specifically, we divide the canvas into a 100-by-100 grid, and assign each sample into its nearest grid cell based on euclidean distance. Then, we normalize the number of samples in each cell into a proper distribution. When recovering the discriminator distribution from the learned energy, we assume that µ∗(x) = 0 (i.e. infinite data support), and discretize the distribution into the same grid cells
pdisc(x) = exp(−c∗(x))∑
x′∈Grid exp(−c∗(x′)) ,∀x ∈ Grid
Based on these approximation, Table 2 summarizes the results. For all measures related to the discriminator distribution, EGAN-Ent-VI and EGAN-Ent-NN significantly outperform the other two baseline models, which matches our visual assessment in Figure 2 and 3. Meanwhile, the generator distributions learned from our proposed framework also achieve relatively lower divergence to both the empirical data distribution and the true data distribution.
B.3 COMPARISON OF THE ENTROPY (GRADIENT) APPROXIMATION METHODS
In order to understand the performance difference between EGAN-Ent-VI and EGAN-Ent-NN, we analyze the quality of the entropy gradient approximation during training. To do that, we visualize some detailed training information in Figure 8.
As we can see in figure 8a, the viarational entropy gradient approximation w.r.t. samples is not accurate:
• It is inaccurate in terms of gradient direction. Ideally, the direction of the entropy gradient should be pointing from the center of its closest mode towards the surroundings, with
the direction orthogonal to the implicit contour in Fig. (1,2). However, the direction of gradients in the Fig. (2,3) does not match this. • It is inaccurate in magnitude. As we can see, the entropy approximation gradient (Fig.
(2,3)) has much larger norm than the discriminator gradient (Fig. (2,2)). As a result, the total gradient (Fig. (2,4)) is fully dominated by the entropy approximation gradient. Thus, it usually takes much longer for the generator to learn to generate rare samples, and the training also proceeds much slower compared to the nearest neighbor based approximation.
In comparison, the nearest neighbor based gradient approximation is much more accurate as shown in 8b. As a result, it leads to more accurate energy contour, as well as faster training. What’s more, from Figure 8b Fig. (2,4), we can see the entropy gradient does have the cancel-out effect on the discriminator gradient, which again matches our theory.
B.4 RANKING NIST DIGITS
Figure 9 shows the ranking of all 1000 generated and real images (from the test set) for three models: EGAN-Ent-NN, EGAN-Const, and GAN. We can clearly notice that in EGAN-Ent-NN the topranked digits look very similar to the mean digit. From the upper-left corner to the lower-right corner, the transition trend is: the rotation degree increases, and the digits become increasingly thick or thin compared to the mean. In addition, samples in the last few rows do diverge away from the mean image: either highly diagonal to the right or left, or have different shape: very thin or thick, or typewriter script. Other models are not able to achieve a similar clear distinction for high versus low probability images. Finally, we consistently observe the same trend in modeling other digits, which are not shown in this paper due to space constraint.
B.5 CLASSIFIER PERFORMANCE AS A PROXY MEASURE
As mentioned in Section 5, evaluating the proposed formulation quantitatively on high-dimensional data is extremely challenging. Here, in order to provide more quantitative intuitions on the learned discriminator at convergence, we adopt a proxy measure. Specifically, we take the last-layer activation of the converged discriminator network as fixed pretrained feature, and build a linear classifier upon it. Hypothetically, if the discriminator does not degenerate, the extracted last-layer feature should maintain more information about the data points, especially compared to features from degenerated discriminators. Following this idea, we first train EGAN-Ent-NN, EGAN-Const, and GAN on the MNIST till convergence, and then extract the last-layer activation from their discriminator networks as fixed feature input. Based on fixed feature, a randomly initialized linear classifier is trained to do classification on MNIST. Based on 10 runs (with different initialization) of each of the three models, the test classification performance is summarized in Table 3. For comparison purpose, we also include a baseline where the input features are extracted from a discriminator network with random weights.
Based on the proxy measure, EGAN-Ent-NN seems to maintain more information of data, which suggests that the discriminator from our proposed formulation is more informative. Despite the positive result, it is important to point out that maintaining information about categories does not necessarily mean maintaining information about the energy (density). Thus, this proxy measure should be understood cautiously.

Review:
The submission explores several alternatives to provide the generator function in generative adversarial training with additional gradient information. The exposition starts by describing a general formulation about how this additional gradient information (termed K(p_gen) could be added to the generative adversarial training objective function (Equation 1). Next, the authors prove that the shape of the optimal discriminator does indeed depend on the added gradient information (Proposition 3.1), which is unsurprising. Finally, the authors propose three particular alternatives to construct K(p_gen): the negative entropy of the generator distribution, the L2 norm of the generator distribution, and a constant function (which resembles the EBGAN objective of Zhao et al, 2016).
The exposition moves then to an experimental evaluation of the method, which sets K(p_gen) to be the approximate entropy of the generator distribution. At this point, my intuition is that the objective function under study is the vanilla GAN objective, plus a regularization term that encourages diversity (high entropy) in the generator distribution. The hope of the authors is that this regularization will transform the discriminator into an estimate of the energy landscape of the data distribution.
The experimental evaluation proceeds by 1) showing the contour plots of the obtained generator distribution for a 2D problem, 2) studying the generation diversity in MNIST digits, and 3) showing some samples for CIFAR-10 and CelebA. The 2D problem results are convincing, since one can clearly observe that the discriminator scores translate into unnormalized values of the density function. The MNIST results offer good intuition also: the more prototypical digits are assigned larger scores (unnormalized densities) by the discriminator, and the less prototypical digits are assigned smaller scores. The sample experiments from Section 5.3 are less convincing, since no samples from baseline models are provided for comparison.
To this end, I would recommend the authors to clarify three aspects. First, we have seen that entropy regularization leads to a discriminator that estimates the energy landscape of the data distribution. But, how does this regularization reshape the generator function? It would be nice to see the mean MNIST digit according to the generator, and some other statistics if possible. Second, how do the samples produced by the proposed methods compare (visually speaking) to the state-of-the art? Third, what are the *shortcomings* of this method versus vanilla GAN? Too much computational overhead? What are the qualitative and quantitative differences between the two entropy estimators proposed in the manuscript?
Overall, a clearly written paper. I vote for acceptance.
As an open question to the authors: What breakthroughs should we pursue to derive a GAN objective where the discriminator is an estimate of the data density function, after training?