O N THE D ISCRIMINATION -G ENERALIZATION T RADEOFF IN GAN S
Anonymous authors
Paper under double-blind review
A BSTRACT
Generative adversarial training can be generally understood as minimizing certain moment matching loss deﬁned by a set of discriminator functions, typically neural networks.
The discriminator set should be large enough to be able to uniquely identify the true distribution ( discriminative ), and also be small enough to go beyond memorizing samples ( generalizable ).
In this paper, we show that a discriminator set is guaranteed to be discriminative whenever its linear span is dense in the set of bounded continuous functions.
This is a very mild condition satisﬁed even by neural networks with a single neuron .
Further, we develop generalization bounds between the learned distribution and true distribution under different evaluation metrics.
When evaluated with neural distance, our bounds show that generalization is guaranteed as long as the discriminator set is small enough, regardless of the size of the generator or hypothesis set.
When evaluated with KL divergence, our bound provides an explanation on the counter-intuitive behaviors of testing likelihood in GAN training.
Our analysis sheds lights on understanding the practical performance of GANs.
I NTRODUCTION
Generative adversarial networks (GANs) (Goodfellow et al., 2014) and their variants can be generally understood as minimizing certain moment matching loss deﬁned by a set of discriminator functions.
Mathematically, GANs minimize the integral probability metric (IPM) (M ¨ uller, 1997), that is,
[Equation]
where ˆ µ m is the empirical measure of the observed data, and F and G are the sets of discriminators and generators, respectively.
Wasserstain GAN (W-GAN) (Arjovsky et al., 2017).
F = Lip 1 ( X ) :
= { f : ||
f
|| Lip ≤ 1 } , corresponding to the Wasserstain-1 distance.
MMD-GAN (Li et al., 2015; Dziugaite et al., 2015; Li et al., 2017a).
F is taken as the unit ball in a Reproducing Kernel Hilbert Space (RKHS), corresponding to the Maximum Mean Discrepency (MMD).
Energy-based GANs (Zhao et al., 2016).
F is taken as the set of continuous functions bounded between 0 and M for some constant M > 0 , corresponding to the total variation distance (Arjovsky et al., 2017).
f-GAN (Nowozin et al., 2016) minimizes the f -divergence, which can be viewed a form of regularized moment matching loss deﬁned over all possible functions as shown by Liu et al.
See also Appedix B.
Due to computational tractability, however, the practical GANs take F as a parametric function class, typically, F nn = { f θ ( x ): θ ∈ Θ } where f θ ( x ) is a neural network indexed by parameters θ that take values in Θ ⊂ R p .
Consequently, the related d F nn ( µ, ν ) is called neural network distance, or neural distance (Arora et al., 2017).
Although d F nn ( µ, ν ) is meant to be a surrogate, its properties can be fundamentally different from the original objective functions.
For example, in W-GAN, because F nn is a much smaller discriminator set than Lip 1 ( X ) , it is unclear from the current GAN literature whether d F nn ( µ, ν ) is a discriminative metric in that d F nn ( µ, ν ) = 0 implies µ = ν .
This discrimination is critical to ensure the consistency of the learning result.
This motivated us to study the properties of d F nn ( µ, ν ) with parametric function sets F nn , instead of the original Wasserstein distance or f -divergence.
A broader question is in developing learning bounds and studying how they depend on the discriminator set F and the generator set G , under different evaluation metrics of interest.
Speciﬁcally, assuming ν m is an (approximate) solution of (1), we are interested in obtaining bounds between ν m and the underlying true distribution µ , under a given evaluation
metric d eval ( µ, ν m ) .
Existing analysis has been mostly focusing on the case when the evaluation metric coincides with the optimization metric, that is, d eval ( µ, ν ) =
d F ( µ, ν ) , which, however, favors smaller discriminator sets that deﬁne “easier” evaluation metrics.
It is of interest to develop bounds for evaluation metrics independent of F , such as bounded Lipschitz distance that metrizes weak convergence, and KL divergence that connects to testing likelihood.
Contribution.
We show that the role of discriminators F is best illustrated by the conditions under which d F ( µ, ν ) metrizes weak convergence (or convergence in distribution), that is,
[Equation]
if and only if
[Equation]
for any probability measures µ and ν m .
The choice of F should strike a balance to achieve (2): i) F should be large enough to make d F ( µ, ν ) discrimiantive in that d F ( µ, ν m )
→ 0 can imply that ν m weakly converges to µ .
Further, with a given metric d eval ( µ, ν ) , the discriminator set F should be large enough so that a small d F ( µ, ν
) implies a small d eval ( µ, ν ) in certain sense.
These are basic requirements in justifying d F ( µ, ν ) as a valid learning objective function.
ii) F should also be relatively small so that ν m (cid:42) µ implies that d F ( µ, ν m ) approaches to zero.
This is essential to guaranteeing that the training and testing loss are similar to each other and hence the algorithm is generalizable .
Further, in order to obtain a low sample complexity, F should be sufﬁciently small so that d F ( µ, ν m ) decays with a fast rate, preferably O (1 / √ m ) .
The theme of this work is to characterize the conditions under which i) and ii) hold and develop bounds of d eval ( µ, ν m ) that characterize the role of discriminators F and generators G .
Our contributions are summarized as follows.
We show that a discriminator set F is discriminative once the linear span of F is dense in the bounded continuous (or Lipschitz) function space.
This is a mild condition that can satisﬁed, for example, even for neural networks consists of a single neuron .
See Section 2. 2.
We develop techniques using neural distance d F ( µ, ν ) to provide upper bounds of different evaluation metrics d eval ( µ, ν ) of interest, including bounded Lipschitz (BL) distance and KL divergence, which provides a key step for developing learning bounds of GANs under these metrics.
See Section 2.1. 3.
We characterize the generalizability of GANs using the Rademacher complexity of discriminator set F and put together bounds between the true distributions µ and GAN estimators ν m under different evaluation metrics in Section 3.
Under the neural distance, our bounds (Corollary 3.23.3) show that generalization is guaranteed as long as the discriminator set is small enough, regardless of the size of the generator or hypothesis set G .
This seemingly-surprising result is reasonable because in this case the evaluation metric d F ( µ, ν m ) depends on the discriminator set.
When the KL divergence is used as the evaluation metric, our bound (Corollary 3.5) suggests that the generator and discriminator sets have to be compatible in that the log density ratios of the generators and the true distributions should exist and be included inside the linear span of the discriminator set.
The strong condition that log-density ratio should exist partially explains the counter-intuitive behavior of testing likelihood in ﬂow GANs (e.g., Danihelka et al., 2017; Grover et al., 2017).
We extend our analysis to study neural f -divergences that are the learning objective of f -GANs, and establish similar results on the discrimination and generalization properties of neural f - divergences; see Appedix B. Different from neural distance, a neural f -divergence is discriminative if linear span of its discriminators without the output activation function is dense in the bounded continuous function space.
1. N OTATIONS
We use X to denote a subset of R d .
For each continuous function f : X → R , we deﬁne the maximum norm as (cid:107) f (cid:107) ∞ =
sup x ∈ X | f ( x ) | , and the Lipschitz norm (cid:107) f (cid:107) Lip = sup {| f ( x ) − f ( y ) | / (cid:107) x − y (cid:107) : x,
y ∈ X, x (cid:54) =
y } , and the bounded Lipschitz (BL) norm (cid:107) f (cid:107) BL = max {(cid:107)
f
(cid:107) Lip , (cid:107) f (cid:107) ∞ } .
The set of continuous functions on X is denoted by C ( X ) , and the Banach space of bounded continuous function is C b ( X )
= { f ∈ C ( X ): (cid:107) f (cid:107) ∞ < ∞} .
The set of Borel probability measures on X is denoted by P B ( X ) .
In this paper, we assume that all measures involved belong to P B ( X ) , which is sufﬁcient in all practical applications.
We denote by E µ [ f ] the integral of f with respect to probability measure µ .
The weak convergence, or convergence in distribution, is denoted by ν n (cid:42) ν .
Given a base measures τ (e.g., Lebesgue measure), the density of µ ∈ P B ( X ) , if it exists, is denoted by ρ µ = d µ d τ .
We do not assume density exists in our main theoretical results, except the cases when discussing KL divergence.
D ISCRIMINATIVE PROPERTIES OF NEURAL DISTANCES FOR GAN
As listed in the introduction, many variants of GAN can be viewed as minimizing the integral probability metric (1).
Without loss of generality, we assume that the discriminator set F is even, i.e., f ∈ F implies − f ∈ F .
Intuitively speaking, minimizing (1) towards zero corresponds to matching the moments E µ [ f ] =
E ν [ f ] for all discriminators f ∈ F .
In their original formulation, all those discriminator sets are non-parametric, inﬁnite dimensional, and large enough to guarantee that d F ( µ, ν ) = 0 implies µ = ν .
In practice, however, the discriminator set is typically restricted to parametric function classes of form F nn = { f θ : θ ∈ Θ } .
When f θ is a neural network, we call d F nn ( µ, ν ) a neural distance following Arora et al.
Neural distances are the actual object function that W-GAN optimizes in practice because they can be practically optimized and can leverage the representation power of neural networks.
Therefore, it is of great importance to directly study neural distances, instead of
Wasserstein metric, in order to understand practical performance of GANs.
Because the parameter function set F nn is much smaller than the non-parametric sets like Lip 1 ( X ) , a key question is whether F nn is large enough so that moment matching on F nn (i.e., d F nn ( µ, ν ) = 0 ) implies µ = ν .
It turns out the answer is afﬁrmative once F nn is large enough so that its linear span (instead of F nn itself) forms a universal approximator.
This is a rather weak condition, which is satisﬁed even by very small sets such as neural networks with a single neuron .
We make this concrete in the following.
Deﬁnition 2.1 .
Let ( X, d X ) be a metric space and F be a set of functions on X .
We say that d F ( µ, ν ) (and F ) is discriminative if
[Equation]
for any two Borel probability measures µ, ν ∈ P B ( X ) .
In other words, F is discriminative if the moment matching on F , i.e., E µ [ f ] = E ν [ f ] for any f ∈ F , implies µ = ν .
The key observation is that E µ [ f ] = E ν
[ f ] for any f ∈ F implies the same holds true for all f in the linear span of F .
Therefore, it is sufﬁcient to require the linear span of F , instead of F itself, to be large enough to well approximate all the indicator test functions.
Theorem 2.2.
For a given function set F ⊂ C b ( X ) , deﬁne
[Equation]
Then d F ( µ, ν ) is discriminative if span F is dense in the space of bounded continuous functions
C b ( X ) under the uniform norm || · || ∞ , that is, for any f ∈ C b ( X ) and (cid:15) > 0 , there exists an f (cid:15) ∈ span F such that (cid:107)
f − f (cid:15) (cid:107) ∞ ≤ (cid:15) .
An equivalent way to put is that C b ( X ) is included in the closure of span F , that is,
[Equation]
Further, (4) is a necessary condition for d F ( µ, ν ) to be discriminative if X is a compact space.
Remark 2.1.
The basic idea of characterizing probability measures using functions in C b ( X ) is closely related to the concept of weak convergence.
Recall that a sequence ν n weakly converges to µ , i.e., ν n (cid:42) µ , if and only if E ν n [ f ] → E µ [ f ] for all f ∈ C b ( X ) .
Proof of the sufﬁcient part of
Theorem 2.2 is standard and the same with proof of the uniqueness of weak convergence; see, e.g., Lemma 9.3.2 in Dudley (2002).
Remark 2.2.
We obtain similar results for neural f -divergence d φ, F ( µ || ν ) in Theorem B.1.
The difﬁculty in analyzing neural f -divergence is that moment matching on the discriminator set is only a sufﬁcient condition for minimizing neural f -divergence, i.e., [Equation] Consequently, cl ( span F ) ⊇ C b ( X ) is only necessary but not sufﬁcient for a neural f -divergence to be discriminative. When the discriminators are neural networks, we show that moment matching on the function set that consists of discriminators without their output activation function, denoted as F 0 , is a necessary condition for minimizing neural f -divergence, i.e.,
[Equation]
We refer to Theorem B.1 (ii) for a precise statement.
Therefore, a neural f -divergence is discriminative if linear span of its discriminators without the output activation function is dense in the bounded continuous function space. Remark 2.3. Because the set of bounded Lipschitz functions BL( X ) = { f ∈ C b ( X ): ||
f || Lip < ∞} is dense in C b ( X ) , the condition in (4) can be replaced by a weaker condition cl ( span F ) ⊇ BL( X ) . One can deﬁne a norm (cid:107) · (cid:107) BL for functions in BL( X ) by (cid:107) f (cid:107) BL = max {(cid:107) f (cid:107) Lip , (cid:107) f (cid:107) ∞ } .
This deﬁnes the bounded Lipschitz (BL) distance,
[Equation]
The BL distance is known to metrize weak convergence in sense that d BL ( µ, ν n ) → 0 is equivalent to ν n (cid:42) µ for all Borel probability measures on R d ; see section 8.3 in Bogachev (2007).
Neural distances are discriminative.
The key message of Theorem 2.2 is that it is sufﬁcient to require cl (
span F ) ⊇
C b
( X ) (Condition (4)), which is a much weaker condition than the perhaps more straightforward condition cl ( F ) ⊇ C b ( X ) .
In fact, (4) is met by function sets that are much smaller than what we actually use in practice.
For example, it is satisﬁed by the neural networks with only a single neuron, i.e.,
[Equation]
This is because its span span F nn includes neural networks with inﬁnite numbers of neurons, which are well known to be universal approximators in C b ( X ) according to classical theories (e.g., Cybenko, 1989;
Hornik et al., 1989; Hornik, 1991; Leshno et al., 1993; Barron, 1993).
We recall the following classical result.
Theorem 2.3 (Theorem 1 in Leshno et al.
Let σ : R → R be a continuous activation function and X
⊂ R
d be any compact set.
Let F nn be the set of neural networks with a single neuron as deﬁned in (5) , then span F nn is dense in C ( X ) if and only if σ is not a polynomial.
The above result requires that the parameters [ w, b ] take values in R d +1 .
In practice, however, we can only efﬁciently search in bounded parameter sets of [ w, b ] using local search methods like gradient descent.
We observe that it is sufﬁcient to replace R d +1 with a bounded parameter set Θ for non-decreasing homogeneous activation functions such as σ ( u ) =
max { u, 0 } α with α ∈ N ; note that α = 1 is the widely used rectiﬁed linear unit (ReLU).
Corollary 2.4.
Let X ⊂ R d be any compact set, and σ ( u ) = max { u, 0 } α ( α ∈ N ), and F nn = { σ ( w (cid:62) x + b ): [ w, b ] ∈ Θ } .
Then span F nn is dense in C b ( X ) if
[Equation]
For the case when Θ = { θ ∈ R d +1 : (cid:107) θ (cid:107) 2 ≤ 1 } , Bach (2017) not only proves that span F nn is dense in Lip 1 ( X ) (and thus dense in C b ( X ) ), but also gives the convergence rate.
Therefore, for ReLU activation functions, F nn with bounded parameter sets, like { θ : (cid:107) θ (cid:107) ≤ 1 } or { θ : (cid:107) θ (cid:107) = 1 } for any norm on R d +1 , is sufﬁcient to discriminate any two Borel probability measures.
Note that this is not true for some other activation functions such as tanh or sigmoid, because there is an approximation gap between span { σ ( w (cid:62) x + b ) : [ w, b ] ∈ Θ ⊂ R d +1 } and
C b ( X ) when Θ ⊂ R d +1 is bounded; see e.g., Barron (1993)
(Theorem 3).
From this perspective, homogeneous activation functions such as ReLU are preferred as discriminators.
One advantage of using bounded parameter set Θ is that it makes F nn have a bounded Lipschitz norm, and hence the corresponding neural distance is upper bounded by Wasserstein distance.
In fact, W-GAN uses weight clipping to explicitly enforce (cid:107) θ (cid:107) ∞ ≤ δ .
However, we should point out that the Lipschitz constraint does not help in making F discriminative since the constraint decreases, instead of enlarges, the function set F .
Instead, the role of the Lipschitz constraint should be mostly in stabilizing the training (Arjovsky et al., 2017) and assuring a generalization bound as we discuss in Section 3.
Another related way to justify the Lipschitz constraint is its relation to metrizing weak convergence, as we discuss in the sequel.
Neural distance and weak convergence.
If F is discriminative, then d F ( µ, ν ) = 0 implies µ = ν .
In practice, however, we often cannot achieve d F ( µ, ν ) = 0 strictly.
Instead, we often have d F ( µ, ν n ) → 0 for a sequence of ν n and want to establish the weak convergence ν n (cid:42) µ .
Theorem 2.5.
Let ( X, d X ) be any metric space.
If span F is dense in C b ( X ) , we have lim n →∞ d F ( µ, ν n ) = 0 implies ν n weakly converges to µ .
Additionally, if F is contained in a bounded Lipchitz function space, i.e., there exists 0 < C < ∞ such that || f || BL ≤ C for all f ∈ F , then ν n weakly converges to µ implies lim n →∞ d F ( µ, ν n ) = 0 .
Theorem 10 of Liu et al.
(2017) states a similar result for generic adversarial divergences, but does not obtain the speciﬁc weak convergence result for neural distances due to lacking of Theorem 2.2.
Another difference is that Theorem 10 of Liu et al.
(2017) heavily relies on the compactness assumption of X , while our result does not need this assumption.
We provide the proof for Theorem 2.5 in
Appendix C.
When X is compact, Wasserstein distance and the BL distance are equivalent and both metrize weak convergence.
As we discussed earlier, the condition cl ( span F ) =
C b ( X ) and F ⊆ Lip K ( X ) are satisﬁed by neural networks F nn with ReLU activation function and bounded parameter set Θ .
Therefore, the related neural distance d F nn is topologically equivalent to the Wasserstein and BL distance, because all of them metrize the weak convergence.
This does not imply , however, that they are equivalent in the metric sense (or strongly equivalent) since the ratio d BL ( µ, ν ) /d
F nn ( µ, ν ) can be unbounded.
In general, the neural distances are weaker than the BL distance because of smaller
F .
In Section 2.1 (and particularly Corollary 2.8), we draw more discussions on the bounds between
BL distance and neural distances.
2. D ISCRIMINATIVE POWER OF NEURAL DISTANCES
Theorem 2.2 characterizes the condition under which a neural distance is discriminative, and shows that even neural networks with a single neuron are sufﬁcient to be discriminative.
This does not explain, however, why it is beneﬁcial to use larger and deeper networks as we do in practice.
What is missing here is to frame and understand how discriminative or strong a neural distance is.
This is because even if d F ( µ, ν ) is discriminative, it can be relatively weak in that d F ( µ, ν ) may be small when µ and ν are very different under standard metrics (e.g., BL distance).
Obviously, a larger F yields a stronger neural distance, that is, if F ⊂ F (cid:48) , then d F ( µ, ν ) ≤ d F (cid:48) ( µ, ν ) .
For example, because it is reasonable to assume that neural networks are bounded Lipschitz when X and Θ are bounded, we can control a neural distance with the BL distance:
[Equation]
where C := sup f ∈F {|| f || BL } < ∞ .
A more difﬁcult question is if we can establish inequalities in the other direction, that is, controlling d BL ( µ, ν ) , or in general a stronger d F (cid:48) ( µ, ν ) , with a weaker d F ( µ, ν ) in some way.
In this section, we characterize conditions under which this is possible and develop bounds that allow us to use neural distances to control stronger distances such as BL distance, and even KL divergence.
These bounds are used in Section 3 to translate generalization bounds in d F ( µ, ν ) to that in BL distance and KL divergence.
The core of the discussion involves understanding how d F ( µ, ν ) can be used to control the difference of the moment | E µ g − E ν g | for g outside of F .
We address this problem by two steps: ﬁrst controlling functions in span F , and then functions in cl ( span F ) that is large enough to include
C b ( X ) for neural networks.
Controlling functions in span F .
We start with understanding how d F ( µ, ν ) can bound | E µ g − E ν g | for g ∈ span F .
This can be characterized by introducing a notion of norm on span F .
Proposition 2.6.
For each g ∈ span F that can be decomposed into g = (cid:80) ni
=1 w i
f
i + w 0 as we deﬁne in (3) , the F -variation norm ||
g || F , 1 of g is the inﬁmum of (cid:80) ni =1 | w i | among all possible decompositions of g , that is,
[Equation]
Then we have
[Equation] µ, ν ) , ∀ g ∈ span F .
Intuitively speaking,
|| g || F , 1 denotes the “minimum number” of functions in F needed to represent g .
As F becomes larger, || g || F , 1 decreases and d F ( µ, ν ) can better control | E µ g − E ν g | .
Precisely, if F ⊆ F (cid:48) then || g || F (cid:48) , 1 ≤ || g || F , 1 .
Therefore, although adding more neurons in F may not necessarily enlarge span F , it decreases ||
g || F , 1 and yields a stronger neural distance.
Controlling functions in cl ( span F ) .
A more critical question is how the neural distance d F ( µ, ν ) can also control the discrepancy E µ g − E ν g for functions outside of span F but inside cl ( span F ) .
The bound in this case is characterized by a notion of error decay function deﬁned as follows.
Proposition 2.7.
Given a function g , we say that g is approximated by F with error decay function (cid:15) ( r ) if for any r ≥ 0 , there exists an f r ∈ span F with || f r || F , 1 ≤ r such that || f − f r || ∞ ≤ (cid:15) ( r ) .
Therefore, g ∈ cl ( span F )
if and only if inf r ≥ 0 (cid:15) ( r ) = 0 .
We have
[Equation]
In particular, if (cid:15) ( r ) =
O ( r − κ ) for some κ > 0 , then | E µ g − E ν g | =
O ( d F ( µ, ν ) κκ +1 ) .
It requires further efforts to derive the error decay function for speciﬁc F and g .
For example,
Proposition 6 of Bach (2017) allows us to derive the decay rate of approximating bounded Lipschitz functions with rectiﬁed neurons, yielding a bound between BL distance and neural distance.
Corollary 2.8.
Let X be the unit ball of R d under norm ||·|| q for some q ∈ [2 , ∞ ) , that is, X = { x ∈
R d : || x || q ≤ 1 } .
Consider F consisting of a single rectiﬁed neuron F = { max( v (cid:62) [ x ; 1] , 0) α : v ∈
R d +1 ,
||
v || p
= 1 } where α ∈ N , 1 p + 1 q = 1 .
Then we have
[Equation]
where ˜ O denotes the big-O notation ignoring the logarithm factor.
The result in (6) shows that d F ( µ, ν ) gives an increasingly weaker bound when the dimension d increases.
This is expected because we approximate a non-parametric set with a parametric one.
Likelihood and KL divergence.
Maximum likelihood has been the predominant approach in statistical learning, and testing likelihood forms a standard criterion for testing unsupervised models.
The recent advances in deep unsupervised learning, however, make it questionable whether likelihood is the right objective for training and evaluation (e.g., Theis et al., 2015).
For example, some recent empirical studies (e.g., Danihelka et al., 2017; Grover et al., 2017) showed a counter-intuitive phenomenon that both the testing and training likelihood (assuming generators with valid densities are used) tend to decrease, instead of increase, as the GAN loss is minimized.
A hypothesis for explaining this is that the neural distances used in GANs are too weak to control the KL divergence properly.
Therefore, from the theoretical perspective, it is desirable to understand under what conditions (even if it is a very strong one), the neural distance can be strong enough to control KL divergence.
This can be done by the following simple result.
Proposition 2.9.
Assume µ
and ν have positive density functions
ρ µ ( x ) and ρ ν ( x ) , respectively.
Then
[Equation]
If log( ρ µ /ρ ν ) ∈ span F , then
[Equation]
If log( ρ µ /ρ ν ) ∈ cl ( span F ) with an error decay function (cid:15) ( r ) =
O ( r − κ ) , then
[Equation]
This result shows that we require that the density ratio log( ρ µ /ρ ν ) should exist and behave nicely in span F or cl ( span F ) in order to bound KL divergence with d F ( µ, ν ) .
If either µ or ν is an empirical measure, the bound is vacuum since KL( µ, ν ) + KL( µ, ν ) equals inﬁnite, while d F ( µ, ν ) remains ﬁnite once F is bounded, i.e., || f || ∞ ≤ ∆ < ∞ for all f ∈ F .
Obviously, this strong condition is hard to satisfy in practice, because practical data distributions and generators in GANs often have no densities or at least highly peaky densities.
We draw more discussions in Corollary 3.5.
G ENERALIZATION PROPERTY OF GAN S
Section 2 suggests that it is better to use larger discriminator set F in order to obtain stronger neural distance.
However, why do regularization techniques, which effectively shrink the discriminator set, help GAN training in practice?
The answer has to do with the fact that we observe the true model µ only through an i.i.d.
sample of size m (whose empirical measure is denoted by ˆ µ m ), and hence can only optimize the empirical loss d F (ˆ µ m , ν ) , instead of the exact loss d F ( µ, ν ) .
Therefore, generalization bounds are required to control the exact loss d F ( µ, ν ) when we can only minimize its empirical version d F (ˆ µ m , ν ) .
Speciﬁcally, let G be a class of generators that may or may not include the unknown true distribution µ .
Assume ν m minimizes the GAN loss d F (ˆ µ m , ν ) up to an (cid:15) ( (cid:15) ≥ 0 ) accuracy, that is,
[Equation]
We are interested in bounding the difference between ν m and the unknown µ under certain evaluation metric.
Depending on what we care about, we may be interested in the generalization error in terms of the neural distance d F ( µ, ν m ) , or other standard quantities of interest such as BL distance d BL ( µ, ν m ) and KL divergence KL( µ, ν m ) or the testing likelihood.
In this section, we adapt the standard Rademacher complexity argument to establish generalization bounds for GANs.
We show that the discriminator set F should be small enough to be generalizable, striking a tradeoff with the other requirement that it should be large enough to be discriminative.
We ﬁrst present the generalization bound under neural distance, which purely depends on the Rademacher complexity of the discriminator set F and is independent of the generator set G .
Then using the results in Section (2.1), we discuss the generalization bounds under other standard metrics, like BL distance and KL divergence.
3. G ENERALIZATION UNDER NEURAL DISTANCE
Using the standard derivation and the optimality condition (9), we have (see Appendix D)
[Equation]
This reduces the problem to bounding the discrepancy d F ( µ, ˆ µ m ) :
= sup f ∈F | E µ [ f ] − E [Equation] m [ f ] | between the true model µ and its empirical version ˆ µ m .
This can be achieved by the uniform concentration bounds developed in statistical learning theory (e.g., Vapnik & Vapnik, 1998) and empirical process (e.g., Van de Geer, 2000).
In particular, the concentration property related to sup f ∈F | E µ [ f ] − E ˆ µ m [ f ] | can be characterized by the Rademacher complexity of F (w.r.t.
measure µ ), deﬁned as
[Equation]
where the expectation is taken w.r.t.
X i ∼ µ , and Rademacher random variable τ i : prob( τ i = 1) =
prob( τ i = − 1) = 1 / 2 .
Intuitively, R ( µ ) m ( F ) characterizes the ability of overﬁtting with pure random labels using functions in F and hence relates to the generalization bounds.
Standard results in learning theory show that
[Equation]
where ∆ = sup f ∈F || f || ∞ .
Combining this with (10), we obtain the following result.
Theorem 3.1.
Assume that all discriminators are bounded by ∆ , i.e., (cid:107) f (cid:107) ∞ ≤ ∆ for any f ∈ F .
Let ˆ µ m be an empirical measure of an i.i.d.
sample of size m drawn from µ .
Assume ν m ∈ G satisﬁes d F (ˆ µ m , ν m ) ≤ inf ν ∈G d F (ˆ µ m , ν ) + (cid:15) .
Then with probability at least 1 − δ , we have
[Equation]
where R ( µ ) m ( F ) is the Rademacher complexity of F deﬁned in (11) .
We obtain nearly the same generalization bound for neural f -divergence in Theorem B.3.
Theorem 3.1 relates the generalization error of GANs to the Rademacher complexity of the discriminator set F .
The smaller the discriminator set F is, the more generalizable the result is.
Therefore, the choice of F should strike a subtle balance between the generalizability and the discriminative power: F should be large enough to make d F ( µ, ν ) discriminative as we discuss in Section 2.1, and simultaneously should be small enough to have a small generalization error in (12).
It turns out parametric neural discriminators strike a good balance for this purpose, given that it is both discriminative as we show in Section 2.1, and give small generalization bound as we show in the following.
Corollary 3.2.
Let X be the unit ball of R d under norm || · || 2 , that is, X = { x ∈ R d : || x
|| 2 ≤ 1 } .
Assume that F is neural networks with a single rectiﬁed linear unit (ReLU)
F = { max( v (cid:62) [ x ; 1] , 0): v ∈ R d +1 ,
||
v || 2 = 1 } .
Then with probability at least 1 − δ ,
[Equation]
(13) and d
[Equation]
where C = 4 √ 2 + 4 (cid:112) log(1 /δ ) and ˜ O denotes the big-O notation ignoring the logarithm factor.
Note that the three terms in Eqn.
(13) take into account the modeling error ( inf v ∈G d F ( µ, ν ) ), sample complexity and generalization error ( C/ √ m ), and optimization error ( (cid:15) ), respectively.
Assuming zero modeling error and optimization error, we have (1) d F ( µ, ν m ) =
O ( m − 1 / 2 ) , which achieves the typical parametric convergence rate; (2) d BL ( µ, ν m ) =
˜ O ( m − 1 d +3 ) , which becomes slower as the dimension d increases.
This decrease is because of the non-parametric nature of BL distance, instead of learning algorithm.
As we show in Appendix A, we obtain a similar rate of d BL ( µ, ν m ) =
O ( m − 1 d ) , even if we directly use BL distance as the learning objective.
Similar results can be obtained for general parametric discriminator class as follows.
Corollary 3.3.
Under the condition of Theorem 3.1, we further assume that (1) F = { f θ : θ ∈ Θ ⊂ [ − 1 , 1] p } is a parametric function class with p parameters in a bounded set Θ and that (2) every f θ is L -Lipschitz continuous with respect to the parameters θ , i.e., (cid:107) f θ − f θ (cid:48) (cid:107) ∞
≤ L (cid:107) θ − θ (cid:48) (cid:107) 2 .
Then with probability at least 1 − δ , we have
[Equation]
where C = 16 √ 2 πpL + 2∆ (cid:112) 2 log(1 /δ ) .
This result can be easily applied to neural discriminators, since neural networks f θ ( x ) are generally
Lipschitz w.r.t.
the parameter θ , once the input domain X is bounded.
For neural discriminators, we also apply the bound on the Rademacher complexity of DNNs recently derived in Bartlett et al.
(2017), which gives a sharper bound than that in Corollary 3.3; see Appendix A.1.
With the basic result in Theorem 3.1, we can also discuss the learning bounds of GANs with choices of non-parametric discriminators.
Making use of Rademacher complexity of bounded sets in a
RKHS (e.g., Lemma 22 in Bartlett & Mendelson (2003)), we give the learning bound of MMDbased GANs (Li et al., 2015; Dziugaite et al., 2015) as follows.
We present the results for Wasserstein distance and total variance distance in Appendix A.2, and highlight the advantages of using parametric neural discriminators.
Corollary 3.4.
Under the condition of Theorem 3.1, we further assume that F = { f ∈ H : (cid:107) f (cid:107) H ≤ 1 } where H is a RKHS whose positive deﬁnite kernel k ( x, x (cid:48) )
satisﬁes k ( x, x ) ≤ C k < + ∞ for all x ∈ X .
Then with probability at least 1 − δ ,
[Equation]
where C = 2 (cid:16) 2 + (cid:112) 2 log(1 /δ )
(cid:17) √ C k .
Remark 3.1 (Comparisons with results in Arora et al.
Arora et al.
(2017) also discussed the generalization properties of GANs under a similar framework.
In particular, they developed bounds of form | d F ( µ, ν ) − d F (ˆ µ m , ˆ ν m ) | where ˆ µ m and
ˆ ν m are empirical versions of the target distribution µ
and ν with sample size m .
Our framework is similar, but considers bounding the quantity d F ( µ, ν m ) − inf ν ∈G d F ( µ, ν ) , which is of more direct interest.
In fact, our Eqn.
(10) shows that our generalization error can be bounded by the generalization error studied in Arora et al.
Another difference is that we adapt the Rademacher complexity argument to derive the bound, while Arora et al.
(2017) made use of the (cid:15) -net argument.
Bounding the KL divergence and testing likelihood.
The above results depend on the evaluation metric we use, which is d F ( µ, ν ) or d BL ( µ, ν ) .
If we are interested in evaluating the model using even stronger metrics, such as KL divergence or equivalently testing likelihood, then the generator set G enters the scene in a more subtle way, in that a larger generator set G should be companioned with a larger discriminator set F in order to provide meaningful bounds on KL divergence.
This is illustrated in the following result obtained by combining Theorem 3.1 and Proposition 2.9.
Corollary 3.5.
Assume both the true µ and all the generators ν ∈ G have positive densities ρ µ and ρ ν , respectively.
Assume F consists of bounded functions with ∆ := sup f ∈F || f || ∞ < ∞ .
Further, assume the discriminator set F is compatible with the generator set G in the sense that log( ρ ν /ρ µ ) ∈ span F , ∀ ν ∈ G , with a compatible coefﬁcient deﬁned as
[Equation]
Then
[Equation]
[Equation] Different from the earlier bounds, the bound in (17) depends on the compatibility coefﬁcient Λ F , G that casts a more interesting trade-off on the choice of the generator set G : the generator set G should be small and have well-behaved density functions to ensure a small Λ F , G , while should be large enough to have a small modeling error inf ν ∈G (cid:112) KL( µ, ν ) .
Related, the discriminator set should be large enough to include all density ratios log( ρ µ /ρ ν ) in a ball of radius Λ F , G of span F , and should also be small to have a low Rademacher complexity R ( µ ) m ( F ) .
Obviously, one can also extend Corollary 3.5 using (8) in Proposition 2.7, to allow log( ρ µ /ρ ν ) ∈ cl ( span F ) in which case the compatibility of G and F should be mainly characterized by the error decay function (cid:15) ( r ) .
KL( µ, ν m ) =
E µ [log p µ ] − E µ [log p ν m ] is the difference between the testing likelihood E µ [log p ν m ] of estimated model ν m and the optimal testing likelihood E µ [log p µ ] .
Therefore, Corollary 3.
also provides a bound for testing likelihood.
Unfortunately, the condition in Corollary 3.5 is rather strong, in that it requires that both the true distribution µ and the generators ν have positive densities and that the log-density ratio log( ρ µ /ρ ν ) be well-behaved.
In practical applications of computer vision, however, both µ and ν tend to concentrate on local regions or sub-manifolds of X , with very peaky densities, or even no valid densities; this causes the compatibility coefﬁcient Λ F , G very large, or inﬁnite, making the bound in (17) loose or vacuum.
This provides a potential explanation for some of the recent empirical ﬁndings (e.g., Danihelka et al., 2017; Grover et al., 2017) that the negative testing likelihood is uncorrelated with the GAN loss functions, or even increases during the GAN training progress.
The underlying reason here is that the neural distance is not strong enough to provide meaningful bound for KL divergence.
See Appendix F for an illustration using toy examples.
[-If φ is strictly convex, then a standard derivation based on Jensen’s inequality shows that φ -divergence is nonnegative and discriminative: d φ ( µ || ν ) ≥ φ (1) = 
-]
[-We have i) 
-]
R ELATED WORK
There is a surge of research interest in GANs; however, most of the work has been empirical in nature.
There has been some theoretical literature on understanding GANs, including the discrimination and generalization properties of GANs.
The discriminative power of GANs is typically justiﬁed by assuming that the discriminator set F has enough capacity.
For example, Goodfellow et al.
(2014) assumes that F contains the optimal discriminator p data ( x ) p data ( x ) + p g ( x ) .
Similar capacity assumptions have been made in nearly all other GANs to prove their discriminative power; see, e.g., Zhao et al.
; Nowozin et al.
; Arjovsky et al.
However, discriminators are in practice taken as certain parametric function class, like neural networks, which violates these capacity assumptions.
The universal approximation property of neural networks is used to justify the discriminative power empirically.
In this work, we show that the GAN loss is discriminative if span F can approximate any continuous functions.
This condition is very weak and can be satisﬁed even when none of the discriminators is close to the optimal discriminator.
The MMD-based GANs (Li et al., 2015; Dziugaite et al., 2015; Li et al., 2017a) avoid the parametrization of discriminators by taking advantage of the close-form solution of the optimal discriminator in the non-parametric RKHS space.
Therefore, the capacity assumption is satisﬁed in
MMD-based GANs, and their discriminative power is easily justiﬁed.
Liu et al.
(2017) deﬁnes a notion of adversarial divergences that include a number of GAN objective functions.
They show that if the objective function is an adversarial divergence with some additional conditions, then using a restricted discriminator family has a moment-matching effect.
Our treatment of the neural divergence is directly inspired by them.
We refer to Remark B.1 for a detailed comparison.
Liu et al.
(2017) also shows that for objective functions that are strict adversarial divergence, convergence in the objective function implies weak convergence.
However, they do not provide a condition under which an adversarial divergence is strict.
A major contribution of our work is to ﬁll this gap, and to provide such a condition that is sufﬁcient and necessary.
Dziugaite et al.
(2015) studies generalization error, deﬁned as d F ( µ, ν m ) − inf ν ∈G d F ( µ, ν ) in our notation, for MMD-GAN in terms of fat-shattering dimension.
Moreover, Dziugaite et al.
(2015) obtains a generalization bound that incorporates the complexity of the hypothesis set G .
Although their overall error bound is still O ( m − 1 / 2 ) , their work shows the possibility to sharpen our Gindependent bound.
Arora et al.
(2017) studies the generalization properties of GANs through the quantity d F ( µ, ν ) − d F (ˆ µ m , ˆ ν m )
(in our notations).
The main difference between our work and
Arora et al.
(2017) is the deﬁnition of generalization error; see more discussions in Remark 3.1.
Moreover, Arora et al.
(2017) allows only polynomial number of samples from the generated distribution because the training algorithm should run in polynomial time.
We do not consider this issue because in this work we only study the statistical properties of the objective functions and do not touch the optimization method.
Finally, Arora et al.
(2017) shows that the GAN loss can approach its optimal value even if the generated distribution has very low support, and (Arora & Zhang, 2017) provides empirical evidence for this problem.
Our result is consistent with their results because our generalization error is measured by the neural distance/divergence.
Finally, there are some other lines of research on understanding GANs.
Li et al.
(2017b) studies the dynamics of GAN’s training and ﬁnds that: a GAN with an optimal discriminator provably converges, while a ﬁrst order approximation of the discriminator leads to unstable dynamics and mode collapse.
Lei et al.
(2017) studies WGAN and optimal transportation by convex geometry, and provides a close-form formula for the optimal transportation map.
Hu et al.
(2017) provides a new formulation of GANs and variational autoencoders (VAEs), and thus uniﬁes the most two popular methods to train deep generative models.
We’d like to mention other recent interesting research on GANs, e.g., (Guo et al., 2017; Sinn & Rawat, 2017;
Nock et al., 2017; Mescheder et al., 2017; Tolstikhin et al., 2017; Heusel et al., 2017).
C ONCLUSIONS
We studied the discrimination and generalization properties of GANs with parameterized discriminator class such as neural networks.
A neural distance is guaranteed to be discriminative whenever the linear span of its discriminator set is dense in the bounded continuous function space.
On the other hand, a neural divergence is discriminative whenever the linear span of features deﬁned by the last linear layer of its discriminators is dense in the bounded continuous function space.
We also provided generalization bounds for GANs in different evaluation metrics.
In terms of neural distance, our bounds show that generalization is guaranteed as long as the discriminator set is small enough, regardless of the size of the generator or hypothesis set.
This raises an interesting discriminationgeneralization balance in GANs.
Fortunately, several GAN methods in practice already choose their discriminator set at the sweet point, where both the discrimination and generalization hold.
Finally, our generalization bound in KL divergence provides an explanation on the counter-intuitive behaviors of testing likelihood in GAN training.
There are several directions that we would like to explore in the future.
First of all, in this paper, we do not talk about methods to compute the neural distance/divergence.
This is typically a non-concave maximization problem and is extremely difﬁcult to solve.
Many methods have been proposed to solve this kind of minimax problems, but both stable training methods and theoretical analysis of these algorithms are still missing.
Secondly, our generalization bound depends purely on the discriminator set.
It is possible to obtain sharper bounds by incorporating structural information from the generator set.
Finally, we would like to extend our analysis to conditional GANs (see, e.g., Mirza & Osindero (2014); Springenberg (2015); Chen et al.
; Odena et al.
(2016)), which have demonstrated impressive performance (Reed et al., 2016a;b; Zhang et al., 2017).
A G ENERALIZATION ERROR OF OTHER DISCRIMINATOR SETS F A. G ENERALIZATION BOUND FOR NEURAL DISCRIMINATORS
For neural discriminators, we can use the following bound on the Rademacher complexity of DNNs, which was recently proposed in Bartlett et al. Theorem A.1.
Let ﬁxed activation functions ( σ 1 , . . . , σ L ) and reference matrices ( M 1 , . . . , M L ) be given, where σ i is ρ i -Lipschitz and σ i (0) = 0 .
Let spectral norm bounds ( s 1 , . . . , s L ) and matrix (2,1) norm bounds ( b 1 , . . . , b L ) be given. Let F denote the discriminator set consisting of all choices of neural network f A :
[Equation]
where (cid:107) A (cid:107) σ := σ max ( A ) and (cid:107) A (cid:107) 2 , 1 := (cid:107) ( (cid:107) A : , 1 (cid:107) 2 , . . . , (cid:107) A : ,m (cid:107) 2 ) (cid:107) 1 are the matrix spectral norm and (2 , 1) norm, respectively, and
[Equation]
is the neural network associated with weight matrices ( A 1 , . . . , A L ) . Moreover, assume that each matrix in ( A 1 , . . . , A L ) has dimension at most W along each axis and deﬁne the spectral normalized complexity R as
[Equation]
Let data matrix X ∈ R m × d be given, where the m rows correspond to data points. When the sample size m ≥ 3 (cid:107) X (cid:107) F R , the empirical Rademacher complexity satisﬁes
[Equation]
where τ = ( τ 1 , . .
, τ m ) are the Rademacher random variables which are iid with Pr [ τ i = 1] = Pr [ τ i = − 1] = 1 / 2 , (cid:107) X (cid:107) F is the Frobenius norm of X . Proof.
The proof is the same with the proof of Lemma A.8 in Bartlett et al. When m ≥ 3 (cid:107) X (cid:107) F R , we use the optimal α = 3 (cid:107) X (cid:107) F R/ √ m to obtain the above result.
Combined with our Theorem 3.1, we obtain the following generalization bound for the neural discriminator set deﬁned in (18). Corollary A.2. Suppose that the discriminator set F nn is taken as (18) and that (cid:107) f (cid:107) ∞ ≤ ∆ for any f ∈ F nn .
Let data matrix X ∈ R m × d be the m data points that deﬁne the empirical distribution ˆ µ m . Then with probability at least 1 − δ , we have [Equation]
[Equation] where R is the spectral normalized complexity deﬁned in (19) and (cid:15) is the optimization error deﬁned in (9) .
Proof.
In the proof of Theorem 3.1, instead of using [Equation] we use [Equation]
to revise the generalization bound (12) as
[Equation]
Combining the revised bound with Equation (20), we conclude the proof. Compared to Corollary 3.3, the bound in (22) gets rid of the number of parameters p , which can be prohibitively large in practice. Moreover, Corollary A.2 can be directly applied to the spectral normalized GANs (Anonymous, 2018), and may give an explanation of the empirical success of the spectral normalization technique. A. G ENERALIZATION BOUNDS FOR NON - PARAMETRIC DISCRIMINATOR SETS
With the basic result in Theorem 3.1, we can also discuss the learning bounds of GANs with other choices of non-parametric discriminator sets F .
This allows us to highlight the advantages of using parametric neural discriminators.
For simplicity, we assume zero model error and optimization so that the bound is solely based on the generalization error d F ( µ, ˆ µ m ) between µ and its empirical version ˆ µ m .
Bounded Lipschitz distance, F = { f ∈ C ( X ) :
|| f || BL ≤ 1 } , which is equivalent to Wasserstein distance when X is compact.
When X is a convex bounded set in R d , we have R ( µ ) m ( F ) ≤ m − 1 /d for d > 2 (see Corollary 12 in Sriperumbudur et al.
(2009)), and hence d BL ( µ, ν ) =
O ( m − 1 /d ) .
This is comparable with Corollary 3.2.
This bound is tight.
Assume that µ is the uniform distribution on X .
A simple derivation (similar to Lemma 1 in Arora et al.
(2017)) shows that d F ( µ, ˆ µ m ) ≥ c (1 − m exp( − Ω( d ))) for some constant only depending on X .
Therefore, one must need at least m =
exp(Ω( d )) samples to reduce d F ( µ, ˆ µ m ) , and hence the generalization bound, to O ( (cid:15) ) .
Total variation (TV) distance, F = { f ∈ C ( X ) : (cid:107) f (cid:107) ≤ min { 1 , ∆ }} .
It is easy to verify that R ( µ ) m ( F ) = 2 .
Therefore, Eqn.
(12) cannot guarantee generalization even when we have inﬁnite number of samples, i.e., m → ∞ .
The estimate given in Eqn.
(12) is tight.
Assume that µ is the uniform distribution on X .
It is easy to see that d TV ( µ, ˆ µ m ) = [-
-]2 almost surely.
Therefore, ν m is close to ˆ µ m implies that it is order 1 away from µ , which means that generalization does not hold in this case.
With the statement that training with the TV distance does not generalize, we mean that training with TV distance does not generalize in TV distance.
More precisely, even if the training loss on empirical samples is very small, i.e., TV(ˆ µ m , ν m )
=
O ( (cid:15) ) , the TV distance to the unknown target distribution can be large, i.e., d TV ( µ, ν m )
=
O (1) .
However, this does not imply that training with
TV distance is useless, because it is possible that training with a stronger metric leads to asymptotic vanishing in a weaker metric.
For example, d TV (ˆ µ m , ν m )
=
O ( (cid:15) ) implies d F nn (ˆ µ m , ν m )
=
O ( (cid:15) ) , and thus a small d F nn ( µ, ν m ) .
Take the Wasserstein metric as another example, even though we only establish d W ( µ, ν m ) =
O ( m − 1 /d )
(assuming zero model error ( µ ∈ G ) and optimization (cid:15) = 0 ), it does not eliminate the possibility that the weaker neural distance has a faster convergence rate d F nn ( µ, ν m )
=
O ( m − 1 / 2 ) .
From the practical perspective, however, TV and Wasserstein distances are less clearly favorable than neural distance because the difﬁculty of calculating and optimizing them.
B N EURAL φ - DIVERGENCE
f -GAN is another broad family of GANs that are based on minimizing f -divergence (also called φ -divergence)
(Nowozin et al., 2016), which includes the original GAN by Goodfellow et al.
However, φ -divergence has substantially different properties from IPM (see e.g., Sriperumbudur et al.
and is not deﬁned as the intuitive moment matching form as IPM.
In this Appendix, we extend our analysis to φ -divergence by interpreting it as a form of penalized moment matching .
Similar to the case of IPM, we analyze the neural φ -divergence that restricts the discriminators to parametric function set F for practical computability, and establish its discrimination and generalization properties under mild conditions that practical f -GANs satisfy.
Assume that µ and ν are two distributions on X .
Given a convex, lower-semicontinuous univariate function φ that satisﬁes φ (1) = 0 , the related φ -divergence is d φ ( µ || ν ) =
E ν (cid:2) φ (cid:0) d µ d ν (cid:1)(cid:3) .
{+If φ is strictly convex, then a standard derivation based on Jensen’s inequality shows that φ -divergence is nonnegative and discriminative: d φ ( µ || ν ) ≥ φ (1) = 
+}
and the equality holds iff µ = ν .
Different choices of φ recover popular divergences as special cases.
For example, φ ( t ) = ( t − 1) 2 recovers Pearson χ 2 divergence, and φ ( t ) = ( u + 1) log(( u + 1) / 2) + u log u gives the Jensen-Shannon divergence used in the vanilla GAN Goodfellow et al.
B. D ISCRIMINATIVE POWER OF NEURAL φ - DIVERGENCE
In this work, we ﬁnd it helps to develop intuition by introducing another convex function ψ ( t ) :
= φ ( t + 1) , deﬁned by shifting the input variable of φ by +1 ; the φ -divergence becomes
[Equation]
where we should require that ψ (0) = 0 ; in right hand side of (23), we assume ρ µ and ρ ν are the density functions of µ and ν , respectively, under a base measure τ .
The key advantage of introducing ψ is that it gives a suggestive variational representation that can be viewed as a regularized moment matching.
Specially, assume ψ ∗ is the convex conjugate of ψ , that is, ψ ∗ ( t ) =
sup y { yt − ψ ( y ) } .
By standard derivation, we can show that
[Equation]
where A is the class of all functions f : X → dom ( ψ ∗ ) where dom ( ψ ∗
[Equation]
= { t : ψ ∗ ( t ) ∈ R } , and the equality holds if ϕ ∗ ( ρ µ ( x ) ρ ν ( x ) − 1) ∈
A where ϕ is the inverse function of ψ ∗(cid:48) .
In (24), the term Ψ ν,ψ ∗ [ f ] , as we show in Lemma B.1 in sequel, can be viewed as a type of complexity penalty on f that ensures the supreme is ﬁnite.
This is in contrast with the IPM d F ( µ, ν ) in which the complexity constraint is directly imposed using the function class F , instead of a regularization term.
Lemma B.1.
Assume ψ : R → R ∪{∞} is a convex, lower-semicontinuous function with conjugate ψ ∗ and ψ (0) = 0 .
The penalty Ψ ν,ψ ∗ [ f ] in (24) has the following properties
i) Ψ ν,ψ ∗ [ f ] is a convex functional of f , and Ψ ν,ψ ∗ [ f
≥
for
any f . ii)
There exists a constant b 0 ∈ R ∪ {∞} such that ψ ∗ ( b 0 ) = 0 .
Further, if ψ is strictly convex, then Ψ ν,ψ ∗ [ f ] = 0 implies f ( x ) =
b 0 almost surely under measure ν .
In practice, it is impossible to numerically optimize over the class of all functions in (24).
Instead, practical f -GANs restrict the optimization to a parametric set F of neural networks, yielding the following neural φ -divergence :
[Equation]
Note that this can be viewed as a generalization of the F -related IPM d F ( µ, ν ) by considering ψ ∗ = 0 .
However, the properties of the neural φ -divergence can be signiﬁcantly different from that of d F ( µ, ν ) .
For example, d φ, F ( µ || ν ) is not even guaranteed to be non-negative for arbitrary discriminator sets F because of the negative regularization term.
Fortunately, we can still establish the non-negativity and discriminative property of d φ, F ( µ || ν ) under certain weak conditions on F .
Moreover, the property that d F ( µ, ν ) = 0 implies moment matching on F , which is the key step to establish the discriminative power, is not necessarily true for neural divergence.
Fortunately, it turns out that d φ, F ( µ || ν ) = 0 implies moment matching on features deﬁned by the last linear layer of discriminators.
Theorem B.1.
Assume F includes the constant function b 0 ∈ R , which satisﬁes ψ ∗ ( b 0 ) = 0 as deﬁned in Lemma B.1.
{+We have i) 
+}
≤ d φ, F ( µ || ν ) ≤ d F ( µ, ν ) for ∀ µ, ν. As a result, [Equation] implies
[Equation]
In other words, moment matching on F is a sufﬁcient condition of zero neural φ -divergence.
ii)
Further, we assume F has the following form:
[Equation]
[Equation]
where F 0 is any function set, and α f 0 > 0 is positive number associated with each f 0 ∈ F 0 , and c 0 is a constant and σ : R → R is any function that satisﬁes σ ( c 0 )
=
b 0 and σ (cid:48) ( c 0 ) > 0 .
Here σ can be viewed as the output activation function of a deep neural network whose previous layers are speciﬁed by F 0 .
Assume ψ ∗ ( y ) is differentiable at y
= b 0 .
Then
[Equation] implies [Equation] In other words, moment matching on F 0 is a necessary condition of zero neural φ -divergence. iii) cl ( span F 0 ) ⊇ C b ( X ) is a sufﬁcient condition for d φ, F to be discriminative, i.e., d φ, F ( µ || ν ) = 0 implies µ = ν .
Condition (26) deﬁnes a commonly used structure of F that naturally satisﬁed by the f -GANs used in practice; in particular, the output activation function σ plays the role of ensuring the output of F respects the input domain of the convex function ψ ∗ .
For example, the vanilla GAN has ψ ∗ =
− log(1 − exp( t ))
− t with an input domain of ( −∞ , 0) , and activation function is taken to be σ ( t )
=
− log(1 + exp( − t )) .
See Table 2 of Nowozin et al.
(2016) for the list of output activation functions related to commonly used ψ .
Remark B.1.
Our results on neural φ -divergence can in general extended to the more uniﬁed framework of Liu et al.
(2017) in which divergences of form max f E ( x,y ) ∼ µ ⊗ ν [ f ( x, y )] are studied.
We choose to focus on φ -divergence because of its practical importance.
Our Theorem B.1 i) can be viewed as a special case of Theorem 4 of Liu et al.
and our Theorem B.1 ii) is related to
Theorem 5 of Liu et al.
However, Theorem 5 of Liu et al.
(2017) requires a rather counterintuitive condition, while our condition in Theorem B.1 ii) is clear and satisﬁed by all φ -divergence listed in Nowozin et al.
Similar to Theorem 2.5, under the conditions of Theorem B.1, we have similar results for neural φ -divergence.
Theorem B.2.
Using the same notions in Theorem B.1, assume that ( X, d X ) is a compact metric space and that cl ( span F 0 ) ⊇ C b ( X ) .
Then if lim n →∞ d φ, F ( µ || ν n ) = 0 , ν n converges to µ in the distribution sense.
Further, if there exists C > 0 such that F ⊂ { f ∈ C ( X ) : (cid:107) f
(cid:107) Lip ≤ C } , we have
[Equation]
Notice that we assume that ( X, d X ) be a compact metric space here for simplicity.
A non-compact result is available but its proof is messy and non-intuitive.
Proof.
The ﬁrst half is a direct application of Theorem B.1 and Theorem 10 in Liu et al.
For the second half, we have
[Equation]
where we use Theorem B.1 i) in the ﬁrst inequality and the Lipschitz condition of F in the second ineqaulity.
Since d W metrizes the weak convergence for compact X , we obtain d W ( µ, ν n ) → 0 and thus d φ, F ( µ || ν n ) → 0 .
B. G ENERALIZATION PROPERTIES OF NEURAL φ - DIVERGENCE
Similar to the case of neural distance, we can establish generalization bounds for neural φdivergence. Theorem B.3. Assume that (cid:107) f (cid:107) ≤ ∆ for any f ∈ F . ˆ µ m is an empirical distribution with m samples from µ , and ν m ∈ G satisﬁes d φ, F (ˆ µ m || ν m ) ≤ inf ν ∈G d φ, F (ˆ µ m || ν )
+ (cid:15) .
Then with probability at least 1 − δ , we have
[Equation]
where R ( µ ) m ( F ) is the Rademacher complexity of F .
With Theorem B.3, we obtain generalization bounds for difference choices of F , as we had in section 3.
We list those results as follows.
Corollary B.4.
Under the condition of Theorem B.3, we further assume that (1) F = F nn = { f θ : θ ∈ Θ ⊂ [ − 1 , 1] p } is a parametric function class with p parameters in a bounded set Θ and that (2) every f θ is L -Lipschitz continuous with respect to the parameters θ .
Then with probability at least 1 − δ , we have
[Equation]
where
[Equation]
=
[Equation]
Notice that the only difference between Corollary 3.3 and Corollary B.4 is that we change p to p + and L to √ L 2 + ∆ 2 in the constant C .
This comes from the fact that F is typically not even in the neural divergence case.
For example, the vanilla GAN takes σ ( t ) =
− log(1 + exp( − t ))
as the output activation function, and thus f ≤ 0 for all f ∈ F .
Proof.
By adding a new parameter w ∈ [ − 1 , 1] to functions in F , we deﬁne
[Equation]
where the new parameter vector ˜ θ :=
[ w, θ ] comes from ˜Θ :=
[ − 1 , 1] × Θ ⊂ [ − 1 , 1] p +1 .
Now that ˜ F is even and we have
[Equation]
We point out that ˜
f ˜ θ is √ L 2 + ∆ 2 -Lipschitz respect to its parameter ˜
θ due to the following derivation:
[Equation]
Therefore, we can apply the same proof of Corollary 3.3 here, and obtain that
[Equation]
where C = 8 √ 2 π ( p + 1) √ L 2 + ∆ 2 .
C P ROOF OF RESULTS IN SECTION
Proof of Theorem 2.2.
For the sufﬁcient part, the proof is standard and the same as that of the uniqueness of weak convergence.
We refer to Lemma 9.3.2 in Dudley (2002) for a complete proof.
For the necessary part, suppose that F ⊂ C b ( X ) is discriminative in P B ( X ) .
Assume that cl ( span ( F ∪ { 1 } )) is a strictly closed subspace of C b ( X ) .
Take g ∈ C b ( X ) \ cl ( span ( F )) and (cid:107) g (cid:107) ∞
= 1 .
By the Hahn-Banach theorem, there exists a bounded linear functional L : C ( X ) → R such that L ( f ) = 0 for any f ∈ cl ( span ( F ∪ { 1 } )) and L (cid:54) = 0 .
Thanks to the Riesz representation theorem for compact metric spaces, there exists a signed, regular Borel measure m ∈ M B ( X ) such that
[Equation]
Suppose m = µ − ν are the Hahn decomposition of m , where µ and ν are two nonnegative Borel measures.
Then we have L ( f ) =
(cid:82) µ f − (cid:82) ν
f
for any f ∈ C b ( X ) .
Thanks to L ( 1 ) = 0 , we have 0 < µ ( X ) =
ν ( X ) < ∞ .
We can assume that µ and ν are Borel probability measures.
(Otherwise, we can use the normalized nonzero linear functional L/µ ( X ) whose Hahn decomposition consists of two Borel probability measures.)
Since L ( f ) = 0 for any f ∈ cl ( span ( F )) , we have (cid:82) µ f = (cid:82) ν f for any f ∈ F .
Since F ⊂ C b ( X ) is discriminative, we have µ = ν and thus L = 0 , which leads to a contradiction.
Proof of Corollary 2.4.
Thanks to { λθ : λ ≥ 0 , θ ∈ Θ } = R d +1 , for any [ w, b ] ∈ R n +1 , there exists [ w 0 , b 0 ] ∈ Θ and λ > 0 such that
[Equation]
where we used σ ( u ) = max { u, 0 } α in the last step.
Therefore, we have span F nn ⊂ span { σ ( w (cid:62) x + b ): [ w, b ] ∈ R d +1 } .
Thanks to Theorem 2.3, we know that span F nn is dense in C b ( X ) .
Proof of Theorem 2.5.
Given a function g ∈ C b ( X ) , we say that g is approximated by F with error decay function (cid:15) ( r ) if for any r ≥ 0 , there exists f r ∈ span F with || f r || F , 1 ≤ r such that
|| f − f r || ∞ ≤ (cid:15) ( r ) .
Obviously, (cid:15) ( r ) is an non-increasing function w.r.t.
r .
Thanks to cl ( span F ) =
C b ( X ) , we have lim r →∞ (cid:15) ( r )
= 0 .
Now denote r n := d F ( µ, ν n ) − 1 / 2 and correspondingly f n := f r n .
We have
[Equation]
If lim n →∞ d F ( µ, ν n ) = 0 , we have lim n →∞ r n = ∞ .
Thanks to lim r →∞ (cid:15) ( r ) = 0 , we prove that lim n →∞ | E µ g − E ν n g | = 0 .
Since this holds true for any g ∈ C b ( X ) , we conclude that ν n weakly converges to µ .
If F ⊆ BL C ( X ) for some C > 0 , we have d F ( µ, ν ) ≤ Cd BL ( µ, ν ) for any µ, ν .
Because the bounded Lipschitz distance (also called FortetMourier distance) metrizes the weak convergence, we obtain that ν n (cid:42) µ implies d BL ( µ, ν n ) → 0 , and thus d F ( µ, ν n ) → 0 .
Proof of Proposition 2.6.
Let g = (cid:80) ni
=1 w i
f
i + w 0 .
Then we have
[Equation]
The result is obtain by taking inﬁmum over all possible w i .
Proof of Proposition 2.7.
For any r ≥ 0 , we have
[Equation]
Taking the inﬁmum on r > 0 on the right side gives the result.
Proof of Corollary 2.8.
Proposition 5 of Bach (2017) shows that for any bounded Lipschitz function g that satisﬁes
|| g
|| BL : =
max {|| g || ∞ , ||
g || Lip } ≤ η , we have (cid:15) ( r ) =
O ( η ( r/η ) − 1 / ( α +( d − 1) / 2) log( r/η )) .
Using Proposition 2.7, we get
[Equation]
The result follows BL( µ, ν )
= sup g {| E µ g − E ν g | : || g || BL ≤ 1 } .
D P ROOF OF RESULTS IN SECTION
Proof of Equation (10) Using the standard derivation and the optimality condition (9), we have
[Equation]
Therefore [Equation] we obtain
[Equation]
Combining with the deﬁnition (1)
[Equation]
we obtain
[Equation]
Proof of Theorem 3.1.
Consider the function
[Equation]
Since f takes values in [ − ∆ , ∆] , changing X i to another independent copy X (cid:48)
i can change h by no more than 2∆ /m .
McDiarmid’s inequality implies that with probability at least 1 − δ ,
[Equation]
Standard argument on Rademacher complexity gives
[Equation]
Combining the two estimates above and Eqn.
(10), we conclude the proof.
Proof of Corollary 3.2.
Part of the proof is from Proposition 7 in Bach (2017).
More accurately, the discriminator set we use here is
[Equation]
for a ﬁx n ∈ N .
Since (cid:107) x (cid:107) 2 ≤ 1 and (cid:107) v (cid:107) 2 ≤ 1 , it is easy to see that (cid:107) f (cid:107) ∞ ≤ √ 2 for all f ∈ F .
We want to estimate R ( µ ) m ( F ) and then use Theorem 3.1 to prove the result.
First, it’s easy to verify that
[Equation]
[Table]
[Equation]
[Table]
[Equation]
Then we have
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation] [Table] [Equation] where we use the 1-Lipschitz property of max( x, 0) and Talagrand’s contraction lemma (Ledoux & Talagrand, 2013) in the inequality step.
From Kakade et al.
(2009), we get the Rademacher complexity of linear functions
[Equation]
Therefore, we obtain
[Equation]
Combined with (cid:107) f (cid:107) ∞ ≤ √ 2 and Theorem 3.1, we ﬁnish the proof.
Proof of Corollary 3.3.
We need to derive a bound for the Rademacher complexity (11).
Since F is even ( f ∈ F implies − f ∈ F ), we have
[Equation]
For ﬁxed { x i } mi =1 , let’s consider X θ = 1 L √ m (cid:80) mi
=
τ i f θ ( x i ) .
First of all, { X θ : θ ∈ Θ } is a sub-Gaussian process with respect to the Eulidean distance on Θ , i.e., for all θ, θ (
cid:48) ∈ Θ and all λ > E
[Equation]
(cid:19) .
This is a standard result and can be derived by the Hoeffding’s lemma.
Secondly, we have the following bound for the (cid:15) -cover
number of Θ :
[Equation]
This bound is from the following simple construction.
Consider a uniform grid with grid size 2 (cid:15)/ √ p .
The balls with centers as the grid points and with radius (cid:15) cover the unit cubic on R p , i.e., [ − 1 , 1] p .
The number of balls in this construction is ( √ p/(cid:15) ) p .
Finally, by applying Dudley’s entropy integral, we have
[Equation]
Combing (29) and (30) and taking τ = log( √ p/(cid:15) ) , we obtain
[Equation]
Notice that Eqn.
(31) holds true for arbitrary samples { x i } mi =1 , and thus we conclude that
[Equation]
where C = 8 √ 2 πpL .
Proof of Corollary 3.4.
Lemma 22 in Bartlett & Mendelson (2003) shows that if sup x ∈ X k ( x, x ) ≤
C k ≤ + ∞ , we have R ( µ ) m ( F ) ≤ 2 (cid:112)
C k /m for any µ ∈ P B ( X ) .
Also, note that f ( x ) ≤ ||
f || H (cid:112) k
( x, x ) ≤ ||
f || H √ C k .
Combined with Theorem 3.1, we conclude the proof.
Proof of Corollary 3.5.
Use Proposition 3.1 and note that KL( µ, ν m ) ≤ Λ F , G d F ( µ, ν ) and d F ( µ, ν ) ≤ ∆TV( µ, ν ) ≤ ∆ (cid:112) 2KL( µ, ν ) by Pinsker’s inequality.
E P ROOF OF RESULTS IN SECTION B
Proof of Lemma B.1.
i)
It is obvious that Ψ ν,ψ ∗ [ f ] is convex given that f ∗ is convex.
By the convex conjugate, we have ψ ( t ) =
sup y (cid:8) ty − ψ ∗ ( y ) (cid:9) .
Take t = 0 and note that ψ (0) = 0 , then we have ψ ∗ ( y ) ≥ 0 , ∀ y.
This proves Ψ ν,ψ ∗ [ f ] ≥ 0 .
ii)
If ψ is strictly convex, then ψ ∗ is also strictly convex.
This implies there exists at most a single value b 0 such that ψ ∗ ( c ) = 0 .
Given that ψ ∗ ( y ) ≥ 0 for ∀ y , we arrive that E x ∼ ν
[ ψ ∗ ( f ( x ))]
=implies ψ ∗ ( f ( x ))
=
almost surely under x ∼ ν , which then implies f ( x ) =
b 0 almost surely.
Proof of Theorem B.1.
i) because b 0 ∈ F and ψ ∗ ( b 0 )
= 0 , we have d φ, F ( µ || ν ) ≥ E µ [ b 0 ] − E ν
[ b 0 ] − Ψ ν,ψ ∗ [ b 0 ]
= 0 .
Because Ψ ν,ψ ∗ [ f ] ≥ 0 , we obtain d φ, F ( µ || ν ) ≤
d F ( µ, ν ) by comparing (25) with d F ( µ, ν ) = sup f ∈F { E µ f − E ν f } . ii), note
that d ψ, F ( µ || ν ) = 0 implies E µ [ f ] − E ν
[ f ] ≤ Ψ ν,ψ ∗ [ f ] , ∀ f ∈ F .
Therefore,
E µ [ σ ( αf 0 + c 0 )]
[Equation]
This implies that
[Equation]
By the differentiability assumptions,
[Equation]
where we used the fact that ψ ∗ ( σ ( c 0 ))
= ψ ∗ ( b 0 )
= 0 and ψ ∗(cid:48)
( b 0 ) =
because b 0 is a differentiable minimum point of ψ ∗ .
Taking the limit of α → 0 on both sides of (32), we get
[Equation]
Because σ (cid:48) ( c 0 ) > 0 by assumption, this implies E x ∼ µ [ f 0 ( x )] − E x ∼ ν
[ f 0 ( x )] .
The same argument applies to − f 0 , and we thus we ﬁnally obtain E x ∼ µ [ f 0 ( x )]
=
E x ∼ ν [ f 0 ( x )] .
iii)
Combining Theorem 2.2 and the last point, we directly get the result.
F I NCONSISTENCY BETWEEN GAN’ S LOSS AND TESTING LIKELIHOOD
In this section, we will test our analysis of the consistency of GAN objective and likelihood objective on two toy datasets, e.g., a 2D Gaussian dataset and a 2D 8-Gaussian mixture dataset.
F. A 2D G AUSSIAN EXAMPLE
The underlying ground-truth distribution is a 2D Gaussian with mean (0 . 5 , − 0 . 5) and covariance matrix(cid:20)
[Equation]
15(cid:21) .
We take 10 5 samples for training, and 1000 samples for testing.
For a 2D Gaussian distribution, we use the following generator
[Equation] [Equation]
) where z = (cid:20) zz(cid:21) is a standard 2D normal random vector, and l ∈ R , s = (cid:20) ss(cid:21) ∈ R 2 and b = (cid:20) bb(cid:21) ∈
R 2 are trainable parameters in the generator.
We train the generative model by WGAN with weight clipping.
In the ﬁrst experiment, the discriminator set is a neural network with one hidden layer and 500 hidden neurons, i.e.,
F
[Equation] [Equation]
i
[Equation]
Motivated by Corollary 3.5, in the second experiment, we take the discriminators to be the logdensity ratio between two Gaussian distributions, which is the quadratic polynomials:
[Equation]
We plot their results in Figure 1.
We can see that both discriminators behave well: The training loss (the neural distance) converge to zero, and the testing log likelihood increases monotonically during the training.
However, the quadratic polynomial discriminators F quad yields higher log testing likelihood and better generative model at the convergence.
This is expected because Corollary 3.
guarantees that the log testing likelihood is bounded by the GAN loss (up to a constant), while it is not true for F nn .
We can also maximize the likelihood (MLE) on the training dataset to train the model, and we show its result in Figure 3.
We can see that both MLE and Q-GAN (refers to WGAN with the quadratic discriminator F quad ) yield similar results.
However, directly maximizing the likelihood converges much faster than the WGAN in this example.
In this simple Gaussian examples, the WGAN loss and the testing log likelihood is consistent.
We indeed observe that by carefully choosing the discriminator set (as suggested in Corollary 3.5), the testing likelihood can be simultaneously optimized as we optimize the GAN objective.
F. A N EXAMPLE OF 2D 8-G AUSSIAN MIXTURE
The underlying ground truth distribution is a 2D Gaussian mixture with 8 Gaussians and with equal weights.
Their centers distributed equally on the circle centered at the origin and with radius √ 2 , and their standard deviations are all 0.01414.
We take 10 5 samples as training dataset, andsamples as testing dataset.
We show one batch (256) of training dataset and the testing dataset in
Figure 4.
Note that that the density of the ground-truth distribution is highly singular.
We still use Eqn.
(33) as the generator for a single Gaussian component.
Our generator assume that there are 8 Gaussian components and they have equal weights, and thus our generator does not have any modeling error.
The training parameters are eight sets of scaling and biasing parameters in
Eqn. (33), each for one Gaussian component.
We ﬁrst training the model by WGAN with clipping.
We use an MLP with 4 hidden layers and relu activations as the discriminator set.
We show the result in Figure 5.
We can see that the generator’s samples are nearly indistinguishable from the real samples.
However, the GAN loss and the log likelihood are not consistent.
In the initial stage of training, both the negative GAN loss and likelihood are increasing.
As the training goes on, the generator’s density gets more and more singular, the likelihood behaves erratically in the latter stage of training.
Although the negative
GAN loss is still increasing, the log likelihood oscillates a lot, and in fact over half of time the log likelihood is −∞ .
We show the generated samples at intermediate steps in Figure 6, and we indeed see that the likelihood start to oscillate inviolately when the generator’s distribution gets singular.
This inconsistency between GAN loss and likelihood is observed by other works.
The reason for this consistency is that the neural discriminators are not a good approximation of the singular density ratios.
We also trained the model by maximizing likelihood on the training dataset.
We show the result in
Figure 7.
We can see that the maximal likelihood training got stuck in a local minimal, and fail to exactly recover all 8 components.
The log likelihood on training and testing dataset are consistent as expected.
Although the log likelihood ( ≈ 2 . 7 ) obtained by maximizing likelihood is higher than that ( ≈ 2 . 0 ) obtained by WGAN training, its generator is obviously worse than what we obtained in
WGAN training.
The reason for this is that the negative log-likelihood loss has many local minima, and maximizing likelihood is easy to get trapped in a local minimum.
The FlowGAN (Grover et al., 2017) proposed to combine the WGAN loss and the log likelihood to solve the inconsistency problem.
We showed the FlowGAN result on this dataset in Figure 8.
We can see that training by FlowGAN indeed makes the training loss and likelihood consistent.
However,
FlowGAN got stuck in a local minimum as maximizing likelihood did, which is not desirable.