I NCREMENTAL TRAINING OF MULTI - GENERATIVE ADVERSARIAL NETWORKS
Anonymous authors
Paper under double-blind review
A BSTRACT
Generative neural networks map a standard, probability distribution to a complex high-dimensional distribution, which represents the real world data set.
However, a determinate input distribution as well as a speciﬁc architecture of neural networks may impose limitations on capturing the diversity in the high dimensional target space.
To resolve this difﬁculty, we propose a training framework that greedily produce a series of generative adversarial networks that incrementally capture the diversity of the target space.
We show theoretically and empirically that our training algorithm converges to the theoretically optimal distribution, the projection of the real distribution onto the convex hull of the network’s distribution space.
I NTRODUCTION
Generative Adversarial Nets (GAN) Goodfellow et al.
(2014) is a framework of estimating generative models.
The main idea Goodfellow (2017) is to train two target network models simultaneously, in which one, called the generator, aims to generate samples that resemble those from the data distribution, while the other, called the discriminator, aims to distinguish the samples by the generator from the real data.
Naturally, this type of training framework admits a nice interpretation as a twoperson zero-sum game and interesting game theoretical properties, such as uniqueness of the optimal solution, have been derived Goodfellow et al.
It is further proved that such adversarial process minimizes certain divergences, such as Shannon divergence, between the generated distribution and the data distribution.
Simply put, the goal of training a GAN is to search for a distribution in the range of the generator that best approximates the data distribution.
The range is often deﬁned by the input latent variable z and its speciﬁc architecture, i.e., Π = { G ( z , θ ) , θ ∈ Θ } .
When the range is general enough, one could possibly ﬁnd the real data distribution.
However, in practice, the range is usually insufﬁcient to perfectly describe the real data, which is typically of high dimension.
As a result, what we search for is in fact the I-projection Csisz ´ ar & Shields (2004) of the real data distribution on Π .
Consider two cases:
The range of the generator Π is convex (see ﬁgure 1(a)), or it is not convex but the projection of the real data distribution on Π ’s convex hull (C ONV Π ) is in Π (see ﬁgure 1(b)).
The range of the generator is non-convex and the projection of the real data distribution in C ONV Π is not in the range Π (see ﬁgure 1(c)).
[Figure]
In case 1, one can ﬁnd the optimal distribution in Π to approximate real data set in C ONV Π .
But in case 2, using standard GANs with a single generator, one can only ﬁnd the distribution in Π that is nearest to the projection.
It then makes sense to train multiple generators and use a convex combination of them to better approximate the data distribution (than using a single generator in the non-convex case (see ﬁgure 1(c))).
The above argument is based on the assumption that one could achieve global optimality by training, while this is not the case in general.
When reaching a local optimal distribution, in order to improve performance, do we need to add more generators and restart training?
In this paper, we put forward a sequential training procedure that adds generators one by one to improve the performance, without retraining the previously added generators.
Our contributions can be summarized as follows.
• We derive an objective function tailored for such a incremental training process.
The objective function takes both the real data distribution and the pre-learned distribution into consideration.
We show that with this new objective, we actually maximize marginal contribution when adding a new generator.
We also put forward an incremental training algorithm based on the new objective function.
• We prove that our algorithm always converges to the projection of real data distribution to the convex hull of the ranges of generators, which is the optimal solution with multiple generators.
This property continues to hold in online settings where target distribution changes dynamically.
• Our experiments show that our algorithm can overcome the local optimal issue mentioned above.
We perform experiments on a synthetic dataset as well as two real world datasets, e.g., CelebA and MNIST
and conclude that our algorithm could improve the mixture distribution even in the case where the range is not sufﬁcient enough.
• Experiments also show that, compared with previous methods, our algorithm is fast and stable in reducing the divergence between mixture distribution and the real data.
1. R ELATED WORKS
Recently, there have been intensive researches on improving the performance of generative adversarial neural networks.
Two lines of works are closely related to our paper.
They focus mainly on improving the discriminator and the generator respectively
. 1.1. I MPROVING THE DISCRIMINATOR
The Unrolled GAN introduced by Metz et al.
(2016) improves the discriminator by unrolling optimizing the objective during training, which stabilizes training and effectively reduces the mode collapse.
D2GAN proposed by Nguyen et al.
(2017) utilizes two discriminators to minimize
the
KL-divergence and the reverse KL-divergence respectively.
It treats different modes more fairly, and thus avoids mode collapse.
DFM introduced by Warde-Farley & Bengio (2016) brings a Denoising AutoEncoder (DAE) into the generator’s objective to minimize the reconstruction error in order to get more information from the target manifold.
Mroueh et al.
(2017) proposed McGan based on mean and covariance feature matching to stabilize the training of GANs.
Finally, WGAN introduced by Arjovsky et al.
(2017) employs the Wasserstein distance, which is a more appropriate measure of performance, and achieves more stable performance.
These works are different from ours since they focus on the discriminator by measuring the divergence between the generated data and the real data more precisely.
However, our work ﬁxes the discriminator and tries to enrich the expressiveness of the generator by combining multiple generators.
1.1. I MPROVING THE GENERATOR
Wang et al.
(2016) proposes two methods to improve the training process.
The ﬁrst is selfensembling GANs, which assembles the generators from different epochs to stabilize training.
The other is Cascade GAN, where the authors train new generator using the data points with highest values from the discriminator.
These two methods are heuristic ways to improve training, but with no theoretical guarantee.
Hoang et al.
(2017) and Ghosh et al.
(2017) proposed the methods called MGAN and multi-agent
GANs respectively.
The former introduces a classiﬁer into the discriminator to catch different modes, while the later employ a new component into the generators’ objective to promote diversity.
Arora et al.
(2017) introduces a new metric on distributions and proposes a MIX + GAN to search for an equilibrium.
But all these methods need to train the multiple generators simultaneously, and none of them can deal with the case when the training process reaches a local optima.
Also, these models lack ﬂexibility, in the sense that when one tries to change the number of generators, all the generators need to be retrained.
Another closely related work is Tolstikhin et al.
(2017), in which the authors propose a method called
AdaGAN, which is based on a robust reweighting scheme on the data set inspired from boosting.
The idea is that the new generators should focus more on the previous bad training data.
But AdaGAN and other boosting-like algorithms are based on the assumption that one generator could catch some modes precisely, which may not be reasonable since the generator always learns to generate the average samples among the real data set in order to obtain low divergence, especially when the generator’s range is under condition of Figure 1(c).
In Section 5, we compare our algorithm with
AdaGAN with different dataset.
P RELIMINARIES
A GAN Goodfellow et al.
(2014) takes samples (a.k.a.
latent variables z ) from a simple and standard distribution as its input and generates samples in a high dimensional space to approximate the target distribution.
This is done by training a generative neural network and an auxiliary discriminative neural network alternatively.
An f-GAN Nowozin et al.
(2016) generalizes the adversarial training as minimizing the f-divergence between the real data distribution and the generated distribution, deﬁned as D f ( p (cid:107) q )
= (cid:82) x q ( x ) f (cid:16) p ( x ) q ( x ) (cid:17) d x . A GAN is a special f-GAN that minimizes the JensenShannon divergence.
The general objective function of an f-GAN can be deﬁned as follows:
[Equation]
Here f ∗ is the conjugate function of f in f-divergence; T represents a neural network regarded as the corresponding discriminator; ﬁnally, θ and ξ denote the parameters of the generator and the discriminator, respectively.
2. G ENERATOR G ROUP
The adversarial training method proposed by Goodfellow et al.
(2014) is playing a minimax game between the generator and the discriminator.
Such a method can be caught in local optima and thus is undesirable (e.g., mode collapse).
In this paper we propose a novel framework to train multiple generators sequentially: We maintain a group of generators (empty at the beginning) as well as their corresponding weights, then add new generators into the group one by one and rebalance the weights.
In particular, only the newly added generator at each step is trained.
The purpose here is to augment the capacity of the group of generators and mitigate the local optima issue.
Deﬁne the distribution range of a generator as Π = { p | p =
G ( z , θ ) , θ ∈ Θ } , i.e., the set of distributions that the generator can produce with different parameter θ .
The distribution range is determined by the distribution of input z and the architecture of the generative network.
Deﬁne a generator group as G = { G 1 , G 2 , , . . . , G n } , where G i is the generator added in step i .
We associate each generator with a weight ω i > 0 .
Then the mixed distribution of the group is: p pre ( x ) = 1Φ n (cid:80) ni =1 ω
i p i ( x ) , where p i ( x ) =
G i ( z ) and Φ n = (cid:80) ni
=1 ω
i is the sum of weights.
When a new generator G n +1 joins in the group G , the group becomes G (cid:48)
=
G ∪ { G n +1 } and the mixed distribution becomes p now ( x ) =
Φ n + n +1 (cid:88) i =
ω i p
i ( x )
[Equation]
A N INCREMENTAL TRAINING FRAMEWORK
In this section, we describe how we use a generator group to improve the performance and tackle the local optima issue mentioned previously.
To train such a generator group, we propose an incremental training algorithm (algorithm 1) adding generators to the group sequentially.
In algorithm 1, we use
Algorithm 1 Incremental Training Algorithm
Predetermined: p real , p z , G i ( z ) , i ∈ { 1 , 2 , . . .
, N } ω i , i ∈ { 1 , 2 , . . .
, N + 1 }
Input p real , p z , ω i , i > 0 .
Output generator group G .
Initialize i ← 1 .
repeat
Build and initialize generator G i using the same network structure.
Set target distribution for G i to be p target =
Φ i · p real − (cid:80) j<i ω
j p j ω
i .
Train G i to minimize D ( p target , p i ) .
i ← i + 1 .
until Convergence
D ( · , · ) to denote the “distance” between two distributions, which can be any divergence (e.g., fdivergence or Wasserstein distance) or a general norm.
The key step in algorithm 1 is the choice of the target distribution for training G i .
Ideally, if
D ( p target , p i ) = 0 , then we have p i = p target .
In this case, p real =
1Φ i (cid:80) ij =1 ω j p j and after adding G i , the generator group G can perfectly produce the desired distribution p real .
However, in general, we have D ( p target , p i ) (cid:54) = 0 and our algorithm proceeds in a greedy fashion, i.e., it always maximizes the marginal contribution of G i to the generator group G .
We devote the rest of this section to proving the above statement.
3. M ARGINAL CONTRIBUTION MAXIMIZATION AT EACH ROUND
In algorithm 1, we use different loss functions for each generators.
The marginal contribution of the ( N + 1) -th generator is as follows when we adopt f-divergence as the distance measure:
[Equation]
To get a better approximation to the real distribution, we ﬁx the existing generators in the group and tune the parameters of the new generator to minimize the distance between the new group and the real distribution.
In fact, this is equivalent to maximizing the marginal contribution of the new generator V ( G N +1 ) by selecting G N +1 = arg max G (cid:48)
N +1 V ( G (cid:48) N +1 ) .
Formally,
Proposition 1.
For any distribution p real and any existing generator group G =
( G 1 , G 2 , . . .
, G N ) , the optimal target distribution p ∗
N +1 for a new generator G N +1 to join the group G is
[Equation]
+ , where [ · ] + = max {· , 0 } .
To show this, we ﬁrst introduce the χ 2 -divergence.
Deﬁnition 1 ( χ 2 -divergence) .
The χ 2 -divergence between distribution p and q is: D χ 2 ( p || q ) =
(cid:82) x ( p ( x ) − q ( x ))
2 q ( x ) d x .
Note that χ 2 -divergence is a special case of the f-divergence: D χ 2 ( p || q ) =
D f ( p || q ) , when f ( u ) = ( u − 1 )
In fact, with some mild assumptions on f , the f -divergence is well-approximated by χ 2 -divergence when p and q are close.
The following lemma can be obtained via Taylor expansion Csisz´ar &
Shields (2004).
Lemma 1.
For any f-divergence with f ( u ) , if f ( u ) is twice differentiable at u = 1 and f (cid:48)(cid:48) (1) > 0 , then for any q and p close to q we have: D f ( p || q ) ∼ f (cid:48)(cid:48)
(1)2 D χ 2 ( p || q )
[Equation]
Proof of proposition 1.
We rewrite the objective function equation 1 for χ 2 -divergence:
V ( G N +
[Equation]
Input: p real , p z , η , ω N +
and G i ( · , θ i ) , ω
i , i ∈ { 1 , 2 , . . .
, N } repeat
Sample { x jreal } from p real
Sample { z j } from p z for i = 1 to N do
Obtain samples by G i ( z j , θ i ) as { x jG i } in Group G end for
Generate samples { x jgen } by G N +
Update ξ t +1 =
ξ t + η ∇ ξ F ( θ tN +1 , ξ t )
Update θ t +1 N +
= θ tN +1 − η ∇ θ F ( θ tN +1 , ξ t ) until Convergence
Output: G N +1 ( · , θ N +1 )
Based on the former deﬁnition, we obtain V ( G N +1 )
= (cid:82) x ω N +1 · [ − ω N +1 · p 2 N +1 + A · p N +1 + B ] (Φ N +1 ) 2 p real d x , where
A = 2(Φ N +1 · p real − Φ N · p pre ) and B = (Φ N +1 + Φ N ) p 2 pre − 2Φ N +1 · p pre · p real .
To maximize the quadratic function V ( G N +1 ) , we have p ∗
N +
=
[ A/ 2 ω N +1 ] + , which concludes the proof.
3. A LGORITHMS TO T RAIN G i
According to algorithm 1, in each round, a new generator G N +1 is added and the loss function is set to be D ( p target , p N +1 ) .
Therefore, when training each generator G i , the target distribution only depends on the real distribution and the previous generators in G .
In particular, both of them are already known (ﬁgure 2).
To minimize D ( p target , p G N +1 ) , we conduct adversarial training by using an auxiliary discriminator
T:
[Equation]
where by the linearity of expectation:
[Equation]
Based on these, we propose an incremental training algorithm for G N +1 as algorithm 2.
T HEORETICAL A NALYSES
In this section, we show that although our framework which trains each generator in a greedy way, the output distribution of the generator group will always converge.
Furthermore, the converged distribution is the closest one to the target distribution among the set of all possible distributions that a group of generators can produce (i.e., the optimal one within the distribution range of the group of generators).
Recall our notation that the distribution range of a generator is Π .
By taking a convex combination of multiple generators (with the same architecture), the set of all possible output distributions becomes the convex hull of Π : C ONV Π
= { p ( x ) | p ( x ) =
1Φ N (cid:80) Ni =1 ω
i p i ( x ) } , where ω i > 0 is the weight of the i -th generator, p i ( x ) =
G i ( z ) , 1 ≤ i ≤ N , and G i represent the generative neural networks with the same architecture but different parameters.
One can consider the parameters ω i Φ N as the probability of choosing the i -th generator as the ﬁnal output.
[Figure]
W a ss e r s t a i n d
i s t a n c e b e t w ee n D a t a a n d M i x t u r e
Adagan Incremental_gan
Original_Gan
4. C ONVERGENCE A NALYSIS
Our algorithm greedily optimizes each G N +1 to minimize D ( p target , p G N +1 ) .
By the Pinsker’s inequality, the total variation distance between p target and p G N +1 is upper bounded by (cid:112)
D ( p target , p G N +1 ) / 2 and we can easily extend it to χ 2 -divergence by D KL ( p || q ) ≤ D χ 2 ( p || q ) + 0 .
In other words, while greedily optimizing each G N +1 , the distance between p target and p G N +1 is also approximately minimized.
Hence it is reasonable to assume that for each G N +1 , its distance to p target is approximately minimized with some tolerance (cid:15) ≥ 0 , i.e., (cid:107) p G N +1 − p target (cid:107) ≤ inf (cid:107) p − p target (cid:107) + (cid:15) .
Under such an assumption, our algorithm approximately converges to the the optimal distribution in C ONV Π :
Proposition 2.
For any Π that is connected and bounded, algorithm 2 approximately converges to the optimal distribution within the closure of the convex hull C ONV Π of Π .
To simplify the argument, we ﬁx each ω
i to be 1 and embed the discrete probability distributions into a Hilbert space.
In this case, each G N +1 approximately minimizes the distance to p target =
( N + 1) p real −
[Equation]
Ni =1 p G i can be formalized as:
[Equation]
and our algorithm approximately converges to the optimal distribution in C ONV Π if as N → ∞ ,
[Equation]
Then proposition 2 is implied by the following lemma.
Lemma 2.
Consider a connected and bounded subset Π of a Hilbert space H and any target ρ ∈ H .
Let { p ∗ n } ∞ n =1 be a sequence of points in Π such that for ρ target =
( n + 1) ρ − nT n ,
[Equation]
where T n = 1 n (cid:80) ni =1 p ∗ n and (cid:15) ≥ 0 is a constant.
Then for any δ > 0 , there exists N > 0 such that
[Equation]
Corollary 1.
With the ﬁnite change of target distribution, algorithm 2 can converge to the new optimal distribution within C ONV Π .
Due to the space limit, we send the proof to the appendix.
Based on corollary 1, regardless the change of target distribution, as long as it is an ﬁnite variation, algorithm 2 can converge to the projection of new target distribution.
Due to the sequential nature and the above theoretical guarantee, our algorithm naturally generalizes the dynamic online settings.
E XPERIMENTS
We test our algorithm on a synthesized Gaussian distribution dataset and two well-known real world datasets: CelebA and MNIST, which are the complex high dimensional real world distributions.
[Figure]
a n
[Figure]
f l o g
[ D χ 2 ( p r
[Figure]
a l || p n
[Figure]
[Figure] [Figure]
We design the experiment to test our sequential training algorithm.
The main purpose is not to demonstrate high quality results, e.g., high deﬁnition pictures, but to show that our algorithm can search for an appropriate distribution that signiﬁcantly improved the performance of mixture distributions as the number of generators increase, especially when the generator’s range is rather limited.
In all experiments, we use the Adam optimizer Kingma & Ba (2014) with learning rate of 5 × 10 − 5 , and β 1 = 0 .
5 , β 2 = 0 .
Finally, we set weights ω
i = 1 for convenience.
Metric.
As the method mentioned in Nowozin et al.
(2016), when we ﬁx the generator, we can train an auxiliary neural network to maximize the derived lower bound to measure the divergence between the generated data and the real data, i.e., D f ( P || Q ) .
Based on these theories, we import an auxiliary neural network to measure the performance of different methods.
The architecture of the auxiliary neural network is the same as the discriminator used in each experiment.
We train it forepoches, which is enough to measure the differences.
Then we take the mean value of the lastiterations as the ﬁnal output.
Synthesized data.
In this part, we design some experiments in R 2 space.
The dataset is sampled from 8 independent two-dimensional Gaussian distributions (i.e., the blue points in ﬁgure 3).
The model is previously proposed by Metz et al.
Firstly, following the experiment designed in Metz et al.
(2016) and Hoang et al.
(2017), we choose the latent variable z in a high dimensional space as z ∼ N (0 , I 256 ) , i.e., the distribution projection is likely to be in the generator’s range, which meets the condition of ﬁgure 1(a) or ﬁgure 1(b).
In ﬁgure 3, the blue points are the real data while the corresponding colored number represents the data points generated by each generator respectively.
As ﬁgure 3 shows, we train up to 4 generators to approximate the data distribution and the ﬁrst generator tends to catch the data with high probability around the centre of each Gaussian.
As the number of generators increasing, generated data tends to cover the data away from the centre in order to be complementary to previous mixture distributions and thus gains a considerable marginal proﬁt.
These results demonstrate our marginal maximization algorithm can promote the mixture distributions to cover the data with low probabilities.
Secondly, we reduce the dimension of z to 1, i.e., z ∼ N (0 , 1) and simplify the corresponding network architecture, so that the condition of ﬁgure 1(c) is likely met.
In this part, we compare our algorithm with the state of the art incremental training method AdaGAN
Tolstikhin et al.
(2017) and the baseline method Orignal GAN.
1 We train up to 20 generators in each experiment with the same starting generator (i.e., identical ﬁrst generator for each method), then measure the D χ 2 ( p || q ) between real distribution and the generated mixed distribution.
We repeat the experiment fortimes to reduce the effect of random noises.
Figure 5 and ﬁgure 6 illustrate the average and the best performance with different numbers of generators, respectively.
According to the result, our algorithm approaches to p real faster than the other two methods and achieves the best performance among all three methods.
In summary, our algorithm outperforms the other two both in terms of the speed of converging to the real distribution and the quality of the ﬁnal result under the case of ﬁgure 1(c).
MNIST.
In this experiment, we run our algorithm on the MNIST dataset LeCun et al.
We design this experiment to measure the performance of our algorithm for a more complex data distribution.
We choose the latent variable as z ∼ N (0 , 1) to limit the corresponding generator range and
[Figure]
use the Wasserstein distance to measure the gap between mixed distribution and the real dataset
[Figure] Then we train up to 22 generators to approximate the real distribution and the result is showed in ﬁgure 4.
Our algorithm outperforms the Original GAN but is inferior to the AdaGAN with the ﬁrst 8 generators.
As the number of generators increases, AdaGAN seems to run into a bottleneck while both our algorithm and the Original GAN gradually approximate to the real data distribution.
In order to analysis the convergence, we further train up to 100 generators with both our algorithm and the original GAN.
In ﬁgure 7, the horizontal dash lines represent the minimum value of the
Wasserstein distance for the two method respectively.
As showed in ﬁgure, the distance gradually decrease with the number of generators increasing, and our algorithm is much faster to reduce the distance and can even obtain a better performance.
More over, as we tends to investigate the property of each generator in the generators’ group, we measure the Wasserstein distance between distribution G ( z , θ ) and the real data, the experiment result is showed in ﬁgure 8 and the dash lines in ﬁgure 8 represent the mean value of the 100 generators.
Interestingly, the result shows that, in each generator, original GAN tends to search a distribution in the distribution range that is closer to the real data distribution, while our algorithm is searching for a distribution that is complementary to the generators’ group (i.e., a huge decrease in the mixture condition in ﬁgure 7) even if its own performance is poor (i.e., a high distance in the in ﬁgure 8).
CelebA. We also conduct our experiment on the CelebA dataset Liu et al.
As shown in ﬁgure 1, we start with an identical generator and train up to 6 generators using different methods.
The measured Wasserstein distance between mixed distribution of Group G and the real-data distribution is showed in ﬁgure 1.
In this experiment, we use the training method WGAN-GP proposed by
Gulrajani et al.
The experiment results indicates that our algorithm outperforms the other two methods after the second generator.
It demonstrates the potential of our algorithm applying to real world datasets.
[Table]
F UTURE WORK
We intend to further investigate the problem of how to accelerate the convergence via optimal generator weights, i.e., ω i .
Moreover, by measuring the relativity between the current group G and the dynamically changing target distribution, we intend to apply our algorithm to online learning.
A M ISSING P ROOFS A. P ROOF OF L EMMA
Proof of lemma 2.
Without loss of generality, we can assume ρ = 0 , since otherwise we can add an offset − ρ to the Hilbert space H .
Denote d n = n (cid:107) T n (cid:107) = (cid:107) (cid:80) ni
=1 p ∗ i (cid:107) .
Then according to equation (cid:63) ,
[Equation]
Let ˆ p ∈ Π be the point that minimizes the distance between − nT n and its projection ˆ p ⊥ on line − nT n , i.e., ˆ p = arg min p ∈ Π (cid:107) p ⊥ + nT n (cid:107) , where p ⊥ = (cid:104) p,nT n (cid:105) H (cid:107) nT n (cid:107) 2 · nT n .
Then we can further bound d n +1 by (cid:107) ˆ p + nT n (cid:107) ,
[Equation]
On one hand, since T n ∈ C ONV Π , ˆ p ⊥ can be seen as the projection of ˆ p on T n as well, hence (cid:107) ˆ p − ˆ p ⊥ (cid:107) ≤ (cid:107) ˆ p − T n (cid:107) .
Note that Π is bounded, therefore (cid:107) ˆ p − T n (cid:107) is bounded by the diameter of Π , denoted as d ≥ 0 .
On the other hand, suppose that p ∗ is the point closest to ρ = 0 within C ONV Π , i.e., p ∗ = arg min p ∈ C ONV Π (cid:107) p (cid:107) .
Since Π is connected, therefore the projection of Π on − nT n is the same with the projection of C ONV Π .
Hence,
[Equation]
In other words,
[Equation]
If (cid:107) p ∗ (cid:107) =
inf p ∈ C ONV Π (cid:107) p (cid:107) > 0 , then we have d n = n (cid:107) T n (cid:107) ≥ n (cid:107) p ∗ (cid:107) .
Hence
[Equation]
Then for any δ > 0 , let N be sufﬁciantly large such that ln NN ≤ 2 δ(cid:15) (cid:107) p ∗ (cid:107) /d 2 , we have (cid:107) T n −
[Equation]
(cid:107) =
d n /n ≤ inf
[Equation]
Π (cid:107) p − ρ (cid:107) + (cid:15) + δ(cid:15).
Otherwise, (cid:107) p ∗ (cid:107) = 0 and d n +1 ≤ (cid:15) + (cid:112) d 2 + d 2 n .
Note that the upper bound is increasing in d n , hence ∀ n > 0 , d n ≤ d ∗ n for d ∗ n deﬁned as follows:
[Equation]
For which, we can easily prove by induction that d ∗ n ≤ n(cid:15) + √ nd .
Therefore (cid:107) T n (cid:107) =
d n ≤ (cid:15) + d/ √ n , which immediately completes the proof.
A. P ROOF OF C OROLLARY
Proof of corollary 1.
Without loss of generality, we assume the optimal projection of target distribution is changed from ρ to ρ (cid:48) after n 0 iterations, where n 0 ∈ R + is a constant value.
Then we can derive
[Equation]
, where n ∈ R + is the training iteration after change.
Thenbased on lemma 2, we obtain lim n → + ∞ (cid:107) T n − ρ (cid:48) (cid:107) ≤ inf p ∈ Π (cid:107) p − ρ (cid:48) (cid:107) + (cid:15).
On the other side, for a speciﬁc n 0 , (cid:107) T n 0 − ρ (cid:48) (cid:107) ≤ (cid:107) T n 0 − ρ (cid:107) + (cid:107) ρ − ρ (cid:48) (cid:107) is a bounded value if the variation of target distribution is limited.
Finally, we can obtain lim n → + ∞ (cid:107) T n + n 0 − ρ (cid:48) (cid:107) ≤ inf p ∈ Π (cid:107) p − ρ (cid:48) (cid:107) + (cid:15), which concludes the proof.
A. T HE GAP BETWEEN KL DISTANCE AND χ 2 DISTANCE
Lemma 3.
At worst case, D KL ( p || q ) ≤ D χ 2 ( p || q ) + 0 . 42 .
Proof.
According to f-divergence, we have D f ( p || q ) =
(cid:82) x q ( x ) · f (cid:16) p ( x ) q ( x ) (cid:17)
d x .
For KL-divergence and χ 2 -divergence, the corresponding f ( t ) are f KL ( t ) =
t log( t ) and f χ 2 ( t ) =
( t − 1) 2 respectively.
Import an auxiliary function as:
[Equation]
Then based on the monotonicity of F(t), we have F ( t ) min ≥ − 0 .
[Equation]
which conclude the proof.