T OWARDS M ORE T HEORETICALLY -G ROUNDED P ARTICLE O PTIMIZATION S AMPLING FOR D EEP L EARNING
Anonymous authors
Paper under double-blind review
A BSTRACT
Many deep-learning based methods such as Bayesian deep learning (DL) and deep reinforcement learning (RL) have heavily relied on the ability of a model being able to efﬁciently explore via Bayesian sampling.
Particle-optimization sampling (POS) is a recently developed technique to generate high-quality samples from a target distribution by iteratively updating a set of interactive particles, with a representative algorithm the Stein variational gradient descent (SVGD).
Though obtaining signiﬁcant empirical success, the non-asymptotic convergence behavior of SVGD remains unknown.
In this paper, we generalize POS to a stochasticity setting by injecting random noise in particle updates, called stochastic particle-optimization sampling (SPOS).
Notably, for the ﬁrst time, we develop non-asymptotic convergence theory for the SPOS framework, characterizing convergence of a sample approximation w.r.t.
the number of particles and iterations under both convex- and noncovexenergy-function settings.
Interestingly, we provide theoretical understanding of a pitfall of SVGD that can be avoided in the proposed SPOS framework, i.e. , particles tend to collapse to a local mode in SVGD under some particular conditions.
Our theory is based on the analysis of nonlinear stochastic differential equations, which serves as an extension and a complementary development to the asymptotic convergence theory for SVGD such as (Liu, 2017).
With such theoretical guarantees,
SPOS can be safely and effectively applied on both Bayesian DL and deep RL tasks.
Extensive results demonstrate the effectiveness of our proposed framework.
I NTRODUCTION
Recent development of deep-learning techniques have required the ability of an algorithm to efﬁciently explore some particular space ( e.g. , parameter space) via Bayesian sampling, due to high model complexity of modern deep models.
Meanwhile, recent years have seen the development of scalable sampling methods such as stochastic gradient MCMC (SG-MCMC) (Welling & Teh, 2011;
Chen et al., 2014; Ding et al., 2014; Chen et al., 2015) and Stein variational gradient descent (SVGD)
(Liu & Wang, 2016) to facilitate big-data analysis.
SG-MCMC is a family of scalable Bayesian sampling algorithms relying on Itó diffusions, linear stochastic differential equations (SDE) with appropriately designed coefﬁcients such that the corresponding stationary distributions match a target distribution.
One potential issue is that samples might be highly correlated partially due to the nature of Markov chains, leading to low sample efﬁciency, an undesirable property of SG-MCMC.
SVGD, on the other hand, belongs to the family of particle-based sampling methods, which optimize a set of interacting particles to minimize some distance metric between the target distribution and the distribution induced by the particles.
By optimizing the distance measure, one maintains an optimal set of particles at each time.
Recent development of SVGD has shown that the underlying mathematical principle is based on a family of nonlinear SDEs, in the sense that coefﬁcients of the
SDE depend on the current density of the particles.
Though achieving numerous practical successes (Liu & Wang, 2016; Feng et al., 2017; Liu et al., 2017; Haarnoja et al., 2017; Liu & Zhu, 2018), little theory has been developed to understand the convergence property of the algorithm.
A recent theoretical development has interpreted SVGD as a special type of gradient ﬂow in the space of probability measures, and developed theory to disclose its asymptotic convergence behavior (Liu, 2017).
Recently, Chen et al.
uniﬁed SG-MCMC and SVGD
by proposing a particle-optimization sampling framework that interprets both as Wasserstein gradient ﬂows (WGFs).
Generally speaking, a
WGF is a partial differential equation (PDE) deﬁned on the space of probability measures, describing the evolution of a density function over time.
In (Chen et al., 2018), the authors deﬁne a WGF by combining the corresponding Fokker-Planck equations for both SG-MCMC and SVGD, and solve it with deterministic particle approximations.
However, due to the diffusion nature, deterministicparticle approximation leads to a hard-to-control error, challenging for theoretical analysis.
Based on the uniﬁed framework in (Chen et al., 2018), we propose to solve WGFs with stochastic particle approximation, leading to stochastic particle-optimization sampling (SPOS).
The idea is instead of solving the WGF with an uncontrollable deterministic approximation for a diffusion term, we solve it stochastically by injecting random noise to the particle-update equations.
Remarkably, for the ﬁrst time, we develop nonasymptotic convergence theory for the family of SPOS algorithms, considering both convex- and nonconvex-energy functions.
Different from existing theory for SGMCMC based algorithms (Teh et al., 2016; Vollmer et al., 2016; Chen et al., 2015; Raginsky et al., 2017; Zhang et al., 2017; Xu et al., 2018), our development relies on the theory of nonlinear SDEs , which is more involved and less explored in literature.
Particularly, we adopt tools from granular media equations (Malrieu, 2003; Cattiaux et al., 2008) to develop nonasymptotic error bounds in terms of the 1-Wasserstein distance.
Please refer to Section M in the Supplementary Material (SM) for detailed distinctions of our work to existing work.
Within our theoretical framework, we provide a formal theoretical understanding of a pitfall of SVGD, i.e., particles tend to collapse to one point under some particular conditions; whereas this can be avoided in the proposed SPOS framework due to the injected random noise.
Our theory is general enough to be applied for various deep-learning tasks, including Bayesian deep learning and Bayesian exploration in reinforcement learning.
Our extensive experimental results well suggest advantages of our framework compared to existing methods.
P RELIMINARIES
This section introduces necessary preliminaries, along with notations used in this paper.
For the sake of clarity, through out the paper, we use bold letters to denote variables in continuous-time diffusions and model deﬁnitions, e.g. , θ t in (1) deﬁned below (indexed by “time” t ).
By contrast, normal unbold letters are used to denote parameters in algorithms (discrete solutions of continuous-time diffusions), e.g. , θ ( i ) k in (3) below (indexed by “iteration” k ).
For conciseness, all the proofs as well as some extra experimental results are presented in the SM.
Discussion on the complexity of our algorithm is also included in Section L of the SM.
2. S TOCHASTIC GRADIENT MCMC
In Bayesian sampling, we aim to generate random samples from a posterior distribution p ( θ |X )
∝ p ( X| θ ) p ( θ ) , where θ ∈ R d represents the model parameter with a prior distribution p ( θ ) , and X (cid:44) { x q } Nq =1 represents the observed data with likelihood p ( X| θ ) =
(cid:81) q p ( x q | θ ) .
Deﬁne the potential energy as: U ( θ ) (cid:44) − log p ( X| θ ) − log p ( θ ) =
− (cid:80) N q =1 (cid:0) log p ( x q | θ )
+ 1 N log p ( θ ) (cid:1) (cid:44) (cid:80) N q =1 U q ( θ ) .
SG-MCMC algorithms belong to diffusion-based sampling methods, where a continuous-time diffusion process is designed such that its stationary distribution matches the target posterior distribution.
The diffusion process is driven by a speciﬁc stochastic differential equation (SDE).
For example, in stochastic gradient Langevin dynamic (SGLD), the SDE endows the following form:
[Equation]
where F ( θ ) (cid:44) ∇ θ U ( θ )
= (cid:80) Nq =1 ∇ θ U q ( θ ) (cid:44) (cid:80) Nq =1 F q ( θ ) ; t is the time index; β > 0 is the temperature parameter; and W t ∈ R
d is a d -dimensional Brownian motion.
More instances of SDEs corresponding to other SG-MCMC algorithms can be deﬁned by specifying different forms of F and potentially other diffusion coefﬁcients.
We focus on SGLD and (1) in this paper, and refer interested readers to (Ma et al., 2015) for more detailed description of general SG-MCMC algorithms.
Denote the probability density function of θ t in (1) as ν t , and a · b (cid:44) a (cid:62) b for two vectors a and b .
It is known that ν t is characterized by the following Fokker-Planck (FP) equation (Risken, 1989):
[Equation]
where the stationary distribution ν ∞ equals to our target distribution p ( θ |X )
according to Chiang & Hwang (1987).
SGLD generates samples from p ( θ |X ) by numerically solving the SDE (1).
For scalability, it replaces F ( θ k ) in each iteration with an unbiased evaluation by randomly sampling a subset of X , i.e. F ( θ k ) is approximated by: G k (cid:44) NB k (cid:80) q ∈I k F q ( θ k ) , where I k is a random subset of [1 , 2 , · · · , N ] with size B k in each iteration.
Based on the above settings, SGLD uses the
Euler method with stepsize h k to numerically solve (1) and obtains the update equation: θ k +
=
θ k − β − 1 G k h k + (cid:112) 2 β −
h k ξ k , ξ k ∼ N ( 0 , I ) .
2. S TEIN VARIATIONAL GRADIENT DESCENT
Different from SG-MCMC, SVGD is a deterministic particle-optimization algorithm that is able to generate samples from a target distribution.
In the algorithm, a set of particles interact with each other, driving them to high density locations in the parameter space while keeping them far away from each other with repulsive force.
The update equations of the particles follow the fastest descent direction of the KL-divergence between current empirical distribution of the particles and the target distribution, on an RKHS induced by a kernel function κ ( · , · ) (Liu & Wang, 2016).
Formally, Liu &
Wang (2016) derived the following updating rules for the particles { θ ( i ) k } Mi =1 at the k -th iteration with stepsize h k and G ( i )
k
(cid:44) NB k (cid:80) q ∈I k F q ( θ ( i ) k ) :
[Equation]
(3) where the ﬁrst term in the bracket encourages particles to locate on high density modes, and the second term serves as repulsive force that pushes away different particles.
Different from SG-MCMC, only particles at the current iteration, { θ ( i ) k } Mi =1 , are used to approximate the target distribution.
2. P ARTICLE - OPTIMIZATION BASED SAMPLING METHODS
SG-MCMC and SVGD, though look closely related, behave very differently in terms of algorithms, e.g. , stochastic and noninteractive versus deterministic and interactive particle updates.
Recently,
Chen et al.
(2018) proposed a deterministic particle-optimization framework that uniﬁes both SGMCMC and SVGD.
Speciﬁcally, the authors viewed both SG-MCMC and SVGD as Wasserstein gradient ﬂows (WGFs) on the space of probabilistic measures, and derived several deterministic particle-optimization techniques for particle evolutions, like what SVGD does.
For SG-MCMC, the
FP equation (2) for SGLD is a special type of WGFs.
Together with an interpretation of SVGD as a special case of the Vlasov equation in nonlinear PDE literature, Chen et al.
(2018) proposed a general form of PDE to characterize the evolution of the density for the model parameter θ , denoted as ν t at time t with ν ∞ matching our target (posterior) distribution, i.e. ,
[Equation]
where K is a function controlling the interaction of particles in the PDE system.
For example, in
SVGD, Chen et al.
(2018) showed that K and K ∗ ν t ( θ ) endow the following forms:
[Equation] [Equation]
where κ ( · , · ) is a kernel function such as the RBF kernel.
In the following, we introduce a new unary function K ( θ ) =
exp( − (cid:107) θ (cid:107) 2 η 2 ) , thus κ ( θ , θ (cid:48) ) can be rewritten as κ ( θ , θ (cid:48) )
=
K ( θ − θ (cid:48) ) .
Hence, (4) with K deﬁned in (5) can be equivalently rewritten as:
[Equation]
where Y is a random sample from ν t but independent of θ .
Importantly,
Proposition 1 (Chen et al.
The stationary distribution of (6) equals to our target distribution, which means ν ∞ ( θ )
= p ( θ |X ) .
Chen et al.
(2018) proposed to solve (4) numerically with deterministic particle-optimization algorithms such as the blob method.
Speciﬁcally, the continuous density ν t is approximated by a set of
M particles { θ ( i ) t } Mi =1 that evolve over time t , i.e. ν t ≈ 1 M (cid:80) Mi
=1 δ θ ( i ) t ( θ ) , where δ θ ( i ) t
( θ )
=
if θ = θ ( i ) t and 0 otherwise.
Note ∇ θ ν t in (4) is no longer a valid deﬁnition when adopting particle approximation for ν t .
Consequently, ∇ θ ν t needs nontrivial approximations, e.g. , by discrete gradient ﬂows or blob methods proposed in (Chen et al., 2018).
We omit the details here for simplicity.
S TOCHASTIC P ARTICLE -O PTIMIZATION S AMPLING A LGORITHMS
The deterministic particle-approximation methods proposed by Chen et al.
(2018) to approximately solve the WGF problem (4) introduce approximation errors for ∇ θ ν t that are hard to control analytically.
To overcome this problem, we propose to solve (4) stochastically to replace the ∇ θ ν t term with a Brownian motion.
Speciﬁcally, ﬁrst note that the term β − 1 ∇ θ · ∇ θ ν t is contributed from Brownian motion, i.e. , solving the SDE, d θ t = (cid:112) 2 β − 1 d W t , is equivalent to solving the corresponding FP equation: ∂ν t
= β − 1 ∇ θ · ∇ θ ν t .
Consequently, we decompose RHS of (4) into two parts: F 1 (cid:44) ∇ θ · (cid:0) ν t β − 1 F ( θ t )
+ ( K ∗ ν t )
ν t (cid:1) and F 2 (cid:44) β − 1 ∇ θ · ∇ θ ν t .
Our idea is to solve Fdeterministically under a PDE setting, and solve F 2 stochastically based on its corresponding SDE.
When adopting particle approximation for the density ν t , both solutions of F 1 and F 2 are represented in terms of particles { θ ( i ) t } .
Thus we can combine the solutions from the two parts directly to approximate the original exact solution of (4).
Similar to the results of SVGD in Section 3.3 in (Liu, 2017), we ﬁrst formally show in Theorem 2 that when approximating ν t with particles, i.e. , ν t ≈ 1 M (cid:80) Mi
=1 δ θ ( i ) t ( θ ) , the PDE can be transformed into a system of deterministic differential equations with interacting particles.
Theorem 2 When approximating ν t in (4) with particles { θ ( i ) t } i , the PDE ∂ t ν t = F 1 reduces to the following system of differential equations describing evolutions of the particles over time:
∀
i
[Equation]
On the other hand, by solving ∂ t ν t = F 2 stochastically with its equivalent SDE counterpart, we arrive at the following differential equation system, describing evolutions of the particles { θ ( i ) t } over time t : ∀
i
[Equation]
Our intuition is that if the particle evolution (8) can be solved exactly, the solution of (6) ν t will be wellapproximated by the particles { θ ( i ) t } Mi =1 .
In our theory, we show this intuition is actually true.
In practice, however, solving (8) is typically infeasible, and thus numerical methods are adopted.
Furthermore, in the case of big data, following SG-MCMC, F ( θ ( i ) k ) is typically replaced by a stochastic version G ( i )
k
(cid:44) NB k (cid:80) q ∈I k F q ( θ ( i ) k ) evaluated with a minibatch of data of size B k for computational feasibility.
Based on the Euler method (Chen et al., 2015) with a stepsize h k , (8) leads to the following updates for the particles at the k -th iteration
[Equation]
We called the algorithm with particle update equations (9) stochastic particle-optimization sampling (described in Algorithm 3), in the sense that particles are optimized stochastically with extra random
Gaussian noise.
Intuitively, the added Gaussian noise would enhance the ability of the algorithm to jump out of local modes, leading to better ergodic properties compared to standard SVGD.
This serves as one of our motivations to generalize SVGD to SPOS.
To illustrate the advantage of introducing the noise term, we compare SPOS and SVGD on sampling a difﬁcult multi-mode distribution, with the density function given in Section A of the SM.
The particles are initialized on a local mode close to zero.
Figure 1 plots the ﬁnal locations of the particles along with the true density, which shows that particles in SPOS are able to reach different modes, while they are all trapped at one mode in SVGD.
This pitfall of SVGD will be studied formally in Section 4.4.
N ON -A SYMPTOTIC C ONVERGENCE A NALYSIS : THE C ONVEX C ASE Algorithm
Input: Initial particles { θ ( i ) 0 } Mi =1 with θ ( i )∈
R
d , step size h k , batch size B k 1: for iteration k = 0,1,...
,T do 2:
Update θ ( i ) k
+1 with (9) for ∀ i .
3: end for
Output: { θ ( i ) T } Mi =
In this section, we develop nonasymptotic convergence theory for the proposed SPOS when the energy function U ( θ ) is convex.
The nonconvex case is discussed in Section 5.
We prove nonasymptotic convergence rates for
SPOS algorithm under the 1Wasserstein metric W 1 , a special case of p-Wasserstein metric deﬁned as W p ( µ, ν ) =
(cid:16) inf ζ ∈ Γ( µ,ν ) (cid:82) R d × R d (cid:107) X µ − X ν (cid:107) p dζ ( X µ , X ν ) (cid:17) 1 /p , where Γ( µ, ν ) is the set of joint distributions on R d × R d with marginal distribution µ and ν .
Note that SPOS reduces to SVGD when β → ∞ , thus our theory also sheds light on the convergence behavior of SVGD, where non-asymptotic theory is currently missing, despite the asymptotic theory developed recently (Liu, 2017;
Lu et al., 2018).
It is worth noting that part of our proofs are generalization of techniques for analyzing granular media equations in (Malrieu, 2003; Cattiaux et al., 2008).
4. B ASIC SETUP AND ASSUMPTIONS
Due to the exchangeability of the particle system { θ ( i ) t } Mi =1 in (8), if we initialize all the particles θ ( i ) t with the same distribution ρ 0 , they would endow the same distribution for each time t .
We denote the distribution of each θ ( i ) t as ρ t .
Similar arguments hold for the particle system { θ ( i ) k } Mi =1 in (9), and thus we denote the distribution of each θ ( i ) k as µ k .
To this end, our analysis aims at bounding
W 1 ( µ T , ν ∞ ) since ν ∞ equals to our target distribution p ( θ |X )
according to Proposition 1.
Before proceeding to our theoretical results, we ﬁrst present the following basic assumptions.
Assumption 1 Assume F and K satisfy the following conditions:
• There exist positive m F and m K , such that (cid:104) F ( θ ) − F ( θ (cid:48) ) , θ − θ (cid:48) (cid:105) ≥ m F (cid:107) θ − θ (cid:48) (cid:107) 2 and (cid:104)∇ K ( θ ) − ∇ K
( θ (cid:48) ) , θ − θ (cid:48) (cid:105) ≤ − m K (cid:107) θ − θ (cid:48) (cid:107) 2 .
• F is bounded by H F and L F -Lipschitz continuous i
e., (cid:107) F ( θ ) (cid:107) ≤ H F and (cid:107) F ( θ 1 ) − F ( θ 2 ) (cid:107) ≤
L F (cid:107) θ 1 − θ 2 (cid:107) ; .
• K is L K -Lipschitz {+continuous; ∇ K is bounded by H ∇ K and L ∇ K -Lipschitz continuous • F ( 0 ) = 
+}
and K is an even function, i.e., K ( − θ )
=
K ( θ ) .
Note the ﬁrst bullet indicates U to be a convex function and K to be a concave function.
For an RBF kernel, the later could be achieved by setting the bandwidth large enough and only considering the concave region for simplicity.
This assumption is used for revealing some undesired property of
SVGD developed below.
We do not need such an assumption when analyzing under a nonconvex energy function U in Section 5.
Then " F ( 0 ) = 0 " in the second bullet is reasonable, as F in our setting corresponds to an unnormalized log-posterior, which can be shifted such that F ( 0 ) = 0 for a speciﬁc problem.
Since we often care about bounded space in practice, we can realize the third bullet due to the continuity of K and ∇ K .
The high-level idea of bounding W 1 ( µ T , ν ∞ ) in this section is to decompose it as follows:
[Equation]
4. B OUNDS WITH STOCHASTIC PARTICLE APPROXIMATION
We ﬁrstly bound W 1 ( ρ (cid:80) T − 1 k =
h k , ν (cid:80) T − 1 k =0 h k ) and W 1 ( ν (cid:80) T − 1 k =
h k , ν ∞ ) with the following theorems.
Theorem 3 Under Assumption 1 and let ρ 0 = ν 0 , there exist some positive constants c 1 and cindependent of ( M, t ) and satisfying c 2 < β − 1 such that
[Equation]
Remark
According to Theorem 3, we can bound the W 1 ( ρ (cid:80) T [Equation] 1 k =
h k , ν (cid:80) [Equation] =
[Equation]
k ) term as
W [Equation] Furthermore, by letting t → ∞ , we have W 1 ( ρ ∞ , ν ∞ ) ≤ c 1 √ M ( β − 1 − c 2 ) , which is an important intermediate result to prove the following theorem.
Theorem 4 Under Assumption 1, the following holds:
W 1 ( ν t , ν ∞ ) ≤ c 3 e − 2 λ 1 t , where λ 1 = β − 1 m F − 3 H F L K − 2 L F and c 3 is some positive constant independent of ( M, t ) .
Hence, the
W 1 ( ν (cid:80) Tk =
h k , ν ∞ ) term in (10) can be bounded as:
[Equation]
To ensure W 1 ( ν (cid:80) [Equation] 1 k =
h k , ν ∞ ) to decrease over time, one needs to choose β small enough such that λ 1 > 0 .
This also sheds light on a failure case of SVGD (where β → ∞ ) discussed in Section 4.4.
B OUNDS WITH A NUMERICAL SOLUTION
To bound the W 1 ( µ T , ρ (cid:80) T − 1 k =0 h k ) term, we adopt techniques from (Raginsky et al., 2017; Xu et al., 2018) on analyzing the behaviors of SGLD, and derive the following results for our SPOS algorithm:
Theorem 5 Under Assumptions 1, for a ﬁxed step size h k = h that is small enough, the corresponding W 1 ( µ T , ρ Th ) is bounded as
[Equation]
where B is the ﬁxed size of the minibatch in each iteration and ( c 4 , c 5 , c 6 ) are some positive constants independent of ( M, T, h ) .
The dependence of T in the bound above makes the bound relatively loose.
Fortunately, we can improve the bound to make it independent of T by considering a decreasing-stepsize SPOS algorithm, stated in Theorem 6.
Theorem 6 Under Assumptions 1, for a decreasing step size h k = h 0 / ( k + 1) , and let the minibatch size in each iteration k be B k
= B 0 + [log( k + 1)] 100 / 99 , the corresponding W 1 ( µ T , ρ (cid:80) T − 1 k =0 h k ) term is bounded, for some β small enough, as
[Equation]
where B 0 is the initial minibatch size, and ( c 4 , c 7 , c 8 , c 9 ) are some positive constants independent of ( M, T, h 0 ) .
Note B k increases at a very low speed, e.g. , only by 15 after 10 5 iterations, thus it would not affect algorithm efﬁciency.
According to Theorem 6, W 1 ( µ T , ρ (cid:80) T − 1 k =0 h k ) would approach zero when h 1 / 2 0 M → 0 .
By directly combining results from Theorem 3–6, one can easily bound the target
W 1 ( µ T , ν ∞ ) .
Detailed statements are given in Theorem 15–16 in Section H of the SM.
4. A P ITFALL OF SVGD
Based on the above analysis, we now formally show a pitfall of SVGD, i.e. , particles in SVGD tend to collapse to a local mode under some particular conditions.
Inspired by the work on analyzing the granular media equations by Malrieu (2003);
Cattiaux et al.
(2008), we measure this by calculating the expected distance between particles, called expected particle distance (EPD).
Firstly, we bound the EPD for the proposed SPOS algorithm in Theorem 7.
Theorem 7 Under Assumption 1, further assuming every { θ ( i ) t } with the same initial probability law ρ 0 and Γ (cid:44) E θ ∼ ρ 0 , θ (cid:48) ∼ ρ
[ (cid:107) θ − θ (cid:48) (cid:107) 2 ] < ∞ .
Choose a β such that λ
= m F β + m K − H F L K > 0 .
Then the EPD of SPOS is bounded as:
[Equation]
where C 1 = M ( M − 1)Γ − 4 (cid:112) dβ − 1 Mλ .
Remark 2 There are two interesting cases: i ) When C 1 > 0 , the EPD would decrease to the bound 4 (cid:112) dβ − 1 M/λ along time t .
This represents the phenomenon of an attraction force between particles; ii ) When C 1 < 0 , the EPD would increase to the same bound, which represents the phenomenon of a repulsive force between particles, e.g., when particles are initialized with the same value ( Γ = 0 ), they would be pushed away from each other until the EPD increases to the aforementioned bound.
Intuitively, the EPD for SVGD can be obtained by taking the β → ∞ limit.
Corollary 8 formally characterizes the particle-degeneracy phenomenon of SVGD, which has been empirically studied in (Zhuo et al., 2018).
Corollary 8 Under the same conditions of Theorem 7, the EPD in SVGD is bounded as: EPD (cid:44) (cid:113)(cid:80) Mi,j (cid:107)
θ ( i ) t − θ ( j ) t (cid:107) 2 ≤ C
e − 2 λt , where C
=
(cid:113)(cid:80) Mi,j (cid:107)
θ ( i )
− θ ( j ) 0 (cid:107) 2 and λ = m K − H F L K .
Remark 3 We would like to emphasize two points: 1)
In the case of λ ≥ 0 , Corollary 8 indicates that particles in SVGD would collapse to a point when t → ∞ .
In practice, we usually ﬁnd that particles are trapped in a local mode instead of collapsing.
This is due to two reasons: i ) numerical errors inject noise into the particles; ii ) some particles are out of the concave region of K stated in Assumption 1 in SVGD, which is required for the theory to hold.
All these make the empirical
EPD not exactly the same as the true particle distance.
2) Corollary 8 also applies when the energy function is nonconvex.
Our proof in the SM considers the nonconvex case as well.
Consequently, this serves as a strong theoretical motivation to apply SPOS instead of SVGD in deep learning.
N ON -A SYMPTOTIC C ONVERGENCE A NALYSIS : THE N ONCONVEX C ASE
Since the non-convex case is much more complicated than the convex case, we reply on different assumptions and adopt another distance metric, denoted as ˜ B , to characterize the convergence behavior of SPOS under the non-convex case.
Speciﬁcally, ˜ B ( µ, ν ) is deﬁned as ˜ B ( µ, ν ) (cid:44) | E θ ∼ µ [ f ( θ )] − E θ ∼ ν
[ f ( θ )] | for a known L f -continuous function f satisfying Assumption 2 below.
Note such metric has also been adopted in (Vollmer et al., 2016; Chen et al., 2015).
Our analysis considers ( T, M, h k ) as variables in ˜ B .
In addition, we use { ˆ θ ( i ) k } Mi =1 to denote the particles when full gradients are adopted in (9).
The distribution of the particles is denoted as ˆ µ k .
Our high-level idea of bounding ˜ B ( µ T , ν ∞ ) is to decompose it as follows:
[Equation]
Our second idea is to concatenate the particles at each time into a single vector representation, i.e. deﬁning the new parameter at time t as Θ t (cid:44) [ θ (1) t , · · · , θ ( M ) t ] ∈ R Md .
Consequently, the nonlinear
SDE system (8) can be turned into a linear SDE ,which means Θ t is driven by the following linear
SDE:
[Equation]
[Equation]
where F Θ ( Θ t )
[Equation]
− θ ( [Equation] [Equation] ) F ( θ ( j ) t ) , · · · , β −
[Equation]
F
[Equation]
] is a vector function R Md → R Md , and W ( Md ) t is Brownian motion of dimension Md .
Similarly, we can deﬁne ˆ Θ k (cid:44) [ˆ θ (1) k , · · · , ˆ θ ( M ) k ] ∈ R Md for the full-gradient case.
Hence, it can be seen that through such a decomposition in (15), the bound related to a nonlinear SDE system (8) reduces to that of a linear SDE.
The second term ˜ B (ˆ µ T , ˆ µ ∞ ) reﬂexes the geometric ergodicity of a linear dynamic system with a numerical method.
It is known that even if a dynamic system has an exponential convergence rate to its equilibrium, its corresponding numerical method might not.
Our bound for ˜
B (ˆ µ T , ˆ µ ∞ ) is essentially a speciﬁcation of the result of Mattingly et al.
(2002), which has also been applied by Xu et al.
The third term ˜ B (ˆ µ ∞ , ρ ∞ ) reﬂects the numerical error of a linear SDE, which has been studied in related literature such as (Chen et al., 2015).
To this end, we adopt standard assumptions used in the analysis of SDEs (Vollmer et al., 2016; Chen et al., 2015), rephrased in
Assumption 2.
Assumption 2 (Assumption in (Vollmer et al., 2016; Chen et al., 2015))
For the linear SDE (16) and a Lipschitz function f , let ψ be the solution functional of the Poisson equation: G ψ ( ˆ Θ k ) =
1 M (cid:80) Mi =1 f (ˆ θ ( i ) k ) − E θ ∼ p ( θ |D )
[ f ( θ )] , where G denotes the inﬁnite generator of the linear SDE (16) .
Assume ψ and its up to 4th-order derivatives, D k ψ , are bounded by a function V , i.e., (cid:107)D k ψ (cid:107) ≤ H k V p k for k
= (0 , 1 , 2 , 3 , 4) , H k , p k > 0 .
Furthermore, the expectation of V on { Θ t } is bounded: sup l E V p ( Θ t ) < ∞ , and V is smooth such that sup s ∈ (0 , 1) V p ( s Θ + (1 − s ) Θ (cid:48) ) ≤
H ( V p ( Θ ) + V p ( Θ (cid:48) )) , ∀ Θ , Θ (cid:48) , p ≤ max { 2 p k } for H > 0 .
Assumption 3 i ) F , K and ∇ K are L F , L K and L ∇ k Lipschitz; ii ) F satisﬁes the dissipative property, i.e., (cid:104) F ( θ ) , θ (cid:105) ≥ m (cid:107) θ (cid:107) 2 − b for some m, b > 0 ; iii ) Remark 1 applies to the nonconvex setting, i.e. sup (cid:107)
[Equation]
Remark 4 Assumption 2 is necessary to control the gap between a numerical solution and the exact solution of an SDE.
Speciﬁcally, it is used to bound the ˜ B (ˆ µ ∞ , ρ ∞ ) term and the ˜ B ( µ T , ˆ µ T ) term above.
Purely relying on the dissipative assumption in Assumption 3 as in non-convex optimization with SG-MCMC (Raginsky et al., 2017; Xu et al., 2018) would induce a bound increasing linearly w.r.t.time t .
Thus it is not suitable for our goal.
Finally, iii ) in Assumption 3 is a mild condition and reasonable because we expect particles to be able to approximate all distributions equally well in the asymptotic limit of t → ∞ by ergodicity due to the injected noise.
How to remove/replace this assumption is an interesting future work.
Based on the assumptions above, the bounds for ˜ B (ˆ µ T , ˆ µ ∞ ) and ˜ B (ˆ µ ∞ , ρ ∞ ) are summarized below.
Theorem 9 Under Assumption 2–3, if we set the stepsize h k
= h , we can have the following results:
[Equation]
where ς = 2 L Θ ( Mbβ + m Θ β + Md ) /m
Θ , L Θ = √ 2 β − 1 L F + l (cid:48) , m Θ = β − 1 m − m (cid:48) , and ( σ, C 2 , C 3 , l (cid:48) , m (cid:48) ) are some positive constants independent of (T, M, h) and σ ∈ (0 , 1)
Remark
It is seen that in order to make the ˜ B (ˆ µ T , ˆ µ ∞ ) term asymptotically decrease to zero, the number of running iteration T should increase at a rate faster enough to compensate the effect of increasing M .
We believe there is room for improving this bound, which is an interesting future work.
Next we bound the ˜ B ( µ T , ˆ µ T ) term related to stochastic gradients.
By adapting results from linear
SDE (Xu et al., 2018), ˜ B ( µ T , ˆ µ T ) can be bounded with Theorem 10.
Theorem 10 Under Assumptions 2–3, if we set B k = B and h k = h , ˜ B ( µ T , ˆ µ T ) is bounded as [Equation] BM )
[Equation]
where Γ (cid:48)
=
2(1 + 1 /m Θ )
( Mb + 2 M 2 C 24 + Md/β )
and , ( C 4 , C 5 ) is some positive constant independent of (T, M, h)
Finally, by combining the results from Theorem 9, 10 and iii ) in Assumption 3, we arrive at a bound for our target ˜ B ( µ T , ν ∞ ) , summarized in Theorem 11.
Theorem 11 Under Assumptions 2–3, there exist some positive constants ( C 2 , C 3 , C 4 , C 5 , C 6 ) such that
[Equation]
where σ , ς and Γ (cid:48) are the same as those in Theorem 9–10.
E XPERIMENTS
In this section, we illustrate the effectiveness of our proposed Bayesian sampling framework on several deep-learning models, including Bayesian learning of deep neural network and Bayesian exploration in deep reinforcement learning.
We start with a simple illustrative example.
6. B OUNDS ILLUSTRATION WITH A SIMPLE G AUSSIAN EXAMPLE
[Figure]
We follow Chen et al.
and consider a standard Gaussian model where x i ∼
N ( θ, 1) , θ ∼ N (0 , 1) .
1000 data samples { x i } are generated, and every minibatch in the stochastic gradient is of size 10.
The test function is deﬁned as f ( θ ) (cid:44) θ 2 , with explicit expression for the posterior average.
To evaluate the expectations in the bias and MSE [Table] we average over 200 runs with random initializations.
The estimation errors are plotted in Figure 2.
It is observed that at the beginning, the errors for the ones with less particles decrease faster than those with more particles.
This is reﬂected in the bounds in Theorem 15, which are dominated by the bound in Theorem 5 (indicating larger M results in larger errors at the beginning).
When more running time/iterations are given, the increase of error in Theorem 5 by increasing M essentially cancels out the exponentially-decay term in Theorem 4.
According to Theorem 3, the error would eventually decrease with increasing number of particles.
6. B AYESIAN NEURAL NETWORKS FOR REGRESSION
[Table]
We conduct experiments for
Bayesian learning of DNNs, where we Bayesian DNNs are used to model weight uncertainty of neural networks, an important topic that has been well explored (HernándezLobato & Adams, 2015;
Blundell et al., 2015; Li et al., 2016; Louizos & Welling, 2016).
We assign simple isotropic Gaussian priors to the weights, and perform posterior sampling with different methods.
For all methods, we use a
RBF kernel K ( θ , θ (cid:48) )
=
exp( −(cid:107) θ − θ (cid:48) (cid:107) 22 /η 2 ) , with the bandwidth set to η = med 2 / log M .
Here med is the median of the pairwise distance between particles.
We use a single-layer BNN for regression tasks.
Following Li et al.
(2015), 10 UCI public datasets are considered: 100 hidden units forlarge datasets (Protein and YearPredict), and 50 hidden units for the other 8 small datasets.
Following
Zhang et al.
(2018b), we repeat the experiments 20 times with batchsize 100 for all datasets except for Protein and YearPredict, which we repeat 5 times and once with batchsize 1000.
The datasets are randomly split into 90% training and 10% testing.
For a fair comparison, we use the same split of data (train, val and test) for SVGD and SPOS.
The test results are reported on the best model on the validation set.
We adopt the root mean squared error (RMSE) and test log-likelihood as the evaluation criteria.
The experimental results are shown in Table 1, from where we can see the proposed SPOS outperforms other methods, achieving state-of-the-art results.
6. B AYESIAN EXPLORATION IN DEEP REINFORCEMENT LEARNING
it is well-accepted that RL performance directly measures how well the uncertainty is learned , reﬂected by the exploration stage.
As a result, we apply our method for RL.
Following Liu et al.
(2017); Zhang et al. (2018a), we deﬁne policies in RL with Bayesian neural networks.
This naturally introduces uncertainty into action selections under a speciﬁc state-action pair, rendering Bayesian explorations to make policy learning more effective.
Speciﬁcally, denote the policy as π θ ( a | s ) parameterized by θ with prior distribution p ( θ ) , where a represent the action variable, and s the state variable.
According to Liu et al.
(2017), learning the optimal policy corresponds to calculating the following posterior distribution for θ :
[Equation]
[Figure]
where J ( θ ) denotes the expected cumulative reward under the policy with parameter θ and α a hyperparameter.
Consequently, θ could be updated by drawing samples from (19) with the proposed
SPOS.
We denote this method as SPOS-PG.
In addition, when drawing samples with SVGD, the resulting algorithm is called Stein variational policy gradient (SVPG) (Liu et al., 2017).
Note in implementation, the term J ( θ ) can be approximated with REINFORCE (Williams, 1992) or advantage actor critic (Schulman et al., 2015), which we will investigate in our experiments.
We follow the same setting as in (Liu et al., 2017) except using simpler policy-network architectures as Houthooft et al. We conduct experiments on three classical continuous control tasks are considered: Cartpole Swing-Up, Double Pendulum, and Cartpole.
Speciﬁcally, the policy is parameterized as a two-layer (25-10 hidden units) neural network with tanh as the activation function.
The maximal length of horizon is set to 500.
We use a sample size of 10000 for policy gradient estimation, and M = 16 , α = 10 .
For the simplest task, Cartpole, all agents are trained forepisodes.
For the other two complex tasks, all agents are trained up to 1000 episodes.
The average reward versus number of episodes are plotted in Figure 3.
It is observed that our proposed SPOS-PG obtains much larger average rewards as well as smaller variance compared to SVPG, though the convergence behaviors are similar in the simplest task Carpole.
C ONCLUSION
Motivated by the need of effective and efﬁcient Bayesian sampling techniques in modern deep learning, we propose a probability approach for particle-optimization-based sampling that uniﬁes
SG-MCMC and SVGD.
Notably, for the ﬁrst time, by analyzing the corresponding nonlinear SDE, we develop non-asymptotic convergence theory for the proposed SPOS framework, a missing yet important theoretical result since the development of SVGD.
Within our theoretical framework, a pitfall of SVGD, which has been studied empirically (Zhuo et al., 2018), is formally analyzed.
Our theory also indicates the convergence of SPOS to the true posterior distribution in the asymptotic limit of inﬁnite particles and iterations under appropriate conditions.
Our theory is of great practice value, as for the ﬁrst time it provides nonasymptotic theoretical guarantees for the recently proposed particleoptimization-based algorithms such as the SVGD, whose advantages have also been extensively evaluated by experiments on Bayesian learning of DNNs and Bayesian exploration of DRL.
There are a number of interesting future works.
For example, one might explore more recently developed techniques such as (Cheng et al., 2018) to improve the convergence bound; one can also adopt the SPOS framework for non-convex optimization like what SG-MCMC is used for, and develop corresponding theory to study the convergence property of the algorithm to the global optimum.
A D ENSITY F UNCTION OF THE M ULTI -M ODE D ISTRIBUTION IN S ECTION
The negative log-density function of the multi-mode distribution in Section 3 is deﬁned as:
[Equation]
where c = ( − 0 .
47 , − 0 .
83 , − 0 .
71 , − 0 .
27 , − 0 .
87 , − 0 .
37) is a vector, c
i is the i -th element of c .
B G RONWALL L EMMA
The Gronwall Lemma plays an important role in parts of our proofs, which is stated in Lemma 12.
Lemma 12 (Gronwall Lemma) Let I denotes an interval of the form [ a, + ∞ ) for some a ∈ R .
If v ( t ) , deﬁned on I , is differentiable in I and satisﬁes the following inequality:
[Equation]
where β ( t ) is a real-value continuous function deﬁned on I .
Then v ( t ) can be bounded as:
[Equation]
C P ROOF OF T HEOREM
To prove Theorem 2, we rely on the deﬁnition of generalized derivative in Deﬁnition 1.
Deﬁnition 1 (Generalized Derivative) Let g and φ be locally integrable functions on an open set Ω ⊂ R d , that is, Lebesgue integrable on any closed bounded set F ⊂ ω .
Then φ is the generalized derivative of g with respect to θ on Ω , written as φ = ∂ θ g , if for any inﬁnitely-differentiable function u with compact support in Ω , we have
[Equation]
Proof The proof relies on further expansions on the deﬁnition of generalized derivative on speciﬁc functions.
Speciﬁcally, let the function g in Deﬁnition 1 be in a form of g (cid:44)
Gf for two functions G and f .
The generalized derivative of ( Gf )( v, t ) , written as ∂ θ ( Gf )( θ , t ) , satisﬁes
[Equation]
for all differentiable function u ( · ) .
In Theorem 2, we want to prove a particle representation of the following PDE:
[Equation]
where f ( θ , t )
= µ t .
Consequently, we have
[Equation]
By applying (20) in (21), we have
[Equation]
Since f d θ (cid:44) µ (d θ , t ) , we have
[Equation]
In particle approximation, we have µ ( t ) =
1 M (cid:80) Mi
=1 δ ( θ ( i ) t ) ( θ ) .
For each particle, let u ( θ )
=
θ , (22) reduces to the following equation:
[Equation]
This completes the proof.
D P ROOF OF T HEOREM
Note that one challenge in our analysis compared with the analysis for linear SDEs, such as those for
SG-MCMC Vollmer et al.
; Chen et al.
(2015), is how to bound the gap between the original nonlinear PDE (4) and the reduced nonlinear SDE (8).
Based on the techniques on analyzing granular media equations in Malrieu (2003);
Cattiaux et al.
; Durmus et al.
(2018), we introduce a nonlinear SDE as an element in-between (6) and (8) like :
[Equation]
(23) where L (¯ θ t ) denotes the probability law of ¯ θ t ,
¯ W t ∈ R
d is a d -dimensional Brownian motion and Y is a random variable independent of ¯ θ t and just used here for the sake of clarity.
In order to match the
SDE system (8) of the particles { θ ( i ) t } Mi =1 , we duplicate (23) M times, each endowing with an exact solution ¯ θ ( i ) t indexed by i .
The distribution of each particles { ¯ θ ( i ) t } Mi =1 is ν t and the corresponding ¯ W ( i ) t can be set exactly the same as the W ( i ) t in (8):
[Equation]
24) where Y i is a random variable independent of ¯ θ ( i ) t and just used here for the convenience of the proof.
Proof [Proof of Theorem 3] Firstly we have
[Equation]
d
[Equation]
where A ij ( t ) =
− β − 1 (cid:16) F ( θ ( i ) t ) − F (¯ θ ( i ) t ) (cid:17) · (cid:16) θ ( i ) t − ¯ θ ( i ) t (cid:17)
B
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
For the B ij ( t ) term, applying the concave condition for K and the oddness of ∇ K in Assumption 1, we have
[Equation]
For the C ij ( t ) term, we have
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
where the (1) is obtained by applying the Cauchy-Schwarz inequality and (2) by the fact that
E (cid:16) K (¯ θ ( i ) t − ¯ θ ( j ) t ) − K ∗ ν t
(¯ θ ( i ) t ) (cid:17)
= 0 .
We can tune the bandwidth of the RBF kernel to make (cid:107)∇ K (cid:107) ≤ H ∇ K .
Hence (3) is obtain by the boundedness of ∇ K ( θ ) .
Similarly, since K ≤ 1 , we have the following result for H ij ( t ) term,
[Equation]  
[Equation]
[Equation]
For the F ij ( t ) and G ij ( t ) terms
[Table]
we have:
[Equation]
[Table] [Equation] [Table] [Equation]
[Table]
[Equation] [Table] [Equation] [Table]
[Equation]
[Table] [Equation] [Table] [Equation]
[Table]
[Equation]
[Table] [Equation] [Table] [Equation]
[Table]
[Equation]
[Table] [Equation] [Table] [Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
We denote γ i ( t ) (cid:44) E (cid:13)(cid:13)(cid:13)
θ ( i ) t − ¯ θ ( i ) t (cid:13)(cid:13)(cid:13) 2 .
Due to the exchangeability of the particles, γ i ( t ) are the same for all the particles, denoted as γ ( t ) .
Then according to (25), we have
[Equation]
where λ 1 = β − 1 m F − 3 H F L K − 2 L F ⇒ ( (cid:112) γ ( t ) −
[Equation]
Note that θ ( i ) t and ¯ θ ( i ) t are initialized with the same initial distribution µ 0 = ν [-
-]0 .
In the deﬁnition of ¯ θ ( i ) t , there is no restriction on how the initial value is set.
As a result, we can set θ ( i ) 0 to be identical to ¯ θ ( i ) 0 , leading to γ (0)
= 0 .
Then according to the Gronwall Lemma, we have
[Equation]
Hence, there exist some positive constant ( c 1 , c 2 ) such that:
[Equation]
[Equation]
where (1) holds due to the relationship between W 1 and W 2 metric Givens & Shortt (1984), (2) due to the deﬁnition of W 2 and (3) due to the result from the previous proof.
E P ROOF OF T HEOREM
Proof [Proof of Theorem 4] Firstly, what we aim at is W 1 ( ν t , ν ∞ ) ≤ c 3 exp ( − 2 λ 1 t ) in this theorem.
According to the relationship between W 1 and W 2 metric Givens & Shortt (1984), once we bound
W 2 ( ν t , ν ∞ ) as W 2 ( ν t , ν ∞ ) ≤ c 3 exp ( − 2 λ 1 t ) , we will ﬁnish our proof.
Next, look at the equation (8):
If we set the initial distribution of each particle to be ν 0 , which means ρ 0 =
L ( θ ( i ) 0 ) =
ν 0 , we will derive M particles denoted as { θ ( i ) t, 1 } Mi =1 .
We denote the distribution of each θ ( i ) t, 1 at t as ρ t, 1 .
If we set the initial distribution of each particle to be ν ∞ , which means ρ
=
L ( θ ( i ) 0 )
=
ν ∞ , we will derive M particles denoted as { θ ( i ) t, 2 } Mi =1 .
We denote the distribution of each θ ( i ) t, 2 at t as ρ t, 2 .
Since we need to bound W 2 ( ν t , ν ∞ ) , we make the following decomposition:
[Equation]
Note that ρ 0 , 1 = ν 0 and ρ 0 , 2 = ν ∞ .
Then, according to (26), we have
[Equation]
Now we need to focus on the term W 2 ( ρ t, 1 , ρ t, 2 ) .
Since W 2 ( ρ t, 1 , ρ t, 2 ) ≤ E (cid:18)(cid:13)(cid:13)(cid:13) θ ( i ) t, 1 −
[Equation]
( i ) t, 2 (cid:13)(cid:13)(cid:13)(cid:19) (cid:44) r
( t ) , we will derive a bound for E (cid:18)(cid:13)(cid:13)(cid:13)
θ ( i ) t, 1 − θ ( i ) t, 2 (cid:13)(cid:13)(cid:13) 2 (cid:19) in the following.
We have d (cid:16) θ ( i ) t, 1 − θ ( i ) t, 2 (cid:17) =
− β − 1 (cid:16) F ( θ ( i ) t, 1 ) − F ( θ ( i ) t, 2 ) (cid:17) d t +
M
[Equation]
(cid:88)
[Equation]
(cid:104)
[Equation]
−
[Equation]
M
[Equation]
(cid:88) j (cid:16) F ( θ ( j ) t, 1 ) K
( θ ( i ) t, 1 − θ ( j ) t, 1 ) − F
( θ ( j ) t, 2 ) K
( θ ( i ) t
[Equation] [Equation]
( j ) t,
[Equation]
(cid:17) d t ⇒ d (cid:32) M (cid:88) i (cid:13)(cid:13)(cid:13) θ ( i ) t, 1 − θ ( i ) t, 2 (cid:13)(cid:13)(cid:13)(cid:33) =
M
M (cid:88) i,j ( ξ 1 ij ( t ) + ξ 2 ij ( t ) + ξ 3 ij ( t ) + ξ 4 ij ( t ))
d t where ξ 1 ij ( t ) = − β − 1 (cid:16) F ( θ ( i ) t, 1 ) − F
( θ ( i ) t, 2 ) (cid:17) · (cid:16) θ ( i ) t, 1 − θ ( i ) t,
[Equation]
For the ξ 1 ij ( t ) terms, according to the i ) in Assumption 1 for F , we have (cid:88)
[Equation]
For the ξ 2 ij ( t ) term, applying the concave condition for K and the oddness of ∇ K in Assumption 1, we have
[Equation]
For the ξ 3 ij ( t ) terms, after applying the L F -Lipschitz property for F and K ≤ 1 , we have
[Equation]
θ ( j ) t, 1 − θ ( j ) t, 2 (cid:13)(cid:13)(cid:13) (cid:13)(cid:13)(cid:13) θ
( i ) t, 1 − θ ( i ) t, 2 (cid:13)(cid:13)(cid:13) ≤ 2 L F M (cid:88) i (cid:13)(cid:13)(cid:13) θ
( i ) t, 1 − θ ( i ) t, 2 (cid:13)(cid:13)(cid:13) 2 .
For the ξ 4 ij ( t ) terms :
[Equation]
− θ
[Equation]
(cid:13)(cid:13)(cid:13)
[Equation]
Now we have
[Equation]
According to the Gronwall lemma,
[Equation]
where λ 1 = β − 1 m F − 3 H F L W − 2 L F .
Consequently, there exists some positive constant c 3 such that
[Equation]
Then we have
[Equation]
However, it worth noting that ν t is the solution of (6) which has nothing to do with the number of particles, M .
Then let M → ∞ , we can derive that W 2 ( ν t , ν ∞ ) ≤ c 3 e − 2 λ 1 t .
Now we ﬁnish our proof.
F P ROOF OF T HEOREM
To bound the W 1 ( µ T , ρ (cid:80) T − 1 k =0 h k ) term, note the original SDE driving the particles { θ ( i ) t } in (8) is a nonlinear SDE, which is hard to deal with.
Fortunately, (8) can be turned into a linear SDE by concatenating the particles at each time into a single vector representation,
i
[Equation]
e. , by deﬁning the new parameter at time t as Θ t (cid:44) [ θ (1) t , · · · , θ ( M ) t ] ∈ R Md .
Consequently, Θ t is driven by the following linear SDE:
[Equation]
where F Θ ( Θ t )
[Equation]
] is a vector function R Md → R Md
[Equation]
and W ( Md ) t is Brownian motion of dimension Md .
Now we deﬁne the F
[Equation]
Θ
[Equation]
− θ
[Equation]
F
[Equation]
F q ( θ ( M )
[Equation]
− θ ( j ) t )
F q ( θ ( j ) t )] .
We can verify that F Θ ( Θ t ) =
(cid:80) Nq =1 F ( q ) Θ ( Θ t ) .
Then we deﬁne Θ k (cid:44) [ θ (1) k , · · · , θ ( M ) k ] and G Θ I k (cid:44) NB k (cid:80)
q ∈I k F ( q ) Θ ( Θ k ) .
We can verify that the following result holds:
[Equation]
where Ξ k ∼ N ( 0 , I Md × Md ) .
Now we reach the conclusion that Θ k of (29) is accutually the numerical solution of the SDE (28) via stochastic gradients.
We denote the distribution of Θ k as µ Θk and the distribution of Θ t as ρ Θ t .
Before proceeding to our theoretical results, we need to present the following Lemmas which is very important in our proof.
[Equation]
Proof [Proof of Lemma 13] Let us recall the deﬁnition of W 1 metric and its Kantorovich-Rubinstein duality Villani (2008), i.e. W 1 ( µ, ν ) (cid:44) sup (cid:107) g (cid:107) lip ≤ 1 | E θ ∼ µ [ g ( θ )] − E θ ∼ ν
[ g ( θ )] | .
We can prove the fact that if g ( θ ) : R d → R is a L g -Lipschitz function in R d , the g Θ ( Θ ) , deﬁned as g Θ ( Θ ) = 1 √ M (cid:80)
Mi g ( θ ( i ) ) , is a L g -Lipschitz function in R Md , where Θ (cid:44) [ θ (1) , · · · , θ ( M ) ] :
[Equation]
Then we have:
[Equation]
The (1) holds since E θ (1) k ∼ µ k
[ g ( θ (1) k )] = · · · = E θ ( M ) k ∼ µ k
[ g ( θ ( M ) k )] for all the particles θ ( i ) k and
E θ (1) t ∼ ρ t [ g ( θ (1) t )]
= · · · = E θ ( M ) t ∼ ρ t [ g ( θ ( M ) t )] for all the particles θ ( i ) t .
Then according to the deﬁnition of W 1 metric, we derive that
[Equation]
Lemma 14 Assuming F ( 0 ) = 0 .
If F in (9) is Lipschitz with constant L F , and satisﬁes the dissipative property that (cid:104) F ( θ ) , θ (cid:105) ≥ m F (cid:107) θ (cid:107) 2 − b .
Then F Θ in (28) is Lipschitz-continuous with constant √ 2 β − 1 L F + l (cid:48) and satisﬁes (cid:104) F Θ ( Θ ) , Θ (cid:105) ≥ ( β − 1 m F − m (cid:48) )
(cid:107) Θ (cid:107) 2 − β − 1 Mb , where l (cid:48) and m (cid:48) are some positive constants.
Proof [Proof of Lemma 14]
[Equation]
It is easy to verify that there exits some positive constant l (cid:48) such that
[Equation] [Equation]
Next, we have
[Equation]
Notice :
[Equation]
Since we have assumed F (0) =
[Equation]
we have:
[Equation]
Since ∇ K is an odd function, we have:
[Equation]
As a result, we can derive the following result:
[Equation]
Now it is ready to prove Theorem 5.
It worth noting
that after assuming F (0) = 0 , the ﬁrst bullet in
Assumption 1 recovers the dissipative assumption as (cid:104) F ( θ ) , θ (cid:105) ≥ m F (cid:107) θ (cid:107) 2 .
Proof Next we use Lemma C.5 in Xu et al.
(2018) to verify that F Θ satisﬁes the assumptions in
Raginsky et al.
(2017) by setting δ =
a (cid:48) B with a (cid:48) a positive constant and B the size of the random set
I .
Let µ Θk :
=
L ( Θ k ) and ρ Θ t :=
L ( Θ t ) .
Now we can borrow the result of Lemma 3.6 in Raginsky et al.
The relative entropy D KL ( µ Θk (cid:107) ρ Θ kh ) satisﬁes:
[Equation]
with
[Equation]
and a 1 , a 2 are some positive constants.
When the β is small enough, there exist some positive constants a 3 , a 4 such that
[Equation]
Similar to the proof of Lemma 14, it is easy to verify that there exists some positive constant asuch that (cid:104) F Θ ( Θ 1 ) − F Θ ( Θ 2 ) , Θ 1 − Θ 2 (cid:105) ≥ ( β − 1 m F − a 5 ) (cid:107) Θ 1 − Θ 2 (cid:107) 2 .
Notice, when β is small enough, (28) satisﬁes the conditions of Proposition 4.2 in Cattiaux et al.
Hence, there exits some positive constant C such that W 1 ( µ Θk , ρ Θ kh ) ≤ C (cid:113) D KL ( µ Θk (cid:107) ρ Θ kh ) .
According to Corollary 4 and Lemma 8 in Bolley & Villani (2005), we can derive an explicit expression for C :
[Equation]
when β is small enough and a 6 is some positive constant.
Applying the Lemma 13, we have
[Equation]
Let k = T
and we can ﬁnish the poof.
G P ROOF OF T HEOREM
Proof Our proof is based on the proof of Lemma 3.6 in Raginsky et al.
(2017) with some modiﬁcations.
Firstly, adopt the same notations in the Section F and we get the following update:
[Equation]
(30) where Ξ k ∼ N ( 0 , I Md × Md ) and h k = h[-
-]
k +1 .
We assume E ( G Θ I k ) =
F Θ ( Θ k ) , ∀ Θ ∈ R Md , which is a general assumption due to the way that we choose the minibatch I k .
We need to deﬁne q ( t ) , which will be used in the following proof:
[Equation]
Furthermore, we deﬁne (cid:80) − 1 i =
h i (cid:44)
and (cid:80) 0 i =
h i (cid:44) h 0 here just used for the convenience of statement in the following.
Now we focus on the following continuous-time interpolation of Θ k :
[Equation]
, where I ( s ) ≡ I k for t ∈ (cid:104)(cid:80) k − 1 i =
h i , (cid:80) ki =
h i (cid:17) , ˜
G Θ I ( s ) ( Θ ) (cid:44) NB ( s ) (cid:80) q ∈I ( s ) F ( q ) Θ ( Θ ) and B ( s ) is the size of the minibatch
I ( s ) .
And for each k , Θ ( (cid:80) k − 1 i =
h i ) and Θ k have the same probability law ρ
Θk .
Since Θ ( t ) is not a Markov process, we deﬁne the following Itô process which has the same one-time marginals as Θ ( t )
[Equation]
where G t ( x ) :=
E 
[Equation]
G Θ I ( t
[Equation] ( t [Equation] 1 (cid:88)
[Equation]
[Equation]
Let P t Λ :=
L (Λ( s ) : 0 ≤ s ≤ t ) and P t Θ :=
L ( Θ ( s ) : 0 ≤ s ≤ t ) .
According to the proof of lemma 3.6 in Raginsky et al.
(2017), we can derive a similar result for the relative entropy of P t Λ and
P t Θ :
[Equation]
The last line follows that L ( Θ ( s ))
=
L (Λ( s )) , ∀ s .
In the following proof, we will let t = (cid:80)
k − 1 i =
h i for some k ∈ R .
Now we can use the martingale property of Itô integral to derive:
[Equation]
For the ﬁrst part (31) [Equation] we consider some s ∈ [ (cid:80) j − 1 i =
h i , (cid:80) ji =
h i ) .
The following equation holds:
[Equation]
Thus
[Equation]
we can use Lemma 3.1 and 3.2 in Raginsky et al.
[Equation]
and Lemma C.5 in Xu et al.
(2018) to get the following result:
[Equation]
where b 1 is some positive constant.
Consequently, the ﬁrst part above (31) can be bounded as:
[Equation]
where the last line follows from the fact that
[Equation]
According to Lemma C.5 in Xu et al.
[Equation]
the second part (32) can be bounded as follows:
[Equation]
where the last line follows from the fact that when r > 1 ,
[Equation]
Denote µ Θk :=
L ( Θ k ) and ρ Θ t :=
L ( Θ t ) .
Due to the data-processing inequality for the relative entropy, we have
[Equation]
Lemma 3.2 in Raginsky et al.
(2017) has provided a uniform bound to max 0 ≤ j ≤ k − 1 ( L 2 F Θ E (cid:107) Θ j (cid:107) 2 + b 1 ) .
Hence we can tell that D KL ( P (cid:80) {+k − 1 i =
 h i Λ+}
[Equation]
P (cid:80)
[Equation]
−
[Equation]
h
[Equation]
Θ ) would not increase w.r.t.
k .
This is a nice property that the ﬁxed-step-size SPOS does not endow.
Since L F Θ = √ 2 β − 1 L F + l (cid:48) , it is easy to verify that when β is small enough, there exists some positive constants b 3 , b 4 , b 5 and b 6 such that:
[Equation]
Similar to the proof of Theorem 5, we can bound the W 1 ( µ Θk (cid:107)
ρ Θ (cid:80) k − 1 i =
h i ) term with Corollary 4,
Lemma 8 in Bolley & Villani (2005) and Proposition 4.2 in Cattiaux et al.
Speciﬁcally, when β is small enough, there exist some positive constant a 6 such that:
[Equation]
According to Lemma 13, we have
[Equation]
Let k = T
and we can ﬁnish the proof.
H D ETAILED S TATEMENTS OF N ONASYMPTOTIC C ONVERGENCE UNDER THE C ONVEX C ASE
We give detailed statements of non-asymptotic convergence of SPOS under the convex case, which can be proved by directly combining results from Theorem 3–6, thus they are omitted for simplicity.
We consider a ﬁxed-stepsize and a decreasing-stepsize cases in Theorem 15 and Theorem 16, respectively.
Theorem 15 (Fixed Stepsize) Under Assumption 1, if we set h k = h 0 and B k = B 0 , we can bound the W 1 ( µ T , ν ∞ ) as
[Equation]
where ( c 1 , c 2 , c 3 , c 4 , c 5 , c 6 , β ) are some positive constants such that 1 β > c 2 and m F β > 3 H F L K − 2 L F .
Theorem 16 (Decreasing Stepsize) Under Assumption 1, if we set h k = h 0 / ( k + 1) and B k =[-B 
-]
B 0 + [log( k + 1)] 100 / 99 , we can bound the W 1 ( µ T , ν ∞ ) as
[Equation]
where ( c 1 , c 2 , c 3 , c 6 , c 7 , c 8 , β ) are some positive constants such that 1 β > c 2 and m F β > 3 H F L K − 2 L F , as
I P ROOF OF T HEOREM 7 AND C OROLLARY
Proof [Proof of Theorem 7]
[Equation]
t ) K
( θ ( i ) t − θ ( q ) t ) − F ( θ ( q ) t ) K
( θ ( j ) t − θ ( q ) t )) (cid:17) (cid:16) θ ( i ) t − θ ( j ) t (cid:17) d t + E
M (cid:88) ij (cid:114)β (d W ( i ) t − d W ( j ) t ) (cid:16) θ ( i ) t − θ ( j ) t (cid:17) ≤ − 2 β − 1 m F E
M (cid:88) ij (cid:13)(cid:13)(cid:13) θ ( i ) t − θ ( j ) t (cid:13)(cid:13)(cid:13) 2 d t − 2 m K E M (cid:88) ij (cid:13)(cid:13)(cid:13) θ ( i ) t − θ ( j ) t (cid:13)(cid:13)(cid:13) 2 d t + 2 H F L K E
M (cid:88) ij (cid:13)(cid:13)(cid:13) θ ( i ) t − θ ( j ) t (cid:13)(cid:13)(cid:13) 2 d t +(cid:114)β   E
M (cid:88) ij (d W ( i ) t − d W ( j ) t )
 1 / 2   E
M (cid:88) ij (cid:13)(cid:13)(cid:13) θ ( i ) t − θ ( j ) t (cid:13)(cid:13)(cid:13)  1 /.
Denote z(t)= E (cid:80) M ij (cid:13) (cid:13) (cid:13) θ ( i ) t − θ ( j ) t (cid:13) (cid:13) (cid:13) 2 .
We have z ( t ) (cid:48) ≤ − (2 β
[Equation]
We can ﬁnish our proof by applying Gronwall Lemma on (33)
[Equation]
Proof [Proof for Corollary 8]
[Equation]
⇒ d
[Equation]
−
[Equation]
( i
[Equation]
t
Denote z(t)= E (cid:80) Mij (cid:13)(cid:13)(cid:13)
θ ( i ) t − θ ( j ) t (cid:13)(cid:13)(cid:13) 2 .
We have
[Equation]
We can ﬁnish our proof by applying Gronwall Lemma on (34).
J
P ROOF OF T HEOREM
Proof [Proof of Theorem 9] Our conclusion for ˜ B (ˆ µ T , ˆ µ ∞ ) is essentially a speciﬁcation of the result in Mattingly et al.
(2002), which has also been applied in Xu et al.
Speciﬁcally, we rely on the following lemma, which is essentially Theorem 7.3 in Mattingly et al.
(2002) and Lemma C.3 in Xu et al.
(2018), considering the SDE (16):
[Equation]
As mentioned in Section 5, we ﬁrstly denote the distribution of Θ t as ρ Θ t .
Then we deﬁne the ˆ Θ k (cid:44) [
ˆ θ (1) k , · · · , ˆ θ ( M ) k ] ∈ R Md , which is actually the numerical solution of (16) using full gradient with Euler method.
And we denote the distribution of ˆ Θ k as ˆ µ Θk .
Lemma 17 Let F Θ be Lipschitz-continuous with constant L Θ , and satisfy the dissipative property that (cid:104) F Θ ( Θ ) , Θ (cid:105) ≥ m Θ (cid:107) Θ (cid:107) 2 − b Θ .
Deﬁne V Θ ( Θ )
=
C 0 + L Θ / 2 (cid:107) Θ (cid:107) 2 .
The Euler method for (16) has a unique invariant measure ˆ µ Θ ∞ , and for all test function f Θ such that | f Θ | ≤ V Θ ( Θ ) , we have
[Equation]
where ρ ∈ (0 , 1) , C > 0 are positive constants, and κ = 2 L Θ ( b Θ β + m Θ β + Md )
/m
Θ .
Now we adopt the f Θ : R Md → R deﬁned as f Θ ( Θ )
= 1 M (cid:80) Mi f
( θ ( i ) ) , where f : R d → R is a L f -Lipschitz function and satisﬁes our Assumption 2 and Θ (cid:44) [ θ (1) , · · · , θ ( M ) ] .
Similar to the proof of Lemma 12, we can ﬁnd that f Θ : R Md is a L f / √ M -Lipschitz function.
Furthermore, according to the Lemma 14, it is easily check that F Θ is L Θ -Lipschitz where L Θ = √ 2 β − 1 L F + l (cid:48) .
Hence, when the β is small enough, we have L f / √ M ≤ √ 2 β − 1 L F + l (cid:48) .
Then we can set the Clarge enough to force f Θ to satisfy the condition in Lemma 17 that | f Θ | ≤ V Θ ( Θ ) .
Then, according to the exchangeability of the particle system { ˆ θ ( i ) k } and Lemma 14, we can bound the ˜ B (ˆ µ T , ˆ µ ∞ ) as
[Equation]
where ς = 2 L Θ ( Mbβ + m Θ β + Md ) /m
Θ , L Θ = √ 2 β − 1 L F + l (cid:48) , m Θ = β − 1 m − m (cid:48) , and ( σ, C 2 , l (cid:48) , m (cid:48) ) are some positive constants independent of (T, M, h) and σ ∈ (0 , 1) .
To prove the bound for ˜ B (ˆ µ ∞ , ρ ∞ )
Since ˆ Θ k = (ˆ θ (1) k , · · · , ˆ θ ( M ) k ) can be considered as a solution to the SDE (16), standard results from linear FP equation can be applied.
Speciﬁcally, for the ˜
B (ˆ µ ∞ , ρ ∞ ) term, we rely on the following lemma adapted from Lemma C.4 in Xu et al.
(2018)Chen et al. (2015), and is essentially the work of Chen et al.
(2015) when taking K → ∞ .
Lemma
Under the same assumption as in Lemma 17, for the Lipschitz-continuous function f Θ ( Θ ) = 1 M (cid:80) Mi f
( θ ( i ) ) mentioned above, the following bound is satisﬁed for some positive constant C :
[Equation]
The uniqueness of invariant measure of the Euler method from Lemma 17 implies the numerical solution ˆ Θ k to be ergodic.
Then similar to the proof of Lemma 4.2 in Xu et al.
(2018), we consider the case where K → ∞ .
Take average over the { ˆ Θ k } K =1 k =0 , we have
[Equation]
Now according to the exchangeability of the particle system { ˆ θ ( i ) k } and { θ ( i ) t } , we can bound the ˜ B (ˆ µ ∞ , ρ ∞ ) as :
[Equation]
where C 3 are some positive constant.
K P ROOF OF T HEOREM
Proof [Proof of Theorem 10]
Adopting the same notation used in the proof of the Theorem 5, we deﬁne Θ k (cid:44) [ θ (1) k , · · · , θ ( M ) k ] and G Θ I k (cid:44) NB k (cid:80) q ∈I k F ( q ) Θ ( Θ k ) .
We denote the distribution of Θ k as µ Θk .
[Equation]
We ﬁrstly give a bound to the W 2 ( µ Θk , ˆ µ Θ k ) (the deﬁnition of the ˆ µ Θ k has been mention in the last section).
According to the proof of the Lemma 4.4 in Xu et al.
[Equation]
where Γ (cid:48)
=
2(1 + 1 /m Θ )
( Mb + 2 M 2 C 24 + Md/β )
and C 4 is some positive constant independent of (T, M, h).
Note the fact that W 1 ( µ Θk , ˆ µ Θ k ) ≤ W 2 ( µ Θk , ˆ µ Θ k ) and W 1 ( µ k , ˆ µ k ) ≤ 1 √ M W
[Equation]
ˆ µ Θ
[Equation]
) (see the proof of Lemma 13, similar result holds here), We get
[Equation]
Let us compare the deﬁnition of W 1 ( µ, ν ) and ˜ B ( µ, ν )
[Equation]
and we can derive the result that ˜ B ( µ T , ˆ µ K ) ≤ L f W 1 ( µ T , ˆ µ T ) .
Now we ﬁnish our proof.
L D ISCUSSION ON THE COMPLEXITY OF OUR METHOD
The complexity of an algorithm mainly refers to its time complexity (corresponding to the number of iterations in our method i.e. T ) and space complexity (corresponding to the number of particles used in our method i.e. M ).
Hence the complexity of our method can be well explored with our work, since our non-asymptotic convergence theory was developed w.r.t.
the number of particles i.e. M and iterations i.e. T .
Their relationship (tradeoff) was even discussed further in Experiment 6.1.
Moreover, comparing (9) with (3) , one can easily ﬁnd that our space complexity is exactly the same as SVGD and our computational time in each iteration is almost the same as SVGD with an extra addition operation.
However, it worth noting that our method have much better performance in practice and no "pitfall" veriﬁed by both our theory and our experiments. .
M C OMPARISON WITH R ELATED W ORK
Firstly, our proposed framework SPOS is different from the recently proposed particle-optimization sampling framework (Chen et al., 2018), in the sense that we solve the nonlinear PDE (6) stochastically.
For example they deterministically solve the equation in (6) ∂ν t = β − 1 ∇ θ ·∇ θ ν t approximately using blob method adopted from (Carrillo et al., 2017).
Secondly, our method is also distinguishable to existing work on granular media equations such as (Durmus et al., 2018).
Their work about the Granular media equations focuses on the following PDE:
[Equation]
[Equation]
whereas our framework focuses on the following one:
[Equation]
t (cid:1) . The extra term ν t ( E Y ∼ ν t K ( θ − Y ) F ( Y )) in our framework makes the analysis much more challenging.
The main differences are summarized below:
• Formulations are different.
The extra term E Y ∼ µ t K ( θ − Y ) F ( Y ) cannot be combined with the F ( θ ) term in their (35).
This is because function F ( θ ) itself is a function independent of t ; while E Y ∼ µ t K ( θ − Y ) F ( Y ) depends on both θ and t .
This makes our problem much more difﬁcult.
• Assumptions are different.
For example, the analysis on granular media equations in (Cattiaux et al., 2008) requires that F satisﬁes a special condition C
( A , α ) , which is a strong condition impractical to be satisﬁed in our case; And Durmus et al.
(2018) adopts different assumptions from ours with a different goal.
• For the Euler integrator, Durmus et al.
(2018) does not consider an Euler solution.
Furthermore, our sampling method needs "stochastic gradient" i.e. G ( i ) k
(cid:44) N B k (cid:80) q ∈I k F q ( θ ( i ) k ) in (9) for computational feasibility, which is quite different from the former work on particleSDE such as (Malrieu, 2003; Cattiaux et al., 2008).
Few of the former work on particle-SDE considered the stochastic gradient issue.
To sum up, the main purpose of our paper is to provide a non-asymptotic analysis of our method instead of improving the former work on a certain type of PDE.
This is also the reason why we said that part of our proof techniques are based on those for analyzing granular media equations.
N E XTRA E XPERIMENTS N. T OY E XPERIMENTS
We compare the proposed SPOS with other popular methods such as SVGD and standard SGLD on four mutil-mode toy examples.
We aim to sample from four unnormalized 2D densities p ( z ) /exp
{ U ( z ) } , with the functional form provided in Rezende & Mohamed (2015).
We optimize/sample 50 and 2000 particles to approximate the target distributions.
The results are illustrated in Figure 4 and Figure 5, respectively.
B AYESIAN N EURAL N ETWORKS FOR MNIST CLASSIFICATION
We perform the classiﬁcation tasks on the standard MNIST dataset.
A two-layer MLP 784-X-Xwith ReLU activation function is used, with X being the number of hidden units for each layer.
The training epoch is set to 100.
The test errors are reported in Table 2.
Surprisingly, the proposed
SPOS outperforms other algorithms such as SVGD at a signiﬁcant level, though it is just a simple modiﬁcation of SVGD by adding in random Gaussian noise.
This is partly due to the fact that our
SPOS algorithm can jump out of local modes efﬁciently, as explained in Section 4.
[Table]