U NDERSTANDING
THE V ARIANCE C OLLAPSE OF SVGD IN H IGH D IMENSIONS
Anonymous authors
Paper under double-blind review
A BSTRACT
Stein variational gradient descent (SVGD) is a deterministic inference algorithm that evolves a set of particles to ﬁt a target distribution.
Despite its computational efﬁciency, SVGD often underestimates the variance of the target distribution in high dimensions.
In this work we attempt to explain the variance collapse in SVGD.
On the qualitative side, we compare the SVGD update with gradient descent on the maximum mean discrepancy (MMD) objective; we ﬁnd that the variance collapse phenomenon relates to the bias from deterministic updates present in the “driving force” of SVGD, and empirically verify that removal of such bias leads to more accurate variance estimation.
On the quantitative side, we demonstrate that the variance collapse of SVGD can be accurately predicted in the proportional asymptotic limit, i.e., when the number of particles n and dimensions d diverge at the same rate.
In particular, for learning high-dimensional isotropic Gaussians, we derive the exact equilibrium variance for both SVGD and MMD-descent, under certain empirically veriﬁed near-orthogonality condition on the converged particles, and conﬁrm that SVGD suffers from the “curse of dimensionality”.
I NTRODUCTION
A typical challenge in Bayesian learning is to efﬁciently and accurately learn a complex posterior distribution.
Markov Chain Monte Carlo (Robert and Casella, 2013) provides asymptotically accurate samples, but simulating the chain till convergence can be time-consuming.
On the other hand, variational inference (Wainwright et al., 2008) approximates the intractable target distribution with parametric families by minimizing the KL divergence, which can be more computationally efﬁcient.
Among a plethora of variational approximations, particle-based inference algorithms are particularly compelling as they bypass the inherent distributional assumptions on the variational family.
As a noticeable example, Stein Variational Gradient Descent (SVGD) (Liu and Wang, 2016) is a deterministic particle-based inference algorithm that iteratively transports the particles by the functional gradient of KL divergence in the reproducing kernel Hilbert space (RKHS).
The functional gradient takes the form of a kernelized Stein’s operator and only requires access to the unnormalized target density.
Despite the empirical successes of SVGD (Liu, 2017; Haarnoja et al., 2017; Kim et al., 2018), very few convergence guarantees have been established except in the mean-ﬁeld limit, i.e., the number of particles n → ∞ under ﬁxed dimensionality d , for which it has been shown that the distribution of particles converges to the true invariant solution (Liu, 2017; Lu et al., 2019).
Moreover, it has been observed that as the problem dimensionality d becomes larger, the variance estimated by SVGD can be much smaller than variance of the target distribution (Zhuo et al., 2017).
This observation, which we refer to as the variance collapse phenomenon (see Figure 1), is highly undesirable for practitioners due to two reasons: ( i ) underestimating the variance leads to failure in explaining the uncertainty of model predictions, which is a key beneﬁt of being Bayesian; ( ii ) modern
Bayesian inference problems are usually high-dimensional; for instance, training Bayesian neural networks (BNNs) (MacKay, 1992) requires inferring the posterior distribution of network weights, which could be more than millions of dimensions in real-world problems (Krizhevsky et al., 2012).
Our Contribution.
We study the algorithmic bias of SVGD that leads to variance collapse in high dimensions.
We focus on the commonly-used Euclidean distance kernel (e.g., Gaussian RBF kernel), and provide: ( i ) qualitative understanding on the pitfall of SVGD by comparing it to another interacting particle algorithm termed MMD-descent ; ( ii ) quantitative understanding on the variance underestimation in learning simple target distributions.
Our ﬁndings can be summarized as follows.
• In Section 3 we connect SVGD with MMD-descent, a kernel-based particle inference algorithm that performs maximum mean discrepancy (MMD) minimization, and empirically show that despite their similar updates, MMD-descent does not collapse the variance in high dimensions.
• In Section 4 we identify the log derivative driving force as the problematic term in SVGD, and design experiments to illustrate that the combination of this driving force and the deterministic bias ,
i
e. the absence of particle resampling, prevents accurate variance estimation in high dimensions.
• In Section 5 we argue that the proportional asymptotic limit : n, d → ∞ , d/n → γ ∈ (0 , ∞ ) , is more relevant to understanding the variance collapse phenomenon.
As an example, we derive the exact dimension-averaged variance of SVGD and MMD-descent in learning Gaussian distribution in the proportional limit, under certain concentration assumption which we empirically verify.
Our analysis conﬁrms that variance estimated by SVGD scales inversely with the dimensionality.
B ACKGROUND 2. I NTEGRAL P ROBABILITY M ETRIC
To measure the “closeness” between distributions on X ⊆ R d , one may consider the maximum discrepancy between the target p and sample distribution q over test functions F : D F ( p, q ) :
=
sup h
∈F E q [ h ( x )] − E p [ h ( y )] , known as the integral probability metric (IPM) (M¨uller, 1997).
If F is a unit ball in the reproducing kernel Hilbert space (RKHS) H , the resulting D F is termed the maximum mean discrepancy (MMD) (Gretton et al., 2012), and its squared value can be evaluated as:
[Equation]
where x , x (cid:48) ∼ p , y , y (cid:48) ∼ q , and the kernel k : R d × R d
→ R satisﬁes E (cid:112) k ( x , x ) < ∞ .
Stein’s Discrepancy.
When integration under the target p is intractable, Stein’s method (Stein et al., 1972) can be used to construct zero-mean test functions w.r.t.
p .
For differentiable h in the Stein Class of p , i.e., (cid:82) x ∇ x ( h ( x ) p ( x ))
d x = 0 , the Stein’s discrepancy (Gorham and Mackey, 2015) is given as
[Equation]
Note that the Stein’s discrepancy only involves the score of p and thus the normalization constant is not required.
When h is restricted in the product RKHS H d with inner product (cid:104) f, g (cid:105) H d = (cid:80) di =1 (cid:104) f
i , g i (cid:105) H , the corresponding maximum discrepancy, known as kernel Stein discrepancy (KSD), can be estimated efﬁciently from samples (Liu et al., 2016; Chwialkowski et al., 2016).
2. (D ETERMINISTIC ) P ARTICLE I NFERENCE A LGORITHMS
We now consider the approximation of p using a set of particles constructed as follows: starting from initial particles
X
= { x i } ni =1 , we iteratively optimize their positions using the deterministic update: x i
= x
i + η ∆( x i ) , where η is the step size, and ∆( · ) : R d → R d represents the update direction.
Motivated by the SVGD algorithm, we focus on the setting where the update ∆ is deterministic , in which case particles may converge to a deterministic ﬁxed point.
Note that there are other particlebased inference algorithms that do not ﬁt into this description, such as sequential Monte Carlo (Doucet et al., 2001), and stochastic variants of particle gradient method (Chen et al., 2018a; Gallego and Insua, 2018), which our current analysis does not cover (see Appendix A.4 for more discussion).
Stein Variational Gradient Descent.
SVGD constructs the update direction ∆ as the optimal perturbation in the RKHS that decreases Kullback-Leibler divergence.
In particular, constrain ∆ in
RKHS unit ball, and take q = 1 n (cid:80) ni
=
δ x i , then the update for each particle x can be computed as:
[Equation]
Intuitively, the log derivative term in the update rule S 1 ( x i , x ) :
= k
( x i , x ) ∇ x i log p ( x i ) corresponds to a driving force that guides particles towards high likelihood regions, whereas the kernel derivative term S 2 ( x i , x ) :=
∇ x i k
( x i , x ) provides a repulsive force to prevent the particles from collapsing into the mode.
Typically-used kernels in SVGD include the Gaussian RBF kernel k ( x , x (cid:48) )
=
exp( − (cid:107) x − x (cid:48) (cid:107) 22 / 2 σ 2 ) (Liu and Wang, 2016; Zhuo et al., 2017), the linear kernel k ( x , x (cid:48) )
=
x (cid:62) x (cid:48) + 1 (Liu and Wang, 2018), and the inverse
[Table]
C ONNECTING SVGD WITH MMD M INIMIZATION
In this section, we introduce another particle inference algorithm termed MMD-descent , whose update rule closely relates to that of SVGD.
Despite the similarity, we empirically observe MMD-descent accurately estimates the target variance independent of the dimensionality.
By analyzing the similarity of the update rules and contrast in the algorithmic performances, we identify factors that lead to the variance collapse of SVGD.
In this work we focus on the following class of kernel functions.
Deﬁnition 1 (Euclidean Distance Kernel) .
A positive semi-deﬁnite kernel is referred to as a Euclidean distance kernel if it can be represented as: k
[Equation]
x
[Equation]
(cid:1) .
For example, the Gaussian RBF kernel and IMQ kernel both satisfy this deﬁnition.
Note that σ is the tunable bandwidth that usually scales with the distance between particles.
We now construct a kernel-based particle inference algorithm termed MMD-descent, which draws inspiration from the kernel herding algorithm
(Welling, 2009) and particle gradient descent (Chizat and Bach, 2018).
Instead of greedily reducing the MMD by adding one particle a time, we consider an update rule similar to SVGD that transport all particles together to approximate the target distribution, by minimizing the MMD between the particles and the target via gradient descent:
[Equation]
This update can be seen as the ﬁnite-particle discretization of the MMD gradient ﬂow (Arbel et al., 2019).
For Euclidean distance kernels we have −∇ x k ( x , x (cid:48) )
=
∇ x (cid:48) k
( x ,
[Equation]
) = S
[Equation]
x
[Equation]
, x ) ; hence,
MMD-descent and SVGD share the same repulsive force.
Furthermore, when k is in the Stein class of p , the driving force of MMD-descent can be connected SVGD via a simple integration by parts:
[Equation]
Therefore, the MMD-descent update for a set of particles { x i } ni =1 can be equivalently written as:
[Equation]
We remark that MMD-descent is not a practical algorithm due to the required integration under p , which is typically intractable.
Instead, the purpose of introducing MMD-descent is to compare the update with SVGD and understand the cause of the variance collapse phenomenon.
SVGD vs. MMD-descent.
By comparing SVGD (2) and MMD-descent (4), we observe that:
• SVGD and MMD-descent have identical repulsive force .
• In MMD-descent, the driving force is integrated under the target distribution p , whereas in SVGD the expectation is under the current particle distribution q .
At the inﬁnite particle limit, p = q is a ﬁxed point for both update rules.
In fact, under certain conditions, uniqueness of the ﬁxed point has been established for both updates in this mean-ﬁeld limit (Lu et al., 2019; Arbel et al., 2019).
However, in the practical setting where the particle size is not signiﬁcantly greater than the dimensionality, it is not clear if the two algorithms: ( i ) provide a reasonable “approximation” to the target distribution; ( ii ) converge to similar ﬁxed points.
Given the analogous updates, one natural question to ask is: do SVGD and MMD-descent reliably estimate the target variance in high dimensions, and do they give similar solutions?
The empirical answer is in the negative: SVGD and MMD-descent converge to completely different solutions: SVGD is known to underestimate the marginal variance in high dimensions, even for simple Gaussian targets (Zhuo et al., 2017).
In contrast, MMD-descent does not exhibit the same issue empirically.
Figure 1 illustrates this discrepancy in a Bayesian neural network (BNN) experiment.
We consider a two-hidden-layer BNN with 100 hidden units each, so that inference in the weight space is a high-dimensional problem.
We obtain (approximate) target samples via Hamiltonian Monte
Carlo (HMC) (Neal et al., 2011), and then initialize 100 particles from the target to be optimized by
SVGD or MMD-descent with the Gaussian RBF kernel (for detailed setup see Appendix D).
Observe that although we used the same number of particles for SVGD and MMD-descent, SVGD severely underestimates the variance, whereas MMD-descent generates diverse predictions similar to HMC.
3 input x 1.
m o d e l o u t p u t
HMC (ground truth)      3 input x 1.
m o d e l o u t p u t
SVGD      3 input x 1.
m o d e l o u t p u t
MMD descent
U NDERSTANDING THE P ITFALL OF SVGD
We now qualitatively characterize the algorithmic bias of SVGD.
Motivated by the observation that
SVGD and MMD-descent only differs in the driving force which involves S 1 , we ﬁrst compare Sand S 2 in SVGD and argue that the former term is more likely problematic in high dimensions due to larger ﬂuctuation.
We then design controlled experiments by modifying the updates of SVGD and
MMD-descent to show that when the log derivative S 1 is coupled with the deterministic bias (i.e., absence of particle resampling), the algorithm does not reliably estimate the target variance .
4. H IGH V ARIANCE FROM I NTEGRATION BY P ARTS
Recall that the update rule of SVGD involves a driving force ( log derivative ) term 1 n (cid:80) ni
=1 S 1 ( x i , x ) and a repulsive force ( kernel derivative ) term 1 n (cid:80) ni
=1 S 2 ( x i , x ) .
From integration by parts we have the following equality: E y ∼ p
[ S 1 ( y , x )] =
− E y ∼ p
[ S 2 ( y , x )] .
Thus the target distribution p is a ﬁxed point of SVGD when the number of particles tends to inﬁnity.
However, in the ﬁnite sample setting, the ﬂuctuations in S 1 and S 2 might have impact on the converged solution.
In many settings, we expect the log derivative driving force S 1 to have higher variance.
For instance, in the case of unit Gaussian target p and Gaussian RBF kernel k ( x , x (cid:48) )
=
exp( −(cid:107) x − x (cid:48) (cid:107) 22 / 2 σ 2 ) , we know that S 1 ( y , x ) = − y k
( y , x ) and S 2 ( y , x ) =
σ − 2 ( x − y ) k ( y , x ) .
Roughly speaking, when the norm of x , y and the bandwidth σ scale with √ d , (cid:107) S 1 (cid:107) 22 increases with the dimensionality, whereas (cid:107) S 2 (cid:107) 22 remains bounded.
The large magnitude of S 1 results in large ﬂuctuation in the driving force term in SVGD.
In Figure 2(a), we visualize the distributions of S 1 and S 2 when the particles x are drawn i.i.d.
from Gaussian distributions, and indeed observe the higher variance of S 1 .
The following proposition quantiﬁes this discrepancy in an idealized setting.
Proposition 2.
Deﬁne the mean squared error as: MSE p [ f ( y )]
=
E p ( y ) (cid:107) f
( y ) − E p
[ f ( y )]
(cid:107) 22 .
Then for y ∼ N ( a , I d ) where (cid:107) a (cid:107)
= O ( d ) , Gaussian RBF kernel k ( x , y )=
exp( − (cid:107) x − y (cid:107) 22 / 2 σ 2 ) with bandwidth σ = Θ( √ d ) , and
x ∈ S d − 1 ( √ d ) , we have
[Equation]
More generally, for x ∈ S
d − 1 ( √ d ) , strongly log-concave p , and Euclidean distance kernel with Lipschitz f and lower-bounded by a scaled Gaussian kernel, MSE p
[ S 1 ( y , x )]
= Ω( d ) · MSE p [ S 2 ( y , x )] .
The proposition indicates that when particles are i.i.d.
sampled from some target p , then S 1 would have much larger ﬂuctuation than S 2 1 .
Intuitively speaking, the higher variance in S 1 suggests that more samples are required to estimate the term accurately.
Therefore, when the dimensionality is large compare to the particle size, poor estimation of S 1 may relate to the variance collapse.
The following subsection empirically demonstrate that this is indeed the case; in particular, we show that in the presence of deterministic bias , the high-variance S 1 leads to “biased” particles in SVGD.
4. B IAS FROM D ETERMINISTIC U PDATE
In this subsection we isolate a cause of variance collapse that relates to the deterministic nature of the update rule.
Observe that in the derivation of SVGD, particles { x i } ni =1 are assumed to be sampled
[Figure] [Figure]
from some underlying distribution q .
But due to the deterministic update, q is entirely represented by the same set of particles , and redrawing i.i.d.
samples is not feasible during optimization.
We now demonstrate that this failure to resample, which we refer to as the deterministic bias , when combined with the log derivative driving force S 1 , may cause the algorithm to converge to the wrong target.
Deterministic Bias in MMD-descent.
We start with an illustration of deterministic bias in MMDdescent.
Given samples { y i } mi =1 ∼ p drawn from the target, we have two distinct MMD-gradient updates (connected via integration by parts (3)) that differs in the driving force.
∆ MMD
[Equation]
We refer to ∆ MMDand ∆ MMDas the log derivative update and the kernel derivative update, respectively.
Section 4.1 suggests that ∆ MMDtend to have larger ﬂuctuation than ∆ MMD.
For both update rules, we consider the two variations: ( i ) ﬁxed particles, where we draw { y i } mi =1 in the beginning and use this same set of target samples throughout optimization; ( ii ) resampled particles, where we redraw new i.i.d.
samples { y i } mi =1 from p at each iteration.
In this construction, the ﬁxed-particle update ( i ) emulates the deterministic bias in S 1 , due to the absence of resampling from p .
We simulate the two variants of MMD-descent in Figure 2(b), in which the target is a 50-dimensional unit Gaussian, n =
and we vary m from 50 to 1000 .
First observe that the log derivative update with ﬁxed particles (solid red), which combines the high variance S 1 and the deterministic bias, results in diverged particles.
In contrast, the log derivative update with resampled particles (dashed red), in which S 1 is present but not the deterministic bias, converges to the desired variance.
Moreover, if the driving force is estimated by S 2 , then the variance is accurate even in the presence of deterministic bias, as shown in the kernel derivative update with ﬁxed particles (solid blue).
This indicates that the combination of deterministic bias and the log derivative driving force S 1 leads to biased convergence 2 .
Input: Initial density q 0 .
Number of Steps S .
for s = 1 to S do
Sample { x si } ni =
∼ q 0 ( x ) .
for t = 1 to s − 1 do either Compute S 1 from resampled particles: x si ← x si + ηn (cid:80) nj =1 S 1 ( x tj , x si )+ S 2 ( x
, x si )
Particle Resampling in SVGD.
The previous experiment on MMD-descent suggests that the deterministic bias arises from the algorithm not being able to “redraw” i.i.d.
samples.
We further validate this observation in SVGD by constructing a variant that achieves particle resampling.
The modiﬁed update exhibits a doubleloop structure (see Algorithm 1): at iteration i of the outer loop, we obtain a new set of particles indexed as q i after ( i − 1) inner loop steps.
In the inner loop, we ﬁrst draw n i.i.d.
particles ˆ q i from the initial distribution q 0 , and then update ˆ q i via ( i − 1) SVGD steps to obtain q i ; at each step j of the inner loop, either the driving force S 1 or the repulsive force S 2 is computed between ˆ q i and q j , as opposed to between particles within ˆ q i as in the original SVGD.
We remark that this resampled update resembles transport-based particle algorithms (e.g. Nitanda and
Suzuki (2017)), where old particles deﬁne a transport map that the new particles (initialized from q 0 ) follow.
By construction, at each outer loop iteration, a new set of independent particles are generated and updated; importantly, one of S 1 and S 2 is not evaluated on the same set of particles themselves, and hence deterministic bias is not present.
However, the complexity of such resampled updates scales quadratically with the iterations, which renders the algorithm computationally prohibitive 3 .
Figure 2(c) compares the modiﬁed SVGD updates (with Gaussian RBF kernel) in learning a Gaussian target.
Observe that when the driving force S 1 is computed on resampled particles (blue), SVGD accurately estimates the target variance even with small particle size ( n = 20 ) at each step.
In contrast, if only the repulsive force S 2 is evaluated on particles resampled from q 0 , then the variance is still underestimated (green), due to deterministic bias in S 1 .
This conﬁrms our argument that combination of log derivative S 1 and deterministic bias leads to the algorithmic bias of SVGD.
SVGD IN THE P ROPORTIONAL L IMIT
While the previous discussion provides qualitative understanding of the algorithmic bias of SVGD, no quantitative characterization is provided.
In this section, we make use of the deterministic nature of the SVGD update, and directly analyze the particle ﬁxed point; this allows us to precisely compute the variance of SVGD particles and conﬁrm the variance collapse in simple settings.
5. S CALING L IMIT AND B ASIC A SSUMPTIONS
Previous works have considered the inﬁnite-particle limit of SVGD under ﬁxed dimensionality, which is known as the mean-ﬁeld limit (Lu et al., 2019; Duncan et al., 2019).
In this regime, the particle dynamics of SVGD is described by a nonlinear PDE:
∂ t ρ = ∇ · ( ρ ( K ∗ ( ∇ ρ + ∇ V ρ ))) , which converges weakly to the unique target measure as t → ∞ under certain non-degeneracy assumptions.
However, since the invariant solution is the desired target distribution, the mean-ﬁeld limit does not capture the variance collapse phenomenon.
Moreover, similar guarantees can also be derived for
MMD-descent under similar assumptions (Arbel et al., 2019).
Therefore, the mean-ﬁeld limit does not explain the observed discrepancy between SVGD and MMD-descent (Figure 1 in Section 3).
Instead, we propose to analyze the algorithm in the proportional asymptotic limit :
• (A1) Proportional Limit.
n, d → ∞ and d/n → γ ;
we also assume
γ > 1 .
Note that larger γ is analogous to higher dimensionality or smaller particle size; we focus on the γ > 1 regime, where the number of particles is less than the problem dimensionality — this is a common feature of modern Bayesian inference problems.
In this asymptotic limit, prior works studied spectral properties of the kernel matrix (e.g., El Karoui et al.
(2010)), which the SVGD update crucially depends upon.
However, these random matrix results typically require the particles to be i.i.d., which is not satisﬁed by SVGD due to the interacting update.
To overcome this technical difﬁculty, we make an additional assumption on the ﬁxed point:
• (A2) Near-orthogonality.
The optimized particles { x i } ni =1 satisfy | x (cid:62) i x
[Equation] x [Equation]
i x j | < υ(cid:15) d for all i (cid:54)
=
j , some υ > 0 , and d − 1 / 2 (cid:15) d → 0 as d → ∞ with probability 1.
[Figure]
The above assumption simply ensures that the particles tend to equilibrate at a ﬁxed point where they “repel” one another as much as possible (due to the repulsive force), as shown in Figure 3 for the case of Gaussian target (for more empirical veriﬁcation see
Appendix A.2).
We emphasize that this is a condition only on the ﬁxed point of the SVGD algorithm, not on the particles produced by the SVGD iterations (unless initialized at the ﬁxed point).
Under assumptions (A1) and (A2), we can approximate the Gram matrix via Taylor expansion of the kernel matrix, which allows us to simplify the ﬁxed point equation.
We restrict ourselves to the following class of kernels:
• (A3) Kernel Function.
k ( x , y ) =
f (cid:16) (cid:107) x − y (cid:107) 22 2 σ 2 (cid:17) , where f : R ≥
→ R + is bounded, monotonically decreasing, and differentiable 4 on R ≥ 0 .
Remark.
(A3) covers common choices of Euclidean distance kernels with decaying tail, such as the Gaussian RBF kernel, the IMQ kernel and the log-inverse kernel.
Our analysis also allows for different choices of bandwidth σ , which we specify in the sequel.
5. L EARNING H IGH - DIMENSIONAL G AUSSIANS
We now consider a simple example in which SVGD does not reliably estimate the variance – learning an isotropic Gaussian target (empirically observed in Zhuo et al.
Note that our goal is to demonstrate a negative result : we do not claim that SVGD underestimates the variance in all settings; instead, we show that variance collapse is present even in learning the simplest target distribution.
• (A4) Gaussian Target.
p ( x ) ∝ exp (cid:0) − 12 x (cid:62) x (cid:1) .
Remark.
We assume (A4) for simple and concise presentation, and we believe that the result can be extended to more general sub-Gaussian targets.
In addition, since Gaussian prior is widely used in Bayesian inference, it is natural to expect that target potentials in many real-world problems are “Gaussian-like” (i.e., exhibit quadratic growth) outside of some radius (Cheng et al., 2018a).
Indeed, our empirical observations on BNN align with our theoretical predictions for Gaussian target.
We aim to compute the dimension-averaged marginal variance : v = 1 d (cid:80) dj
=1 Var j ( { x i } ni =1 ) , where
Var j ( · ) is the particle variance at the j -th coordinate.
In the following subsections we derive the equilibrium variance of SVGD and MMD-descent under two different choices of kernel bandwidth.
Kernels with Adaptive Bandwidth.
We ﬁrst consider the case where the bandwidth σ is adaptively tuned based on the optimized particles.
In particular, we analyze the median heuristic (Scholkopf and Smola, 2001): σ = (cid:113)
Med {(cid:107) x
i − x j (cid:107) 22 } / 2 , which is the most common choice made in practice.
Under the previous assumptions, we have the following result for SVGD with median bandwidth.
Proposition 3.
Given (A1-4) and the choice of bandwidth σ = (cid:113)
Med {(cid:107) x
i − x j (cid:107) 22 } / 2 , particles driven by SVGD (Equation (2) ) equilibriate at the dimension-averaged variance
[Equation]
Remark.
The proposition precisely captures our observation that SVGD underestimates the dimension-averaged variance inversely proportional to the problem dimensionality ( γ = d/n ).
In contrast, previous analysis (Zhang et al., 2020) predicts that SVGD would collapse to zero variance as t → ∞ , which does not align with the empirical observations.
For Gaussian kernel, we also provide a precise contrast between SVGD and MMD-descent.
Corollary 4.
Given (A1-4), for the Gaussian RBF kernel f ( x ) = exp ( − x ) with the median bandwidth, SVGD converges to the following dimension-averaged variance: v SVGD =
( e − 1) − 1 · γ − 1 < 1 , whereas MMD-descent (Equation (4) ) leads to v MMD = 1 .
In other words, as the problem dimensionality γ increases, more particles are required for SVGD to reliably estimate the dimension-averaged variance (speciﬁcally, the number of particles should grow linearly with d ).
In contrast, the variance estimated by MMD-descent remains accurate and is independent to γ 5 ; this aligns with the empirical observations in Section 3.
Remarkably, Figure 4(a) demonstrates that Proposition 3 is accurate even for reasonably small n, d : observe that once γ >(i.e
Kernels with Fixed Bandwidth.
One possibility remains that the culprit of variance collapse is the median bandwidth.
Speciﬁcally in kernel Stein discrepancy (KSD), it has been argued that the IMQ kernel can outperform the Gaussian RBF kernel (Gorham and Mackey, 2017), and the IMQ kernel is often employed without an adaptive bandwidth.
Due to the connection between KSD and SVGD, one may speculate that the IMQ kernel with ﬁxed bandwidth can alleviate the variance collapse problem.
[Figure]
We show that this is unfortunately not the case.
We ﬁrst provide a general characterization:
Proposition 5.
Given (A1-4) and ﬁxed bandwidth 6 σ = √ d , the SVGD variance satisﬁes
[Equation]
) − f [Equation] [Equation] In addition, if f (cid:48) is also monotone on R ≥ 0 , then v SVGD decreases as γ > 1 increases.
Note that the monotonicity assumption is again satisﬁed by many Euclidean distance kernels of interest.
While the equation may not provide an explicit expression of the stationary variance, we can verify that the variance decreases as the problem becomes more high-dimensional (i.e., larger γ ).
Our next proposition speciﬁcally handles the IMQ kernel considered in Gorham and Mackey (2017).
Corollary 6.
Given (A1-4), for the IMQ kernel f ( x ) =
(1 + x ) − 1 / 2 with ﬁxed bandwidth, we have the following stationary variance of SVGD under two different scalings:
• When σ = √ d , v SVGD < 1 and is decreasing as γ > 1 increases.
• When σ = 1 , v SVGD → 0 as n, d → ∞ at a rate of d − 1 / 3 .
This corollary suggests that the IMQ kernel with ﬁxed bandwidth is not a remedy to the variance collapse problem.
We remark that the second setting ( σ = 1 ) is considered in Gorham and Mackey (2017) for KSD.
In both cases (and including the median bandwidth covered by Proposition 3), SVGDIMQ underestimates the target variance as the dimensionality increases (large γ ).
The agreement between the theoretical predictions and the empirical simulations (using ﬁnite particles) is illustrated in Figure 4(b)(c).
Finally, in Appendix A we include a more general (but less precise) characterization of variance collapse beyond Gaussian target, as well as additional empirical evidence.
5. A M ODIFICATION OF SVGD
We have thus far shown that in the simple setting of learning high-dimensional Gaussian, SVGD underestimates the dimension-averaged variance unless the number of particles is larger than the dimensionality.
Now we further validate our theoretical ﬁndings by introducing a heuristic modiﬁcation of SVGD that corrects for this variance collapse in the overparameterized regime 7 .
The starting observation is that the variance collapse indicates that the deterministic bias causes the driving force term to dominate.
Because during each update every particle x i is most “correlated” with itself, one should expect S 1 ( x i , x i ) to contribute signiﬁcantly to this bias.
We thus consider a modiﬁcation of SVGD which simply shrinks (damps) the term S 1 ( x i , x i ) by λ = min { 1 , ( f (1) − γ − 1 f (cid:48) (1)) /f
(0) } , where λ is chosen such that when d > n , the equilibrium variance matches the target variance in the setup of Proposition 3.
We refer to this update as damped SVGD :
[Equation]
Remark.
At high level, (5) resembles the annealed SVGD algorithm (D’Angelo and Fortuin, 2021), in that they both ”weaken” the driving force S 1 .
However, annealed SVGD uses a heuristic learning rate schedule (target-independent) to modify the strength of S 1 , whereas we derive our λ to ensure that the dimension-averaged variance is correct under (A1-4) in the d > n regime.
Simulating Toy Densities.
We demonstrate the usefulness of this bias-correction term in learning simple target distributions.
We stack 250 independent 2dimensional distributions to create a high-dimensional target distribution, and visualize the ﬁrst two dimensions in Figure 5.
We optimize 50 particles using
SVGD or SVGD (damped) with the Gaussian RBF kernel and median bandwidth.
As shown in Figure 5 (top), SVGD collapses its particles to the mode, whereas damped SVGD generates more dispersed particles with variance exactly matching that of the Gaussian target.
It is also possible that the heuristic modiﬁcation leads to more accurate samples for other “Gaussian-like” target distributions 8 , as shown in the second row of Figure 5.
We provide additional experimental results in Appendix A.3.
R ELATED W ORKS
Guarantees and Applications of SVGD.
Liu and Wang (2018); Lu et al.
(2019) characterized
SVGD in the mean-ﬁeld limit and showed the weak convergence to the target distribution.
Under further assumptions on the potential and kernel, quantitative convergence rate can be derived (Duncan et al., 2019; Korba et al., 2020).
In addition, Liu and Wang (2018) shows SVGD using kernels with ﬁnite-dimensional feature maps, exactly estimates the expectations for some set of functions, casting
SVGD as a moment matching method.
In the high-dimensional setting, Zhuo et al.
; Wang et al.
(2018) observed in experiments that particles driven by SVGD tend to underestimate the marginal variance, but did not provide any quantitative understanding of the phenomenon.
On the application side, Haarnoja et al.
; Liu (2017) adopt SVGD to learn a stochastic sampling network to approximate the policy in Q-learning (Sutton et al., 1998), whereas in Gangwani et al.
(2018), SVGD encourages diverse policies for exploration.
SVGD can be used in meta-learning to quickly obtain parameter samples from training sets (Yoon et al., 2018).
Recent works also applied
SVGD in Batch Bayesian optimization
(Gong et al., 2019) and the learning of mixture models (Wang and Liu, 2019).
Leveraging the Markov blanket structure, SVGD also achieves strong performance in learning graphical models (Zhuo et al., 2017; Wang et al., 2018).
Stein’s Method.
Stein’s method provides powerful tools in approximating probability distributions and specifying convergence rates (Erdogdu, 2016; Gorham et al., 2016).
Liu and Wang (2016) utilizes the connection between Stein’s operator and the gradient of KL divergence to construct particle inference algorithm.
Stein’s lemma is also useful in implicit variational inference (Husz ´ ar, 2017) to estimate the score using samples from an implicit distribution (Li and Turner, 2017; Shi et al., 2018).
Related to our analysis in Section 4, Erdogdu et al.
(2016) observed that algorithms that are equivalent in expectation via Stein’s lemma might have different convergence properties.
In addition, the “curse of dimensionality” of kernel Stein estimators has also been studied in Oates et al.
C ONCLUSION
We analyzed the variance collapse of SVGD in high dimensions based on a connection between
SVGD and a proposed MMD-descent algorithm.
We qualitatively identiﬁed factors that lead to this phenomenon, and also quantitatively characterized the equilibrium variance in the proportional limit for simple models.
Looking forward, we believe that understanding interacting particle systems (SVGD, two-layer neural nets, etc.)
in the proportional limit instead of the mean-ﬁeld limit is an important direction.
In addition, while our analysis conﬁrms the variance collapse of the original algorithm, it remains possible that certain dimension reduction schemes could alleviate the issue.
Chen and Ghattas (2020); Gorham et al.
Gong et al.
(2020) recently proposed dimension-reduced versions of SVGD, and it would be interesting to theoretically justify these approaches
T ABLE OF C ONTENTS 1 Introduction
2 Background 2.
Integral Probability Metric . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
(Deterministic) Particle Inference Algorithms . . . . . . . . . . . . . . . . . . . .
3 Connecting SVGD with MMD Minimization 4 Understanding the Pitfall of SVGD 4.
High Variance from Integration by Parts . . . . . . . . . . . . . . . . . . . . . . .
Bias from Deterministic Update . . . . . . . . . . . . . . . . . . . . . . . . . . .
5 SVGD in the Proportional Limit 5.
Scaling Limit and Basic Assumptions . . . . . . . . . . . . . . . . . . . . . . . .
Learning High-dimensional Gaussians . . . . . . . . . . . . . . . . . . . . . . . .
A Modiﬁcation of SVGD . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
6 Related Works 7 Conclusion
A Additional Results
A.1 Additional Results for Section 5 . . . . . . . . . . . . . . . . . . . . . . . . . . .
A.2 Empirical Veriﬁcation of (A2) . . . . . . . . . . . . . . . . . . . . . . . . . . . .
A.3 Additional Results for Modiﬁed SVGD . . . . . . . . . . . . . . . . . . . . . . .
A.4 Discussion on Other Sampling Algorithms . . . . . . . . . . . . . . . . . . . . . . A.5 Additional Figures for Section 4 . . . . . . . . . . . . . . . . . . . . . . . . . . .
B Proof of Propositions in Section
C Proof of Propositions in Section
C.1 Gaussian Kernel with the Median Bandwidth
C.2 General Kernel with Adaptive Bandwidth (Median Heuristic)
C.3 General Kernel with Fixed Bandwidth
C.4 “Almost-Gaussian” Target
D Experiment Setup
A A DDITIONAL R ESULTS A. A DDITIONAL R ESULTS FOR S ECTION
We include additional ﬁgures to illustrate our analytic results in Section 5.
In Figure 6 we plot the stationary variance of SVGD and MMD-descent (with the Gaussian RBF kernel and median bandwidth heuristic) under ﬁxed dimensionality d against varying number of particles n .
As shown in Figure 6(a), the particle variance of SVGD scales linearly with 1 /d when γ > 1 (i.e., d > n ), as predicted by Proposition 3 and Corollary 4; this again conﬁrms the variance collapse phenomenon.
When n > d , our analysis cannot predict the stationary variance accurately, but the value empirically approaches the target variance from below as γ decreases.
Non-asymptotic Correction for MMD-descent.
As noted in Section 5, there is a small gap between the empirical particle variance of MMD-descent and the prediction of Proposition 3, due to ﬁnite-particle error in the empirical simulation.
To characterize this discrepancy, we now consider a regime where n is ﬁxed and d is large.
In this case, we show that SVGD collapses all the particles to 0, whereas MMD-descent with the Gaussian RBF kernel can still estimate the marginal variance up to O (1 /n ) error (see Appendix C for derivation).
Proposition 7. Consider the setting of ﬁxed n and d → ∞ , then given (A2)(A3), particles driven by
SVGD (2) collapses to zero variance: v SVGD → 0 , whereas MMD-descent (4) with the Gaussian RBF kernel leads to v MMD →
( n − 1) / ( n + 1) .
Figure 6(b) demonstrates the agreement between the proposition and ﬁnite-particle behavior of MMDdescent.
Observe that the ﬁnite-particle error decays at 1 /n and is independent to the dimensionality.
In other words, our analysis suggests that to achieve non-trivial estimation of the target variance,
SVGD requires Θ( d ) particles, whereas for MMD-descent O d (1) particles sufﬁce.
Based on this observation, we speculate that to match the accuracy of MMD-descent with n particles (in terms of marginal variance), SVGD requires N = nd particle.
This conjecture is empirically conﬁrmed in Figure 6(c): when we increase the number of particles in SVGD by an additional factor of d , then the equilibrium variance on longer depends on the dimensionality and becomes similar to that of MMD-descent with n particles.
This again suggests that SVGD may not be reliable unless the number of particles scale with the problem dimensionality.
[Figure]
Variance Collapse in Almost-Gaussian Target.
In Section 5 we focus on Gaussian target for precise characterization of the stationary variance.
For more general target distributions, if the potential behaves “Gaussian-like” outside of a certain radius (smaller than the true target variance), then under the same assumption on the ﬁxed point particles (A3), we intuitively expect SVGD to underestimate the marginal variance when γ = d/n is large.
Proposition 8.
Let p ( x ) ∝ exp( − f ( x )) be an isotropic density, i.e., E p [ x ] = 0 , E [ xx (cid:62) ]
= I d , and assume f exhibits Gaussian tail growth outside a Euclidean ball with radius 1, then given (A1-3), there exists γ ∗ =
d/n > 1 such that SVGD underestimates the dimension-averaged variance for γ > γ ∗ .
While the proposition above only considers quadratic growth of the potential (similar to Mattingly et al.
, we empirically verify that SVGD underestimates the target variance under anisotropy as well as different tail growth condition.
In Figure 7(a) we construct the target distribution to be a mixture of two Gaussians with mean µ 1 , µ 2 such that µ 1 + µ 2 = 0 , (cid:107) µ 1 − µ 2 (cid:107) 2 = 1 , whereas in Figure 7(b)
we consider a factorized distribution with a cubic-growth potential: log p ( x ) ∝ − (cid:81) di =1 ( x ( i ) ) 3 / 3 .
In both cases we observe similar variance collapse phenomenon of SVGD across different choices of kernel and bandwidth.
Log-inverse Kernel.
We also illustrate the variance collapse phenomenon of SVGD with the loginverse kernel proposed in (Chen et al., 2018b).
This heavy-tailed kernel is deﬁned as k ( x , x (cid:48) )
= (cid:16)
α + log(1 + (cid:107) x − x (cid:48) (cid:107) 22 /σ 2 ) (cid:17) − 1 .
One can easily verify that it is a Euclidean distance kernel satisfying assumption (A2), and thus we also expect variance collapse to occur in learning the unit
Gaussian target (A3).
This is indeed conﬁrmed in Figure 7(c)(d), where we see that the log-inverse kernel with both the median heuristic and ﬁxed bandwidth ( σ = 1 ) underestimates the marginal variance, and in the case of median bandwidth, we can analytically predict the equilibrium variance for γ > 1 via Proposition 3.
[Figure]
A. E MPIRICAL V ERIFICATION OF (A2)
We empirically validate the near-orthogonality assumption in Section 5 in learning Gaussian targets.
In particular
[Figure]
we plot the following quantities for particles driven by SVGD and MMD-descent till convergence (from random initialization):
[Equation]
Note that for (A2) to hold, the above quantity should decay with the dimensionality d
[Figure]
In Figure 8 we validate this hypothesis for both SVGD and MMD-descent on two different kernels: the Gaussian RBF kernel and the IMQ kernel, and two target distributions: unit Gaussian and mixture of Gaussians (MoG).
For the Gaussian RBF kernel we use the median bandwidth heuristic, and for the IMQ kernel we set the bandwidth σ = √ d .
The MoG model consists of two Gaussians with mean µ 1 + µ 2 = 0 , (cid:107) µ 1 − µ 2 (cid:107) 2 = 1 .
For both algorithms we initialize the particles from N (0 , 2 I d ) .
Observe that in all cases the quantities of interest indeed decrease with d .
We leave the rigorous justiﬁcation of this observation as future work.
A. A DDITIONAL R ESULTS FOR M ODIFIED SVGD
In Section 5 we proposed a modiﬁcation of SVGD based on derivations of learning a unit Gaussian target distribution in high dimensions (and kernels with the median bandwidth),
[Equation]
where λ = max { 1 , ( f (1) − γ − 1 f (cid:48) (1))
/f (0) } .
Note that λ
= 1 recovers the original SVGD update, whereas under extreme overparameterization ( p (cid:29) n ) we have λ = f (1) /f
(0) < 1 due to (A3).
In addition, larger λ implies smaller stationary variance of SVGD, vice versa.
In the ﬁgures below we refer to the SVGD update with λ min = min { 1 , ( f (1) − γ − 1 f (cid:48) (1)) /f
(0) } as “fully-damped”, and the update with λ in between λ min and 1 as “intermetidate”.
Bayesian Neural Network.
We now apply this modiﬁed update to the BNN setting discussed in Section 3 (for detailed setup see Section D).
We initialize 100 particles from a standard normal distribution and then optimize them via (modiﬁed) SVGD with the Gaussian RBF kernel and median bandwidth for 50000 steps with learning rate η = 5 e − 3 .
As shown in Figure 9, the original SVGD update (Figure (b), original) signiﬁcantly underestimates the target variance (estimated using HMC), and as we decrease λ , the variance of the predictions gradually increases (Figure (c), intermediate).
Observe that at our proposed damping factor (Figure (d), fully-damped), the modiﬁed update produces diverse samples similar to HMC. 3 input x 1.
m o d e l o u t p u t
HMC (ground truth) (a) HMC (ground truth).
3 input x 1.
m o d e l o u t p u t = (b) SVGD (original).
3 input x 1.
m o d e l o u t p u t =( f 1 10 1 f 0 1 )/ f(c) SVGD (intermediate).
3 input x 1.
m o d e l o u t p u t =( f1 f 0 1 )/ f(d) SVGD (fully-damped).
In real-world problems, it is not likely that our proposed modiﬁcation (derived for Gaussian target) always results in optimal performance.
This being said, Figure 9 suggests that practitioners could tune the damping term λ in the range [ f (1) /f
(0) , 1] and adjust the desired level of uncertainty.
Theoretical justiﬁcation of our modiﬁed update is left as future work.
Bayesian Logistic Regression.
We also conduct additional experiment of the modiﬁed (damped)
SVGD update in a Bayesian logistic regression problem.
Given input Z = { z i } mi =1 ∈ R m × d , labels y =
{ y i } mi =1 ∈ R m , and parameter θ ∈ R d , we model the Bernoulli conditional distribution with probability Pr
( y i {+=
 | z+} i ) = 1 / (1 + exp( − θ (cid:62) z i )) .
We place an isotropic Gaussian prior on θ ; the posterior density is given as
[Equation]
Following Dalalyan (2014), we sample the coordinates of z i from a Rademacher distribution and then normalize the vector by its Euclidean norm; the labels are generated from a Bernoulli distribution with true parameters θ ∗ = 1 d ; we set m = 500 , d = 100 , and the regularization parameter α = 1 .
We report the dimension-averaged marginal variance of the particles (in the n < d regime) optimized by variants of SVGD in Figure 10(b).
We use the Gaussian RBF kernel with the median bandwidth, and for the ”intermediate” update we set λ min = min { 1 , ( f (1) − 2 γ − 1 f (cid:48) (1)) /f
The ground truth variance is estimated by Langevin Monte Carlo (see Appendix A.4 for more discussion).
Figure 10(b) illustrates that for this problem, the original SVGD update (red) underestimates the target variance, similar to the Gaussian case.
However, our heuristic modiﬁcation (fully damped,
[Figure]
blue) does not remove this bias completely – this is not surprising because the correction term is derived only for the isotropic Gaussian target.
In particular, the damped update might overestimate the dimension-averaged variance in the small-particle regime.
Nevertheless, we observe that the proposed modiﬁcation generates more diverse samples (with variance closer to the true value)
[Figure]
To further quantify the convergence of the particles, in Figure 10(a)
we report the MMD between the SVGD particles and (approximate) ground-truth samples obtained from LMC; we employ the
V-statistics MMD with the IMQ kernel (ﬁxed bandwidth).
Observe that the modiﬁed update (blue, green) results in smaller MMD compared to the original algorithm (red).
Complete Removal of S 1 ( x i , x i ) . Recall that the SVGD update can be decomposed into three terms: 1) driving force from each particle itself S 1 ( x i , x i ) ; 2) driving force from other particles (cid:80)
j (cid:54)
=
i S 1 ( x j , x i ) ; 3) repulsive force (cid:80) j S 2 ( x j , x i ) (note that S 2 ( x i , x i )
= 0 ).
Speciﬁcally,
S 1 ( x i , x i ) has a large impact on the update and the variance collapse: in the modiﬁcation above, we are able to generate diverse samples just by decreasing the strength of this single term.
One might speculate that instead of using the speciﬁc λ derived for Gaussian target, an easier alternative would be to completely remove S 1 ( x i , x i ) , i.e., setting λ = 0 .
This is however not the case.
Following the same derivation as in Appendix C, one can see that when λ = 0 , Equationdoes not equilibriate at any ﬁnite variance for γ > 1 .
We empirically validate this ﬁnding in Figure 10(c), where we ﬁx d = 50 and vary the particle size n from 25 to 500, and optimize the particle using (modiﬁed) SVGD with the Gaussian RBF kernel and the median bandwidth.
Observe that while the original SVGD (red) underestimates the target variance, setting λ = 0 (blue) leads to signiﬁcant overestimation or even divergence when d > n .
One minor remark is that heuristically speaking, SVGD with λ = 0 roughly resembles MMD-descent with ﬁxed log derivative S 1 (see Figure 2(b) in Section 4), due to the presence of deterministic bias, and that the driving force in both updates involves the log derivative S 1 , but does not include the “self” term S 1 ( x i , x i ) .
As a result, we observe an overestimation of marginal variance (or diverging particles) in both modiﬁed updates.
A. D ISCUSSION ON O THER S AMPLING A LGORITHMS
In addition to SVGD and MMD-descent, there are many other approaches to draw (approximate) samples from a given target distribution. We brieﬂy discuss two popular methods: Langevin Monte Carlo (LMC) and Herding. In particular, we show that LMC does not suffer from variance collapse in high dimensions; in fact, under some conditions, one particle sufﬁces to estimate the dimensionaveraged variance of the target.
Langevin algorithm. Langevin Monte Carlo (LMC) is a time discretization of the Langevin diffusion, which can be interpreted as the (Wasserstein) gradient ﬂow of relative entropy (Jordan et al., 1998). Convergence rate of the discrete-time algorithm has been analyzed under various metrics (Dalalyan, 2014; Durmus and Moulines, 2017; Erdogdu et al., 2021) and assumptions on the target distribution (Vempala and Wibisono, 2019; Erdogdu and Hosseinzadeh, 2020). Here we provide a short sketch that existing quantitative convergence guarantee of LMC implies that the algorithm does not underestimate the dimension-averaged variance.
For strongly convex and smooth potentials (which includes the Gaussian setting in Section 5), Erdogdu et al. (2021, Theorem 4) showed that when the strong convexity parameter is dimension-free, then under appropriate choice of step size, running LMC for T = [-1 / (cid:113) 
-]˜ O ( d ) iterations leads to χ 2 ( q T || p ) = O (1) .
We denote the covariance E p [ xx (cid:62) ] = Σ , and assume E p [ x ] = 0 , Tr(Σ)
= d WLOG. By properties of the χ 2 -divergence and strongly log-concave concentration, given one sample x q drawn from q T ,
[Equation]
This concentration directly implies that in the high-dimensional limit d → ∞ (Assumption 1), we have (cid:107) x q (cid:107) 22 / Tr(Σ) p → 1 . In other words, the dimension-averaged variance Tr(Σ) /d can be reliably estimated using one single particle driven by LMC for ˜ O
( d ) steps.
We remark that similar results are expected to hold true under more general conditions such as the log-Sobolev inequality on the target distribution, as well as other variants of the algorithm (Roberts and Tweedie, 1996; Cheng et al., 2018b). We therefore speculate that variance collapse is a unique property of certain interacting particle algorithms such as SVGD.
Kernel herding.
Consider the approximation of an intractable distribution p ( x ) with a set of particles X
= { x i } ni =1 .
To generate these particles, the kernel herding algorithm was introduced by
Welling (2009) to minimize the MMD between the particles and the target distribution.
The algorithm proceeds in a greedy manner: given the current set of selected particles { x 1 , · · · , x n − 1 } , the next particle is chosen based on the following:
[Equation]
Intuitively, the ﬁrst term encourages sampling in high density areas for the target, whereas the second term discourages drawing samples close to existing ones.
It has been shown that the kernel herding algorithm reduces the MMD at a rate O( 1 N ) for ﬁnite-dimensional Hilbert spaces H (Welling, 2009;
Bach et al., 2012; Husz ´ ar and Duvenaud, 2012).
Note that the same procedure can also be used to greedily minimize the kernel Stein discrepancy (Chen et al., 2018b; 2019).
In addition, the non-greedy gradient descent update on KSD (analogous to MMD-descent) has been recently analyzed (Korba et al., 2021).
We leave the high-dimensional characterization of these algorithms as future work.
A. A DDITIONAL F IGURES FOR S ECTION
{+In Section 4 we conducted experiments that qualitatively identiﬁed the cause of variance collapse, primarily for the Gaussian RBF kernel. Here we reproduce these ﬁndings for the IMQ kernel: k ( x , y ) = 1 / (cid:113) 
+}
+ (cid:107) x − y (cid:107) 22 / 2 σ 2 ) , which is also commonly-used in SVGD.
As shown in Figure 11, results are qualitatively similar to that of Gaussian RBF kernel.
(a) MSE in estimating S 1 and S 2 (IMQ).
(b) MMD-descent (IMQ).
[Figure]
[Figure]
Finally, we provide additional empirical evidence that SVGD with proper resampling procedure (see algorithm 1) reliably estimates the target variance.
We consider the cubic-growth potential: p ( x ) ∝ exp (cid:16) − (cid:81) di
=1 ( x ( i ) ) 3 / 3 (cid:17) , and optimize the particles using SVGD with the Gaussian RBF kernel (ﬁxed bandwidth σ = √ d ).
As shown in Figure 12, the original SVGD update (red) underestimates the dimension-averaged marginal variance inversely proportional to the problem dimensionality; in contrast, when S 1 is resampled (blue), then the estimated variance is stable as d increases.
B P ROOF OF P ROPOSITIONS IN S ECTION
Proof of Proposition 2.
For simplicity we ﬁrst assume the target is zero-centered: y ∼ N (0 , I d ) .
In this case when the kernel is Gaussian RBF, the expectation of S 1 , which we denote as µ x , has the following closed-form: µ x = −
[Equation]
x .
We compute the mean squared error of interest as follow:
[Equation]
Similarly, for the kernel derivative S 2 we have,
[Equation]
x (cid:107) 22 1+ σ(1 + σ 2 )(cid:0) σ1 + σ(cid:1) d (cid:107) x (cid:107) 22 .
The simpliﬁcation above largely follows from E x ∼N ( µ, Σ) [ (cid:107) x (cid:107) 22 ] = µ T µ + Tr(Σ) .
For non-centered y ∼ N ( a , I d ) , note that replacing (cid:107) x (cid:107) 22 with (cid:107) x − a (cid:107) 22 does not affect the order dependence on d as long as (cid:107) a (cid:107)
= O ( d ) , and therefore the order that we aim to estimate remains the same.
Given the bandwidth heuristic σ = Θ( √ d ) and (cid:107) x (cid:107)
=
d , one can easily verify that:
[Equation]
To extend the result to general Euclidean distance kernel k ( x , y ) =
f (cid:16) (cid:107) x − y (cid:107) 22 2 σ 2 (cid:17) and strong logconcave distributions satisfying (cid:13)(cid:13) E p ( y )
[ y ] (cid:13)(cid:13) 2
= O ( d ) and E p ( y )
[ (cid:107) y (cid:107) 22 ]
= O ( d ) , ﬁrst note that
[Equation]
where (i) is by Cauchy-Schwarz and the Lipschitzity of f .
Similarly, following the expansion of
MSE p [ S 2 ] and the assumptions that k is upper-bounded, one can verify that MSE p [ S 2 ( y , x )]
= O (1) .
Finally, we lower-bound the variance of S 1 via the following calculation,
[Equation]
where (i) is by the strongly log-concavity of p and the fact that the kernel being lower-bounded by a scaled Gaussian RBF kernel, and (ii) directly follows from result in the Gaussian case.
Combining the calculations yields the desired statement.
C P ROOF OF P ROPOSITIONS IN S ECTION
In this section we aim to calculate the equilibrium variance of SVGD and MMD-Descent under the proportional asymptotics.
We ﬁrst restate and comment on the Assumptions in Section 5.
• (A2) Near-orthogonality.
Particles at ﬁxed point of SVGD (or MMD-descent) { x i } ni =1 satisfy [Equation]
d [Equation] | [Equation]
(cid:62) i x j | < υ(cid:15) d for all i (cid:54)
=
j , some υ > 0 , and d − 1 / 2 (cid:15) d → 0 as d → ∞ with probability 1. • (A4) Gaussian Target.
p ( x ) ∝ exp (cid:0) − 12 x (cid:62) x (cid:1) .
Under (A4) the goal of SVGD or MMD-descent is to draw samples from a unit Gaussian distribution, and we aim to calculate the dimension-averaged variance v of the particles evolved by the two updates.
Note that since both SVGD and MMD-descent form an interacting particle system, one can no longer treat the converged particles as i.i.d Gaussian samples, i.e. x i ∼ N ( 0 , vI ) .
This signiﬁcantly complicates the analysis in the proportional limit.
However, empirical results (see Figure 8) demonstrate that starting from non-degenerate random initialization, the equilibrium solution of both algorithms admits thin-shell concentration around the radius, i.e., the norm of x i concentrates around √ dv , and particles are almost orthogonal to one another.
This observation is captured by (A2).
We conjecture that such property holds for the updates we consider due to the repulsive force pushing the particles away from one another, and thus leading to this near-orthogonal conﬁguration.
Under assumptions (A2)(A4), we are able to compute the stationary variance of both SVGD and
MMD-descent in the asymptotic limit.
We ﬁrst provide a detailed derivation for the case of Gaussian
RBF kernel, and then extend the analysis to more general settings.
C. G AUSSIAN K ERNEL WITH THE M EDIAN B ANDWIDTH
Stationary Variance of SVGD in Proportional Limit.
We aim to solve the stationary point of
SVGD update (2).
Under (A2) and the median heuristic σ = (cid:113)
Med {(cid:107) x
i − x j (cid:107) 22 } / 2 , the ﬁxed point condition for the Gaussian RBF kernel is given as
[Equation]
for all k , or equivalently
[Equation]
Note that for the left hand side of (7), we have the following equivalence,
[Equation]
where X =
[ x 1 , · · · , x n ] ∈ R d × n is the data matrix, k k
=
[ k ( x 1 , x k ) , · · · , k ( x n , x k )] (cid:62) ∈ R n is the k -th column of kernel Gram matrix K ∈ R n × n .
As for the RHS of Equation (7), note that for i (cid:54)
=
k , (A2) allows us to take the following Taylor expansion on entries of the kernel matrix around its concentrated value, i.e.,
[Equation]
Where (cid:15) · d 1 / 2 → 0 .
Similarly for i
=
k we have k ( x k , x k )
≈ 1 .
Combining the equations above,
RHS
[Equation]
Equating the RHS and LHS of Eq (7) in matrix form (over all k ) we have
[Equation]
where diag( (cid:15) ) is a squared matrix where the i -th diagonal element is the error from Taylor expansion (cid:15)
i
=
O
( (cid:15) ) .
Deﬁne
[Equation]
The ﬁxed point of SVGD thus simpliﬁes to
[Equation]
Denote A = K − mI n − diag( (cid:15) ) , Recall that the K is an Euclidean kernel matrix with K ij = k
( x i , x k ) =
exp (cid:16) − [Equation] dv ) − 1 (cid:107) x
i − x k (cid:107) 22 (cid:17) .
From Theorem 4 in Bordenave et al.
, it follows that the empirical spectrum of A , which we write as µ ( A ) = n − 1 (cid:80) ni
=
δ λ i ( A ) , converges weakly to the following quantity,
[Equation]
In addition, deﬁne S = n/ ( n − 1) I n − 1 / ( n − 1) 1 n 1 (cid:62) n , then by the Hoffman-Wielandt inequality (see Bordenave et al.
(2013, Lemma 6))
we have
W
[Equation]
where we used (A2), and W 2 ( · , · ) is the 2-Wasserstein distance.
Hence, we know that
[Equation]
When γ > 1 (i.e. d > n ), Equation (8) requires µ ( A ) → 0 .
Consequently,
[Equation]
Therefore, from the deﬁnition of m we have the desired result.
[Equation]
Stationary Variance of MMD-descent in Proportional Limit.
First note that for the Gaussian
RBF kernel, the driving force of MMD-descent (4) admits the following closed-form,
[Equation]
This can be veriﬁed via simple numerical calculation:
[Equation]
Therefore, the stationary point of MMD-descent satisﬁes
[Equation]
∀ k , or equivalently,
[Equation]
Under assumptions (A2)(A3), similar to the SVGD case, we have the matrix form of the ﬁxed point,
[Equation]
with (cid:15) =
o (1) .
Following an anlogous calculation, as n, d → ∞ with d/n = γ ∈ (1 , ∞ ) we obtain,
[Equation]
Moreover, observe that lim d →∞ ( dv/ (1 + dv )) d/ 2+1 =
e − 1 / (2 v ) .
We arrive at the desired result, v MMD → 1 .
Following the exact same reasoning, one can show that for ﬁxed bandwidth σ = c √ d for some constant c ∈ Θ(1) , MMD-descent also estimates the variance correctly, i.e., v MMD = 1 .
SVGD and MMD-descent under ﬁnite n and large d .
From (A2) we have the following decomposition for i (cid:54) =
j ,
[Equation]
For ﬁnite n , this indicates that the Euclidean distance between two particles concentrates around 2( n − 1) − 1 ndv (instead of 2 dv ); this is to say, the Gaussian RBF kernel admits the the following
Taylor expansion for i (cid:54) =
j ,
[Equation]
Therefore, for the MMD-descent algorithm, (11) reduces to
[Equation]
Hence the converged particles of MMD-descent with ﬁnite n and inﬁnite d has variance [Equation] n
[Equation]
And for SVGD, one can use the exact same argument to obtain that
[Equation]
Note that this result agrees with our characterization in the proportional asymptotic limit by taking γ → ∞ (i.e., d (cid:29) n ).
C. G ENERAL K ERNEL WITH A DAPTIVE B ANDWIDTH (M EDIAN H EURISTIC )
Recall the deﬁnition of the (bandwidth-adjusted) Euclidean distance kernel: k ( x , y
[Equation] (cid:107)
[Equation]
(cid:107)
[Equation]
(cid:17) .
Under (A2) and the median heuristic σ = (cid:113)
Med {(cid:107) x
i − x j (cid:107) 22 } / 2 , we know that the kernel can be equivalently written as k ( x , y )
=
f (cid:16) (cid:107) x − y (cid:107) 22 2 dv (cid:17) .
Following the same procedure, we have the following ﬁxed point condition,
[Equation]
for all k , or equivalently
[Equation]
Similar to the previous calculation on the Gaussian RBF kernel, applying (A2) and the differentiability of f around [Equation] we Taylor-expand the kernel and obtain the following equivalence:
[Equation]
where K is the Gram matrix of the kernel k , K (cid:48) is the Gram matrix of its derivative with K (cid:48) ij = f (cid:48) (cid:16) (cid:107) x
i − x j (cid:107) 22 2 dv (cid:17) .
Apply Theorem 4 of Bordenave et al.
to K and K (cid:48) , we have the following relation under (A1)(A2),
[Equation]
which gives the expression of the equilibrium variance,
[Equation]
Heuristic derivation of the modiﬁed (damped) update. We provide a brief sketch of the modiﬁed algorithm in Section 5 for the median bandwidth. Recall that our proposed update introduces a damping term λf (0) x k in the driving force S 1 . To derive the optimal λ for unit Gaussian target, we incorporate this damping term into the ﬁxed point Equation (12), which leads to the following modiﬁcation of Equation (13):
[Equation] Recall the true target variance v = 1 ; this gives following choice of λ : [Equation] Since our goal is to “weaken” the driving force S 1 ( x k , x k ) , we take the minimum between the derived value in (14) and the default λ = 1 .
C. G ENERAL K ERNEL WITH F IXED B ANDWIDTH
We now consider Euclidean distance kernel with invariant (ﬁxed) bandwidth that scales with the dimensionality d , which we write as k ( x , y )
=
f ( (cid:107) x − y (cid:107) 2 2 / 2 d ) .
Following (12), we have the following equilibrium condition for the particles, n
[Equation]
Under Taylor expansion around v , the ﬁxed point condition entails that the stationary variance satisﬁes,
[Equation]
Note that given (A3) and monotone f (cid:48) , the numerator of the RHS of (15) takes a negative value that increases with v , whereas the denominator is also negative but decreases with v .
This implies that when γ becomes larger, v needs to decay towards 0 in order to satisfy the equation.
For the Gaussian RBF kernel with ﬁxed bandwidth σ = √ d , the equation can be easily solved as,
[Equation]
which is a decreasing function of γ > 1 .
On the other hand, for the IMQ kernel with ﬁxed bandwidth σ = √ d , standard calculation yields,
[Equation]
One can numerically verify that the value is less than 1 for γ > 1 and also non-increasing.
IMQ kernel with dimension-independent bandwidth.
Finally, we note that in the context of kernel Stein discrepancy (KSD), the IMQ kernel is often employed without the dimensiondependent bandwidth (Gorham and Mackey, 2017).
In this case we write the kernel as k ( x , y ) =
f ( (cid:107) x − y (cid:107) 22 / 2) , which gives the following stationary condition,
[Equation]
Taking Taylor expansion around dv gives
[Equation]
Note that the LHS is Θ(1) by (A2); in order for the inequality to hold asymptotically, we need to have f (cid:48) ( vd ) =
Θ( n − 1 ) .
On the other hand, for the ﬁxed-bandwidth IMQ kernel,
[Equation]
This implies that as d increases, the stationary variance decays to 0 at a rate of v =
Θ( d − 1 / 3 ) .
Following the exact same procedure, one can show that the log-inverse kernel with ﬁxed dimensionindependent bandwidth also asymptotically collapses the variance of SVGD particles to 0 when γ > 1 ; we omit the derivation.
C. “A LMOST -G AUSSIAN ” T ARGET
For general p ( x ) ∝ exp( − f ( x )) , the ﬁxed point equation of x k is given as,
[Equation]
Assume that SVGD does not underestimate that marginal variance, then by (A2) and the quadratic growth, we know that there exists some α > 0 such that for every particle x k and coordinate m ∈ [ d ] , (cid:34) n (cid:88)
[Equation]
For kernels with the median bandwidth, followings the same simpliﬁcation as in (13), we get
[Equation]
Solving the inequality yields,
[Equation]
Therefore, given any dimension-independent growth of the potential α , there exists a large enough γ such that v < 1 , i.e., SVGD underestimates the marginal variance.
The case for ﬁxed bandwidth follows from the same line of reasoning, the details of which we omit.
D E XPERIMENT S ETUP
Bayesian Neural Network.
We consider a BNN with two hidden layers of 100 units.
In each layer, the preactivations s t +1 are computed via s t +
=
( W t a t + b t ) / √ h t + 1 , where a t ∈ R h t are the input activations.
The target function f is a BNN with the same architecture, whose weights and biases are randomly generated from standard normal distributions.
For the training set, we sampled 10 input locations uniformly from [ − 2 . 5 , − 1 . 5] and [1 . 5 , 2 . 5] , respectively.
We add random noises to the observations, y = f
( x ) + (cid:15), (cid:15) ∼ N (0 , 0 . 01) .
We ﬁrst adopt the Hamiltonian Monte Carlo (HMC) (Neal et al., 2011) to generate asymptotic posterior samples.
HMC used 50 independent chains and each chain selected 50 particles with the frequency of 100 iterations after 5 k burn-in iterations.
HMC generates diverse particles as shown in
Figure 1(a).
We then simulate SVGD and MMD-descent dynamics on this problem.
For both updates we use the Gaussian RBF kernel with the median bandwidth heuristic.
For SVGD, we evolveparticles for 50 k iterations using learning rate η = 5 e − 3 , whereas for MMD-descent, we evolve 10 particles using HMC particles as approximate target samples to compute the driving force term.
The particles are either initialized from the (approximate) target distribution (as in Figure 1), or from standard normal distribution (as in Figure 9).
Note that the plots do not include observation variance.
Hyperparameter Setting in Section 5.
For all experiments in Section 5, we initialize the particles from N (0 , 0 .
8 I d ) , and run SVGD (or MMD-descent) with learning rate η = 10 − 1 for 20 k iterations.
For experiments in the proportional limit (Figure 4(a)(b)), we ﬁx n = 50 and vary d .
For the median heuristic (following Garreau et al.
, we compute the median of Euclidean distance between all particles at each iteration.