A CCELERATING C ONVERGENCE OF R EPLICA E XCHANGE S TOCHASTIC G RADIENT MCMC VIA V ARIANCE R EDUCTION
A BSTRACT
Replica exchange stochastic gradient Langevin dynamics ( reSGLD ) has shown promise in accelerating the convergence in non-convex learning; however, an excessively large correction for avoiding biases from noisy energy estimators has limited the potential of the acceleration.
To address this issue we study the variance reduction for noisy energy estimators, which promotes much more effective swaps.
Theoretically we provide a non-asymptotic analysis on the exponential convergence for the underlying continuous-time Markov jump process; moreover, we consider a generalized Girsanov theorem which includes the change of Poisson measure to overcome the crude discretization based on the Gr¨onwall’s inequality and yields a much tighter error in the 2-Wasserstein ( W 2 ) distance.
Numerically, we conduct extensive experiments and obtain state-of-the-art results in optimization and uncertainty estimates for synthetic experiments and image data
I NTRODUCTION
Stochastic gradient Monte Carlo methods (Welling & Teh, 2011; Chen et al., 2014; Li et al., 2016) are the golden standard for Bayesian inference in deep learning due to their theoretical guarantees in uncertainty quantiﬁcation (Vollmer et al., 2016;
Chen et al., 2015) and non-convex optimization (Zhang et al., 2017).
However, despite their scalability with respect to the data size, their mixing rates are often extremely slow for complex deep neural networks with rugged energy landscapes (Li et al., 2018).
To speed up the convergence, several techniques have been proposed in the literature in order to accelerate their exploration of multiple modes on the energy landscape, for example, dynamic temperatures (Ye et al., 2017) and cyclic learning rates (Zhang et al., 2020), to name a few.
However, such strategies only explore contiguously a limited region around a few informative modes.
Inspired by the successes of replica exchange, also known as parallel tempering, in traditional Monte Carlo methods (Swendsen & Wang, 1986; Earl & Deem, 2005), reSGLD (Deng et al., 2020) uses multiple processes based on stochastic gradient Langevin dynamics (SGLD) where interactions between different SGLD chains are conducted in a manner that encourages large jumps.
In addition to the ideal utilization of parallel computation, the resulting process is able to jump to more informative modes for more robust uncertainty quantiﬁcation.
However, the noisy energy estimators in mini-batch settings lead to a large bias in the na¨ıve swaps, and a large correction is required to reduce the bias, which yields few effective swaps and insigniﬁcant accelerations.
Therefore, how to reduce the variance of noisy energy estimators becomes essential in speeding up the convergence.
A long standing technique for variance reduction is the control variates method.
The key to reducing the variance is to properly design correlated control variates so as to counteract some noise.
Towards this direction, Dubey et al.
; Xu et al.
(2018) proposed to update the control variate periodically for the stochastic gradient estimators and Baker et al.
(2019) studied the construction of control variates using local modes.
Despite the advantages in near-convex problems, a natural discrepancy between theory (Chatterji et al., 2018; Xu et al., 2018; Zou et al., 2019b) and practice (He et al., 2016; Devlin et al., 2019) is whether we should avoid the gradient noise in non-convex problems .
To ﬁll in the gap, we only focus on the variance reduction of noisy energy estimators to exploit the theoretical accelerations but no longer consider the variance reduction of the noisy gradients so that the empirical experience from stochastic gradient descents with momentum (M-SGD) can be naturally imported.
In this paper we propose the variance-reduced replica exchange stochastic gradient Langevin dynamics (VR-reSGLD) algorithm to accelerate convergence by reducing the variance of the noisy energy estimators.
This algorithm not only shows the potential of exponential acceleration via much more effective swaps in the non-asymptotic analysis but also demonstrates remarkable performance in practical tasks where a limited time is required; while others (Xu et al., 2018; Zou et al., 2019a) may only work well when the dynamics is sufﬁciently mixed and the discretization error becomes a major component.
Moreover, the existing discretization error of the Langevin-based Markov jump processes (Chen et al., 2019; Deng et al., 2020; Futami et al., 2020) is exponentially dependent on time due to the limitation of Gr ¨ onwall’s inequality.
To avoid such a crude estimate, we consider the generalized Girsanov theorem and a change of Poisson measure.
As a result, we obtain a much tighter discretization error only polynomially dependent on time .
Empirically, we test the algorithm through extensive experiments and achieve state-of-the-art performance in both optimization and uncertainty estimates.
[Figure]
P RELIMINARIES
A common problem, in Bayesian inference, is the simulation from a posterior P( β | X )
∝
P( β ) (cid:81) Ni
=1 P( x i | β ) , where P( β ) is a proper prior, (cid:81)
Ni
=1 P( x i | β ) is the likelihood function and
N is the number of data points.
When N is large, the standard Langevin dynamics is too costly in evaluating the gradients.
To tackle this issue, stochastic gradient Langevin dynamics (SGLD) (Welling & Teh, 2011) was proposed to make the algorithm scalable by approximating the gradient through a mini-batch data B of size n such that
[Equation]
where β k ∈ R d , τ denotes the temperature, η k is the learning rate at iteration k , ξ k is a standard
Gaussian vector, and L ( · ) :
= − log P( β | X ) is the energy function.
SGLD is known to converge weakly to a stationary Gibbs measure π τ ( β ) ∝ exp ( − L ( β ) /τ ) as η k decays to 0 (Teh et al., 2016).
The temperature τ is the key to accelerating the computations in multi-modal distributions.
On the one hand, a high temperature ﬂattens the Gibbs distribution exp ( − L ( β ) /τ )
(see the red curve in
Fig.1(a))
and accelerates mixing by facilitating exploration of the whole domain, but the resulting distribution becomes much less concentrated around the global optima.
On the other hand, a low temperature exploits the local region rapidly; however, it may cause the particles to stick in a local region for an exponentially long time, as shown in the blue curve in Fig.1(a,b).
To bridge the gap between global exploration and local exploitation, Deng et al.
(2020) proposed the replica exchange
SGLD algorithm (reSGLD), which consists of a low-temperature SGLD to encourage exploitation and a high-temperature SGLD to support exploration
[Equation]
where the invariant measure is known to be π ( β (1) , β (2) )
∝ exp (cid:16) − [Equation]
as η k [Equation]
and τ (1) < τ (2) .
Moreover, the two processes may swap the positions to allow tunneling between different modes.
To avoid inducing a large bias in mini-batch settings, a corrected swapping rate (cid:98) S is developed such that
[Equation]
where (cid:98) σ 2 is an estimator of the variance of Nn (cid:80)
i ∈
B k L ( x i | β (1) k ) − Nn (cid:80) i
∈
B k L ( x i | β (2) k ) and F is the correction factor to balance between acceleration and bias.
In other words, the parameters switch the positions from ( β (1) k , β (2) k ) to ( β (2) k , β (1) k ) with a probability r (1 ∧ (cid:98) S )
η k , where the constant r is the swapping intensity and can set to 1 η k for simplicity.
From a probabilistic point of view, reSGLD is a discretization scheme of replica exchange Langevin diffusion (reLD) in mini-batch settings.
Given a smooth test function f and a swapping-rate function
S , the inﬁnitesimal generator L S associated with the continuous-time reLD follows L
[Equation]
[Equation]
[Equation]
[Equation]
[Equation]
2) , β [Equation] [Equation] f
[Equation]
β (1) , β (2) )) , where the last term arises from swaps and ∆ β ( · ) is the the Laplace operator with respect to β ( · ) .
Note that the inﬁnitesimal generator is closely related to Dirichlet forms in characterizing the evolution of a stochastic process.
By standard calculations in Markov semigroups (Chen et al., 2019), the
Dirichlet form E S associated with the inﬁnitesimal generator L S follows
[Equation]
which leads to a strictly positive acceleration under mild conditions and is crucial for the exponentially accelerated convergence in the W 2 distance (see Fig.1(c))
However, the acceleration depends on the swapping-rate function S and becomes much smaller given a noisy estimate of N n (cid:80) i ∈
B L ( x i | β ) due to the demand of large corrections to reduce the bias.
V ARIANCE R EDUCTION IN R EPLICA E XCHANGE S TOCHASTIC G RADIENT L ANGEVIN D YNAMICS
The desire to obtain more effective swaps and larger accelerations drives us to design more efﬁcient energy estimators.
A na¨ıve idea would be to apply a large batch size n , which reduces the variance of the noisy energy estimator proportionally.
However, this comes with a signiﬁcantly increased memory overhead and computations and therefore is inappropriate for big data problems.
A natural idea to propose more effective swaps is to reduce the variance of the noisy energy estimator
L ( B | β ( h ) ) =
Nn (cid:80) i ∈
B L ( x i | β ( h ) ) for h ∈ { 1 , 2 } .
Considering an unbiased estimator L ( B | (cid:98) β ( h ) ) for (cid:80) Ni =
1 L ( x i | (cid:98) β ( h ) ) and a constant c , we see that a new estimator (cid:101) L ( B | β ( h ) ) , which follows [Equation]
[Equation]
is still the unbiased estimator for (cid:80) Ni
=1 L ( x i | β ( h ) ) .
By decomposing the variance, we have
[Equation]
In such a case, Var( (cid:101) L ( B | β ( h ) )) achieves the minimum variance (1 − ρ 2 )
Var( L ( B | β ( h ) ))
given c (cid:63) :
= − Cov ( L ( B | β ( h ) ) ,L ( B | (cid:98) β ( h ) ))
Var( L ( B | (cid:98) β
( h ) )) , where Cov ( · , · ) denotes the covariance and ρ is the correlation coefﬁcient of L ( B | β ( h ) )
and L ( B | (cid:98) β ( h ) ) .
To propose a correlated control variate, we follow Johnson & Zhang (2013) and update (cid:98) β
( h ) =
β ( h ) m
(cid:98) km (cid:99) every m iterations.
Moreover, the optimal c (cid:63) is often unknown in practice.
To handle this issue, a well-known solution (Johnson & Zhang, 2013) is to ﬁx c = − 1 given a high correlation | ρ | of the estimators and then we can present the VR-reSGLD algorithm in Algorithm 1.
Since the exact variance for correcting the stochastic swapping rate is unknown and even time-varying, we follow Deng et al.
and propose to use stochastic approximation (Robbins & Monro, 1951) to adaptively update the unknown variance.
Variants of VR-reSGLD
The number of iterations m to update the control variate (cid:98) β ( h ) gives rise to a trade-off in computations and variance reduction.
A small m introduces a highly correlated control variate at the cost of expensive computations; a large m , however, may yield a less correlated control variate and setting c = − 1 fails to reduce the variance.
In spirit of the adaptive variance in
Deng et al.
(2020) to estimate the unknown variance, we explore the idea of the adaptive coefﬁcient (cid:101)
c k
= (1 − γ k ) (cid:101) c
k − m + γ k c k such that the unknown optimal c (cid:63) is well approximated.
We present the adaptive VR-reSGLD in Algorithm 2 in Appendix E.2 and show empirically later that the adaptive
VR-reSGLD leads to a signiﬁcant improvement over VR-reSGLD for the less correlated estimators.
A parallel line of research is to exploit the SAGA algorithm (Defazio et al., 2014) in the study of variance reduction.
Despite the most effective performance in variance reduction (Chatterji et al., 2018), the SAGA type of sampling algorithms require an excessively memory storage of O ( Nd ) , which is too costly for big data problems.
Therefore, we leave the study of the lightweight SAGA algorithm inspired by Harikandeh et al.
; Zhou et al.
for future works.
Related work Although our VR-reSGLD is, in spirit, similar to VR-SGLD (Dubey et al., 2016;
Xu et al., 2018), it differs from VR-SGLD in two aspects: First, VR-SGLD conducts variance reduction on the gradient and only shows promises in the nearly log-concave distributions or when
the
Markov process is sufﬁciently converged; however, our VR-reSGLD solely focuses on the variance reduction of the energy estimator to propose more effective swaps, and therefore we can import the empirical experience in hyper-parameter tuning from M-SGD to our proposed algorithm.
Second,
VR-SGLD doesn’t accelerate the continuous-time Markov process but only focuses on reducing the discretization error; VR-reSGLD possesses a larger acceleration term in the Dirichlet form (2) and shows a potential in exponentially speeding up the convergence of the continuous-time process in the early stage, in addition to the improvement on the discretization error.
In other words, our algorithm is not only theoretically sound but also more empirically appealing for a wide variety of problems in non-convex learning.
Input The initial parameters β (1)0 and β (2)0 , learning rate η , temperatures τ (1) and τ (2) , correction factor F and smoothing factor γ . repeatParallel sampling Randomly pick a mini-batch set B k of size n . [Equation] [Table]
[Equation]
[Equation]
) if k mod m = 0 then
[Table]
[Equation]
end if
Bias-reduced swaps Swap β (1)
k +1 and β (2)
k
+1 if u < (cid:101) S
η,m,n , where u ∼ Unif [0 , 1] , and (cid:101) S η,m,n follows
[Equation]
(6) until k = k max .
Output: The low-temperature process { β (1) i T } (cid:98) k max / T (cid:99) i = , where T is the thinning factor.
T HEORETICAL PROPERTIES
The large variance of noisy energy estimators directly limits the potential of the acceleration and signiﬁcantly slows down the convergence compared to the replica exchange Langevin dynamics.
As a result, VR-reSGLD may lead to a more efﬁcient energy estimator with a much smaller variance.
Lemma 1 (Variance-reduced energy estimator) Under the smoothness and dissipativity assumptions 1 and 2 in Appendix A, the variance of the variance-reduced energy estimator (cid:101) L ( B | β ( h ) ) , where h ∈ { 1 , 2 } , is upper bounded by Var (cid:16) (cid:101) L ( B | β ( h ) )
(cid:17) ≤ min (cid:110) O (cid:18) m 2 η n (cid:19) , Var (cid:16) N n (cid:88)
i ∈ B
L ( x i | β ( h ) ) (cid:17) + Var (cid:16) N n (cid:88) i ∈
B L ( x i [Equation]
, where the detailed O ( · ) constants is shown in Lemma B1 in the appendix.
The analysis shows the variance-reduced estimator (cid:101) L ( B | β ( h ) ) yields a much-reduced variance given a smaller learning rate η and a smaller m for updating control variates based on the batch size n .
Although the truncated swapping rate S η,m,n = min { 1 , (cid:101) S η,m,n } still satisﬁes the “stochastic” detailed balance given an unbiased swapping-rate estimator (cid:101) S η,m,n (Deng et al., 2020) † , it doesn’t mean the efﬁciency of the swaps is not affected.
By contrast, we can show that the number of swaps may become exponentially smaller on average .
Lemma 2 (Variance reduction for larger swapping rates)
Given a large enough batch size n , the variance-reduced energy estimator (cid:101) L ( B k | β ( h ) k ) yields a truncated swapping rate that satisﬁes
[Equation]
where S ( β (1) , β (2) ) is the deterministic swapping rate deﬁned in Appendix B.
The proof is shown in Lemma.
B2 in Appendix B. Note that the above lemma doesn’t require the normality assumption.
As n goes to inﬁnity, where the asymptotic normality holds, the RHS of (7) changes to min (cid:110) 1 , S ( β (1) , β (2) ) e −O (cid:16) m 2 ηn (cid:17) (cid:111) , which becomes exponentially larger as we use a smaller update frequency
m and learning rate η .
Since the continuous-time reLD induces a jump operator in the inﬁnitesimal generator, the resulting Dirichlet form potentially leads to a much larger acceleration term which linearly depends on the swapping rate S η,m,n and yields a faster exponential convergence.
Now we are ready to present the ﬁrst main result.
Theorem 1 (Exponential convergence) Under the smoothness and dissipativity assumptions 1 and 2, the probability measure associated with reLD at time t , denoted as ν t , converges exponentially fast to the invariant measure π :
[Equation]
where D 0 is a constant depending on the initialization, δ S η,m,n := inf t>
[Equation]
Sη
[Equation]
m
[Equation]
n
[Equation]
depends on S η,m,n , E S η,m,n and E are the Dirichlet forms based on the swapping rate S η,m,n and are deﬁned in (2), c
LS is the constant of the log-Sobolev inequality for reLD without swaps.
We detail the proof in Theorem.1 in Appendix B. Note that S η,m,n = 0 leads to the same performance as the standard Langevin diffusion and δ S η,m,n is strictly positive when dν t dπ is asymmetric (Chen et al., 2019); given a smaller η and m or a large n , the variance becomes much reduced according to Lemma 1, yielding a much larger truncated swapping rate by Lemma 2 and a faster exponential convergence to the invariant measure π compared to reSGLD.
Next, we estimate the upper bound of the 2-Wasserstein distance W ( µ k , ν kη ) , where µ k denotes the probability measure associated with VR-reSGLD at iteration k .
We ﬁrst bypass the Gr ¨ onwall inequality and conduct the change of measure to upper bound the relative entropy D KL ( µ k | ν kη ) following (Raginsky et al., 2017).
In addition to the approximation in the standard Langevin diffusion Raginsky et al. (2017), we also consider the change of Poisson measure following Yin & Zhu (2010); Gikhman & Skorokhod (1980) to handle the error from the stochastic swapping rate.
We then extend the distance of relative entropy D KL ( µ k | ν kη ) to the Wasserstein distance W 2 ( µ k , ν kη ) via a weighted transportation-cost inequality of Bolley & Villani (2005).
Theorem 2 (Diffusion approximation) Assume the smoothness, the dissipativity and the gradient assumptions 1, 2 and 3 hold.
Given a large enough batch size n , a small enough m and η , we have
[Equation]
(9) where δ is a constant that characterizes the scale of noise caused in mini-batch settings and the detail is given in Theorem 2 in Appendix C .
Here the last term O (cid:0)(cid:0) m 2 n η (cid:1) 1 / 8 (cid:1) comes from the error induced by the stochastic swapping rate, which disappears given a large enough batch size n or a small enough update frequency m and learning rate η .
Note that our upper bound is linearly dependent on time approximately, which is much tighter than the exponential dependence using the Gr¨onwall inequality.
Admittedly, the result without swaps is slightly weaker than the diffusion approximation (3.1) in Raginsky et al.
and we refer readers to Remark 3 in Appendix C.
Applying the triangle inequality for W 2 ( µ k , ν kη ) and W 2 ( ν kη , π ) leads to the ﬁnal result
Theorem 3 Assume the smoothness, the dissipativity and the gradient assumptions 1, 2 and 3 hold.
Given a small enough learning rate η , update frequency m and a large enough batch size n , we have
[Equation]
This theorem implies that increasing the batch size n or decreasing the update frequency m not only reduces the numerical error but also potentially leads to a faster exponential convergence of the continuous-time dynamics via a much larger swapping rate S η,m,n . [Figure]
E XPERIMENTS 5. S IMULATIONS OF G AUSSIAN M IXTURE D ISTRIBUTIONS
We ﬁrst study the proposed variance-reduced replica exchange stochastic gradient Langevin dynamics algorithm (VR-reSGLD) on a Gaussian mixture distribution (Dubey et al., 2016).
The distribution follows from x i | β ∼ 0 .
5 N ( β, σ 2 ) + 0 .
5 N ( φ − β, σ 2 ) , where φ = 20 , σ = 5 and β = − 5 .
We use a training dataset of size N = 10 5 and propose to estimate the posterior distribution over β .
We compare the performance of VR-reSGLD against that of the standard stochastic gradient Langevin dynamics (SGLD), and replica exchange SGLD (reSGLD).
In Figs 2(a) and 2(b), we present trace plots and kernel density estimates (KDE) of samples generated from VR-reSGLD with m = 40 , τ (1) = 10 † , τ (2) = 1000 , η = 1 e − 7 , and F = 1 ; reSGLD adopt the same hyper-parameters except for F = 100 because a smaller F may fail to propose any swaps;
SGLD uses η = 1 e − 7 and τ = 10 .
As the posterior density is intractable, we consider a ground truth by running replica exchange Langevin dynamics with long enough iterations.
We observe that
VR-reSGLD is able to fully recover the posterior density, and successfully jump between the two modes passing the energy barrier frequently enough.
By contrast, SGLD, initialized at β 0 = 30 , is attracted to the nearest mode and fails to escape throughout the run; reSGLD manages to jump between the two modes, however, F is chosen as large as 100 , which induces a large bias and only yields three to ﬁve swaps and exhibits the metastability issue.
In Figure 2(c), we present the evolution of the variance for VR-reSGLD over a range of different m and compare it with reSGLD.
We see that the variance reduction mechanism has successfully reduced the variance by hundreds of times.
In Fig 2(d), we present the sensitivity study of ˜ σ 2 as a function of the ratio n/N and the learning rate η ; for this estimate we average out 10 realizations of VR-reSGLD, and our results agree with the theoretical analysis in Lemma 1.
5. N ON - CONVEX OPTIMIZATION FOR IMAGE DATA
We further test the proposed algorithm on CIFAR10 and CIFAR100.
We choose the 20, 32, 56-layer residual networks as the training models and denote them by ResNet-20, ResNet-32, and ResNet56, respectively.
Considering the wide adoption of M-SGD, stochastic gradient Hamiltonian Monte
Carlo (SGHMC) is selected as the baseline.
We refer to the standard replica exchange SGHMC algorithm as reSGHMC and the variance-reduced reSGHMC algorithm as VR-reSGHMC.
We also include another baseline called cyclical stochastic gradient MCMC (cycSGHMC), which proposes a cyclical learning rate schedule.
To make a fair comparison, we test the variance-reduced replica exchange SGHMC algorithm with cyclic learning rates and refer to it as cVR-reSGHMC.
We run M-SGD, SGHMC and (VR-)reSGHMC for 500 epochs.
For these algorithms, we follow a setup from Deng et al.
We ﬁx the learning rate η (1) k
= 2e-6 in the ﬁrst 200 epochs and decay it by 0.984 afterwards.
For SGHMC and the low-temperature processes of (VR-)reSGHMC, we anneal the temperature following τ (1) k
= 0 .
02 k in the beginning and keep it ﬁxed after the burn-in steps; regarding the high-temperature process, we set η (2) k
= 1 .
5 η (1) k
and τ (2) k
= 5 τ (1) k .
The initial correction factor F 0 is ﬁxed at 1 . 5 e 5 .
The thinning factor T is set to 256 .
In particular for [Figure] cycSGHMC, we run the algorithm for 1000 epochs and choose the cosine learning rate schedule with 5 cycles; η 0 is set to 1e-5; we ﬁx the temperature 0.001 and the threshold 0 .
7 for collecting the samples.
Similarly, we propose the cosine learning rate for cVR-reSGHMC with 2 cycles and run it for 500 epochs using the same temperature 0.001.
We only study the low-temperature process for the replica exchange algorithms.
Each experiment is repeated ﬁve times to obtain the mean andstandard deviations.
We evaluate the performance of variance reduction using VR-reSGHMC and compare it with reSGHMC.
We ﬁrst increase the batch size n from 256 to 512 for reSGHMC and notice that the reduction of variance is around 2 times (see the red curves in Fig.3(c,d)).
Next, we try m = 50 and n = 256 for the VR-reSGHMC algorithm, which updates the control variates every 50 iterations.
As shown in Fig.3(a,b), during the ﬁrst 200 epochs, where the largest learning rate is used, the variance of VR-reSGHMC is slightly reduced by 37% on CIFAR100 and doesn’t make a difference on
CIFAR10.
However, as the learning rate and the temperature decrease, the reduction of the variance gets more signiﬁcant.
We see from Fig.3(c,d) that the reduction of variance can be up to 10 times on
CIFAR10 and 20 times on CIFAR100 .
This is consistent with our theory proposed in Lemma 1.
The reduction of variance based on VR-reSGHMC starts to outperform the baseline with n = 512 when the epoch is higher than 370 on CIFAR10 and 250 on CIFAR100.
We also try m = 392 , which updates the control variates every 2 epochs, and ﬁnd a similar pattern.
For computational reasons, we choose m = 392 and n = 256 for (c)VR-reSGHMC and compare them with the baseline algorithms.
With the help of swaps between two SGHMC chains, reSGHMC already obtains remarkable performance (Deng et al., 2020) and ﬁve swaps often lead to an optimal performance.
However, VR-reSGHMC still outperforms reSGHMC by around 0.2% on CIFAR and 1% improvement on CIFAR100 (Table.1) and the number of swaps is increased to around a hundred under the same setting .
We also try cyclic learning rates and compare cVR-reSGHMC with cycSGHMC, we see cVR-reSGHMC outperforms cycSGHMC signiﬁcantly even if cycSGHMC is running 1000 epochs, which may be more costly than cVR-reSGHMC due to the lack of mechanism in parallelism.
Note that cVR-reSGHMC keeps the temperature the same instead of annealing it as in VR-reSGHMC, which is more suitable for uncertainty quantiﬁcation.
[Table]
Regarding the training cost and the treatment for improving the performance of variance reduction using adaptive coefﬁcients in the early period, we refer interested readers to Appendix E.
For the detailed implementations, we release the code at https://github.com/WayneDW/ Variance_Reduced_Replica_Exchange_Stochastic_Gradient_MCMC .
U NCERTAINTY Q UANTIFICATION FOR UNKNOWN SAMPLES
A reliable model not only makes the right decision among potential candidates but also casts doubts on irrelevant choices.
For the latter, we follow Lakshminarayanan et al.
and evaluate the uncertainty on out-of-distribution samples from unseen classes.
To avoid over-conﬁdent predictions on unknown classes, the ideal predictions should yield a higher uncertainty on the out-of-distribution samples, while maintaining the accurate uncertainty for the in-distribution samples.
Continuing the setup in Sec.5.2, we collect the ResNet20 models trained on CIFAR10 and quantify the entropy on the Street View
House Numbers (SVHN) dataset, which contains 26,032 RGB testing images of digits instead of objects.
We compare cVR-reSGHMC with M-SGD, SGHMC, reSGHMC, and cSGHMC.
Ideally, the predictive distribution should be the uniform distribution and leads to the highest entropy.
We present the empirical cumulative distribution function (CDF) of the entropy of the predictions on SVHN and report it in
Fig.4.
As shown in the left ﬁgure,
M-SGD shows the smallest probability for high-entropy predictions, implying the weakness of stochastic optimization methods in uncertainty estimates.
By contrast, the proposed cVR-reSGHMC yields the highest probability for predictions of high entropy.
Admittedly, the standard ResNet models are poorly calibrated in the predictive probabilities and lead to inaccurate conﬁdence.
To alleviate this issue, we adopt the temperature-scaling method with a scale of 2 to calibrate the predictive distribution (Guo et al., 2017) and present the entropy in Fig.4 (right).
In particular, we see that 77% of the predictions from cVR-reSGHMC yields the entropy higher than 1.5, which is 7% higher than reSGHMC and 10% higher than cSGHMC and much better than the others.
For more discussions of uncertainty estimates on both datasets, we leave the results in Appendix F.
C ONCLUSION
We propose the variance-reduced replica exchange stochastic gradient Langevin dynamics algorithm to accelerate the convergence by reducing the variance of the noisy energy estimators.
Theoretically, this is the ﬁrst variance reduction method that yields the potential of exponential accelerations instead of solely reducing the discretization error.
In addition, we bypass the Gr¨onwall inequality to avoid the crude numerical error and consider a change of Poisson measure in the generalized Girsanov theorem to obtain a much tighter upper bound.
Since our variance reduction only conducts on the noisy energy estimators and is not applied to the noisy gradients, the standard hyper-parameter setting can be also naturally imported, which greatly facilitates the training of deep neural works.
A CKNOWLEDGMENT
We would like to thank Maxim Raginsky and the anonymous reviewers for their insightful suggestions.
Liang’s research was supported in part by the grants DMS-2015498, R01-GM117597 and
R01-GM126089.
Lin acknowledges the support from NSF (DMS-1555072, DMS-1736364), BNL
Subcontract 382247, W911NF-15-1-0562, and DE-SC0021142.
A P RELIMINARIES
Notation We denote the deterministic energy based on the parameter β by L ( β ) =
(cid:80) Ni
=1 L ( x i | β ) using the full dataset of size N .
We denote the unbiased stochastic energy estimator by
Nn (cid:80) i ∈
B L ( x i | β ) using the mini-batch of data B of size n .
The same style of notations is also applicable to the gradient for consistency.
We denote the Euclidean L 2 norm by (cid:107) · (cid:107) .
To prove the desired results, we need the following assumptions:
Assumption 1 (Smoothness)
The energy function L ( x i |· ) is C N -smoothness
if there exists a constant C N > 0 such that ∀ β 1 , β 2 ∈ R d , i ∈ { 1 , 2 , · · · , N } , we have
[Equation]
Note that the above condition further implies for a [Equation] C = NC N and ∀ β 1 , β 2 ∈ R d , we have
[Equation]
The smoothness conditions (10) and (11) are standard tools in studying the convergence of SGLD in (Xu et al., 2018) and Raginsky et al.
(2017), respectively.
Assumption 2 (Dissipativity)
The energy function L ( · ) is ( a, b ) -dissipative if there exist constants a > 0 and b ≥ 0 such that ∀ β ∈ R d , (cid:104) β , ∇ L ( β ) (cid:105) ≥ a (cid:107) β (cid:107) 2 − b.
The dissipativity condition implies that the Markov process is able to move inward on average regardless of the starting position.
It has been widely used in proving the geometric ergodicity of dynamic systems (Mattingly et al., 2002;
Raginsky et al., 2017; Xu et al., 2018).
Assumption 3 (Gradient oracle)
There exists a constant δ ∈ [0 , 1) such that for any β , we have
[Equation]
where Φ is a positive constant.
The same assumption has been used in Raginsky et al.
(2017) to control the stochastic noise from the gradient.
B E XPONENTIAL ACCELERATIONS VIA V ARIANCE REDUCTION
We aim to build an efﬁcient estimator to approximate the deterministic swapping rate S ( β (1) , β (
[Equation]
[Equation]
In big data problems and deep learning, it is too expensive to evaluate the energy (cid:80) Ni =1 L
( x i | β ) for each β for a large N .
To handle the computational issues, a popular solution is to use the unbiased stochastic energy Nn (cid:80) i ∈
B L ( x i | β ) for a random mini-batch data B of size n .
However, a n ¨ aive replacement of (cid:80) N i =1 L ( x i | β ) by N n (cid:80) i ∈
B L ( x i | β ) leads to a large bias to the swapping rate.
To remove such a bias, we follow Deng et al.
and consider the corrected swapping rate (cid:98) S
[Equation]
where (cid:98) σ 2 denotes the variance of Nn (cid:80)
i ∈
B L ( x i | β (1) )
− Nn (cid:80) i
∈
B L ( x i | β (2) ) .
∗ Empirically, (cid:98) σ 2 is quite large
[Equation]
resulting in almost no swaps and insigniﬁcant accelerations.
To propose more effective swaps, we consider the variance-reduced estimator
[Equation] [Equation]
where the control variate β m (cid:98) km (cid:99) is updated every m iterations.
Denote the variance of (cid:101) L ( B | β (1) ) − (
cid:101) L ( B | β (2) ) by (cid:101)
σ 2 .
The variance-reduced stochastic swapping rate follows
[Equation]
Using the strategy of variance reduction, we can lay down the ﬁrst result, which differs from the existing variance reduction methods in that we only conduct variance reduction in the energy estimator for the class of SGLD algorithms.
Lemma B1 (Variance-reduced energy estimator) Under the smoothness and dissipativity assumptions 1 and 2, the variance of the variance-reduced energy estimator (cid:101) L ( B k | β ( h ) k ) , where h ∈ { 1 , 2 } , is upper bounded by
[Equation]
(17) where D R = CR +max i ∈{ 1 , 2 , ··· ,N } N (cid:107)∇ L ( x i | β (cid:63) )
(cid:107) + Cba and R is the radius of a sufﬁciently large ball that contains β ( h ) k for h ∈ { 1 , 2 } .
Proof
[Equation]
[Table]
[Equation] [Table] [Equation] [Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation] [Table] [Equation] [Table]
[Equation] [Table] [Equation] [Table]
[Equation] [Table] [Equation] [Table]
[Equation] [Table] [Equation] [Table]
[Equation]
[Table]
(cid:33)(cid:35) 
[Equation]
[Table] [Equation] [Table] [Equation]
[Table] [Equation] [Table] [Equation]
[Table] [Equation] [Table] [Equation]
[Table] [Equation] [Table] [Equation]
[Table] [Equation] [Table] [Equation]
[Table] [Equation] [Table] [Equation]
[Table]
(cid:33)(cid:35) 
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table] [Equation] [Table] j =
[Table]
L
[Equation]
where the last equality follows from the fact that E [( (cid:80) ni =1 x i ) 2 ] = (cid:80)
ni =1 E [ x 2 i ] for independent variables { x i } ni
=1 with mean 0.
The ﬁrst inequality follows from E [( x − E [ x ]) 2 ] ≤ E [ x 2 ] and the last inequality follows from Lemma D1, where D R = CR +max i ∈{ 1 , 2 , ··· ,N } N (cid:107)∇ L ( x i | β (cid:63) )
(cid:107) + Cba and R is the radius of a sufﬁciently large ball that contains β ( h ) k for h ∈ { 1 , 2 } .
Next, we bound E (cid:20)(cid:13)(cid:13) (cid:13) β
( h )
k − β ( h ) m (cid:98) k m (cid:99) (cid:13)(cid:13) (cid:13) 2 (cid:21) as follows
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
where the ﬁrst inequality follows by E [ (cid:107) a
[Equation]
b (cid:107) 2 ] ≤ 2 E [ (cid:107) a (cid:107) 2 ] + 2 E [ (cid:107) b (cid:107) 2 ] , the i.i.d of the data points and τ (1) ≤ τ (2) for h ∈ { 1 , 2 } ; the second inequality follows by Lemma D2; the last inequality follows from Lemma D3.
Combining (18), (19) and (20), we have
[Equation]
Since Var (cid:16)(cid:101) L ( B k | β ( h ) k ) (cid:17) ≤ Var (cid:0) Nn
(cid:80) i ∈
B L ( x i | β k ) (cid:1) + Var (cid:16)
Nn (cid:80) i
∈
B L (cid:16) x i (cid:12)(cid:12)(cid:12) β m (cid:98) km (cid:99) (cid:17)(cid:17) by deﬁnition, Var (
cid:16)(cid:101) L ( B k | β ( h ) k ) (cid:17) is upper bounded by O (cid:16) min { (cid:98) σ 2 , m 2 ηn } (cid:17) , which becomes much smaller using a small learning rate η , a shorter period m and a large batch size n .
Note that (cid:101) S η,m,n ( β (1) , β (2) ) is deﬁned on the unbounded support [0 , ∞ ] and E [ (cid:101) S η,m,n ( β (1) , β (2) )]
=
S ( β (1) , β (2) )
regardless of the scale of (cid:101) σ 2 .
To satisfy the (stochastic) reversibility condition, we consider the truncated swapping rate min { 1 , (cid:101) S η,m,n ( β (1) , β (2) ) } , which still targets the same invariant distribution (see section 3.1 (Quiroz et al., 2019) for details).
We can show that the swapping rate may even decrease exponentially as the variance increases.
Lemma B2 (Variance reduction for larger swapping rates)
Given a large enough batch size n , the variance-reduced energy estimator (cid:101) L ( B k | β ( h ) k ) yields a truncated swapping rate that satisﬁes
[Equation]
Proof
By central limit theorem, the energy estimator Nn (cid:80) i ∈ B L ( x i | β k ) converges in distribution to a normal distributions as the batch size n goes to inﬁnity.
In what follows, the variance-reduced estimator (cid:101) L ( B k | β k ) also converges to a normal distribution, where the corresponding estimator is denoted by (cid:101) L ( B k | β k ) . Now the swapping rate S η,m,n ( · , · ) based on normal estimators follows
[Equation]
23) where ¯ σ 2 denotes the variance of (cid:101) L ( B | β (1) ) − (cid:101)
L ( B | β (2) ) .
Note that S η,m,n ( β (1) , β (2) ) follows a log-normal distribution with mean log S ( β (1) , β (2) ) − (cid:0) 1 τ (1) − 1 τ (2) (cid:1) 2 ¯ σ 2 2 and variance (cid:0) 1 τ (1) − 1 τ (2) (cid:1) 2 ¯ σ 2 on the log-scale, and S ( β (1) , β (2) ) is the deterministic swapping rate deﬁned in (13).
Applying Lemma D4, we have
[Equation]
Moreover, ¯ σ 2 differs from (cid:101) σ 2 , the variance of (cid:101) L ( B | β (1) ) − (cid:101)
L ( B | β (2) ) , by at most a bias of
O ( 1 n 2 ) according to the estimate of the third term of (S2) in Quiroz et al.
and (cid:101) σ 2 ≤
Var (cid:16)(cid:101)[-L ( B k | β (1) k ) (cid:17) + Var (cid:16)(cid:
 ) L ( B k | β (2) k ) (cid:17) , where both Var (cid:16)(cid:101) L ( B k | β (1) k ) (cid:17) and Var ( cid:16)(cid:101) L ( B k | β (2) k ) (cid:17) are upper bounded by m 2 ηn D 2 R (cid:0) 2 ηn (2 C 2 Ψ d,τ (2) ,C,a,b + 2 Q 2 ) + 4 τd (cid:1) by Lemma B1, it follows that-]
[Equation] [Equation]
+ Var
[Equation]
L ( B
[Equation]
β (2) k ) (cid:17) , where both Var (cid:16)(cid:101) L ( B k | β (1) k ) (cid:17) and Var (
cid:16)(cid:101) L ( B k | β (2) k ) (cid:17) are upper bounded by m 2 ηn D 2 R (cid:0) 2 ηn (2 C 2 Ψ d,τ (2) ,C,a,b + 2 Q 2 ) + 4 τd (cid:1) by Lemma B1, it follows that
[Equation]
Applying min { 1 , A + B } ≤ min { 1 , A } + | B | , we have
[Equation]
By the triangle inequality
[Equation]
we can further upper bound the ﬁrst term
I
[Equation]
[Equation]
where I [-
-]1 and I 2 follow from the proof of S1 without and with normality assumptions, respectively (Quiroz et al., 2019).
Combining (26) and (27) [Equation] we have
[Equation]
This means that reducing the update period m (more frequent update the of control variable), the learning rate η and the batch size n signiﬁcantly increases min { 1 , (cid:101) S η,m,n } on average.
The above lemma shows a potential to exponentially increase the number of effective swaps via variance reduction under the same intensity r .
Next, we show the impact of variance reduction in speeding up the exponential convergence of the corresponding continuous-time replica exchange
Langevin diffusion.
Theorem 1 (Exponential convergence) Under the smoothness and dissipativity assumptions 1 and 2, the replica exchange Langevin diffusion associated with the variance-reduced stochastic swapping rates S η,m,n ( · , · ) = min { 1 , (cid:101) S
η,m,n ( · , · ) } converges exponential fast to the invariant distribution π given a smaller learning rate η , a smaller m or a larger batch size n :
[Equation]
(29) where D 0 = (cid:112)
[Equation]
[Equation]
− 1 is a non-negative constant depending on the truncated stochastic swapping rate S η,m,n ( · , · ) and increases with a smaller learning rate η , a shorter period m and a large batch size n .
c LS is the standard constant of the log-Sobolev inequality asscoiated with the Dirichlet form for replica exchange Langevin diffusion without swaps.
Proof
Given a smooth function f : R d × R d → R , the inﬁnitesimal generator L S η,m,n associated with the replica exchange Langevin diffusion with the swapping rate S η,m,n = min { 1 , (cid:101) S η,m,n } follows
[Equation]
where ∇ β ( h ) and ∆ β ( h ) are the gradient and the Laplace operators with respect to β ( h ) , respectively.
Next, we model the exponential decay of W 2 ( ν t , π ) using the Dirichlet form
[Equation]
where Γ S η,m,n ( f ) = 12 · L S η,m,n ( f 2 ) − f L S η,m,n ( f ) is the Carr´e du Champ operator.
In particular for the ﬁrst term 12 L S η,m,n ( f 2 )
[Equation]
we have
[Equation]
Combining the deﬁnition of the Carr ´ e du Champ operator, (30) and (B), we have
[Equation]
Plugging (32) into (31), the Dirichlet form associated with operator L S η,m,n follows
[Equation]
where f corresponds to dν t dπ ( β (1) , β (2) ) .
Under the asymmetry conditions of
[Equation]
, β (2) ) and S η,m,n > 0 , the acceleration term of the Dirichlet form is strictly positive and linearly dependent on the swapping rate S η,m,n .
Therefore, E S η,m,n ( f ) becomes signiﬁcantly larger as the swapping rate S η,m,n increases signiﬁcantly.
According to Lemma 5 (Deng et al., 2020), there exists a constant δ S η,m,n = inf t> 0 E Sη,m,n ( [Equation]
E [Equation] − 1 depending on S η,m,n that satisﬁes the following log-Sobolev inequality for the unique invariant measure π associated with variance-reduced replica exchange
Langevin diffusion { β t } t ≥
[Equation]
where δ S η,m,n increases rapidly with the swapping rate S η,m,n .
By virtue of the exponential decay of entropy (Bakry et al., 2014), we have
[Equation]
where c LS is the standard constant of the log-Sobolev inequality asscoiated with the Dirichlet form for replica exchange Langevin diffusion without swaps (Lemma 4 as in Deng et al.
Next, we upper bound W 2 ( ν t , π ) by the Otto-Villani theorem (Bakry et al., 2014)
[Equation]
where δ S η,m,n > 0 depends on the learning rate η , the period m and the batch size n .
In the above analysis, we have established the relation that δ S η,m,n = inf t> 0 E Sη,m,n ( (cid:113) dνtdπ )
E ( (cid:113) dνtdπ ) −depending on S η,m,n may increase signiﬁcantly with a smaller learning rate η , a shorter period m and a large batch size n .
For more quantitative study on how large δ S η,m,n is on related problems, we refer interested readers to the study of spectral gaps in Lee et al.
; Dong & Tong (2020);
Futami et al.
C D ISCRETIZATION ERROR
Consider a complete ﬁltered probability space (Ω , F , F = ( F t ) t ∈ [0 ,T ] , P ) which supports all the random subjects considered in the sequel.
With a little abuse usage of notation, the probability measure P (component wise if P is joint probability measure with mutually independent components) would always denote the Wiener measure under which the process ( W t )
≤ t ≤ T is a P -Brownian motion.
To be precise, in what follows, we shall denote P := P W × N , where P W is the inﬁnite dimensional Wiener measure and N is the Poisson measure independent of P W and has some constant jump intensity.
In our general framework below, the jump process α is introduced by swapping the diffusion matrix of the two Langevin dynamics and the jump intensity is deﬁned through the swapping probability in the following sense, which ensures the independence of P W and N S in each time interval [ iη, ( i + 1) η ] , for i ∈ N + .
The precise deﬁnition of the Replica exchange Langevin diffusion (reLD) is given as below.
For any ﬁxed learning rate η > 0 , we deﬁne
[Equation]
dt ) = l, β ( (cid:98)
[Equation]
(cid:99)
[Equation]
=
[Equation]
=
[Equation]
o ( dt
[Equation]
, for l (cid:54) = j [Equation] 34) where ∇ G ( β ) :=
(cid:18) ∇ L ( β (1) ) ∇ L ( β (2) ) (cid:19) , and 1 t = (cid:98) t/η (cid:99) η is the indicator function, i.e. for every t
= iη with i ∈ N + , given β ( iη ) =
β , we have P ( α ( t ) =
j | α ( t − dt )
= l ) = rS ( β ) η , where S ( β ) is deﬁned as min { 1 , S ( β (1) , β (2) ) } and S ( β (1) , β (2) ) is deﬁned in (13).
In this case, the Markov Chain α ( t ) is a constant on the time interval [ (cid:98) t/η (cid:99) η, (cid:98) t/η (cid:99) η + η ) with some state in the ﬁnite-state space { 0 , 1 } and the generator matrix Q follows
[Equation]
where δ ( · ) is a Dirac delta function.
The diffusion matrix Σ( α t ) is thus deﬁned as (Σ(0) , Σ(1)) :
= (cid:26)(cid:18) √
[Equation]
I d [Equation] (cid:19)(cid:27) . From our deﬁnition and following Yin & Zhu (2010)[Section 2.7], the generator matrix Q will depend on the initial value at each time interval [ iη, ( i + 1) η ) .
The distribution of process ( β t ) 0 ≤ t ≤ T is denoted as ν T :
=
P G × N S which is absolutely continuous with respect to the reference measure P := P W × N , under which W is Brownian motion and α ( · ) is a Poisson process with some constant jump intensity.
This fact follows from the result in Gikhman & Skorokhod (1980)[VII, Section 6, Theorem 2] and Yin & Zhu (2010)[Section 2.5, formula (2.40)].
The motivation of only considering the positive swapping rate in iη , for i ∈ N + , and zero elsewhere is due to our construction of the discretized process (cid:101) β as shown below (see equation 35).
A simple illustration of the idea can be seen from the auxiliary process construction in Yin & Zhu (2010)[Section 2.5], following which we want to make sure the stopping time of β and (cid:101)
β happening at the same time.
Otherwise, it is unlikely (and also unreasonable) to derive the Radon-Nikodym derivative of the two process β and (cid:101) β .
Thus, we should think of the process is concatenated on the time interval [ iη, ( i + 1) η ) up to time horizon T .
Similarly, we consider the following Replica exchange stochastic gradient Langevin diffusion , for the same learning rate η > 0 as above, we have  
[Equation]
where ∇ (cid:101) G ( β ) := (cid:18) ∇ (cid:101)
L ( β (1) ) ∇ (cid:101) L ( β (2) )
(cid:19) and (cid:101) S ( (cid:101) β ) =
min { 1 , (cid:101) S η,m,n ( (cid:101) β (1) , (cid:101) β (2) ) } and (cid:101) S η,m,n ( (cid:101) β (1) , (cid:101) β (2) ) is shown in (16).
The distribution of process ( (cid:101) β t ) 0 ≤ t ≤ T is denoted as µ T :
= P (cid:101) G × N (cid:101) S , where (cid:101) α is a
Poisson process with jump intensity r (cid:101) S ( (cid:101) β ) ηδ
( t −(cid:98) t/η (cid:99) η ) on the time interval [ (cid:98) t/η (cid:99) η, (cid:98) t/η (cid:99) η + η ) .
Note that β and (cid:101) β are deﬁned by using the same P -Brownian motion W , but with two different jump intensity on the time interval [ (cid:98) t/η (cid:99) η, (cid:98) t/η (cid:99) η + η ) .
Notice that, if there is no jump, the construction of (cid:101) β based on β follows from the fact that they share the same marginal distributions as shown in Gy¨ongy (1986), where one can ﬁnd the details in Raginsky et al.
Given the jump process α and (cid:101) α introduced into the dynamics of β and (cid:101) β , the construction is more complicated.
Thanks to Bentata & Cont (2009), we can carry on the similar construction in our current setting.
We then introduce the following Radon-Nikodym density for dν T /dµ T .
In the current setting, the change of measure can be seen as the combination of two drift-diffusion process and two jump process simultaneously.
We ﬁrst introduce some notation.
For each vector A ∈ R n , we denote (cid:107) A (cid:107) 2 :
= A ∗ A .
Furthermore, we introduce a sequence of stopping time based on our deﬁnition of process β and (cid:101) β .
For j ∈ N + , we denote ζ (cid:48)
j s as a stopping times deﬁned by ζ j +1 :
= inf { t > ζ j : α ( t ) (cid:54) =
α ( ζ j ) } and N ( T )
=
max { n ∈ N : ζ n ≤ T } .
It is easy to see that for any stopping time ζ i , there exists l ∈ N + such that ζ j = lη .
Similarly, we have the stopping time for the process (cid:101) β by (cid:101) ζ j +1 :
=
inf { t > (cid:101) ζ j : (cid:101) α
( t ) (cid:54)
=
(cid:101) α ( ζ j ) } and ˜ α ( t ) follows the same trajectory of α ( t ) .
To serve the purpose of our analysis, one should think of the process β as the auxiliary process to the process (cid:101) β , see similar constructions in Yin & Zhu (2010)[Section 2.5, formula (2.39)].
The difference is that both of our process β and (cid:101) β are associated with jump process jumping at time iη , for some integer i ∈ N + , instead of jumping at any continuous time.
We combine approximation method from Yin & Zhu (2010)[Section 2.7] for non-constant generator matrix Q and the density representation for
Markov process in Gikhman & Skorokhod (1980)[VII, Section 6, Teorem 2] to get the following
Lemma C1 Let { ζ j | j ∈ { 0 , 1 , · · · , N ( T ) }} be a sequence of stopping time deﬁned by α .
Let k ∈ N + be an ﬁxed integer such that kη ≤ T ≤ ( k + 1) η .
For each ﬁxed learning rate η > 0 and for any ε > 0 , the Radon-Nikodym derivative of dµ T
/dν T is given as below,
[Equation]
Proof
Recall that ζ j is stopping time deﬁned by α (same as deﬁned by (cid:101) α ), i.e. ζ j +1 :=
inf { t > ζ j : α ( t ) (cid:54) =
α ( ζ j ) } , for j = 0 , 1 , · · · , N ( T ) , and for each ζ j , there exists l ∈ { 0 , 1 , · · · , k } such that ζ j = lη .
We now follow Gikhman & Skorokhod (1980)[VII, Section 6, Theorem 2] to derive the Radon-Nikodym density for dµ T /dν T .
In this case, if the generator matrix Q is constant, i.e. the jump intensity is constant, we can follow the similar construction from Yin & Zhu (2010)[Formula (2.40)], see also Eizenberg & Freidlin (1990)[Formula(3.13)].
Next, we adjust our setting to the case that we can treat our generator matrix as constant on each time interval [ ζ j , ζ j +1 ) , then the existing results apply to our case for the density with respect to the Poisson measure (jump process α and (cid:101) α ), i.e. d N S /d N (cid:101) S .
Furthermore, once the generator matrix Q is constant, then the measure P G ( or
P (cid:101) G ) is independent to N S (or N (cid:101) S ).
We show the following steps to give a clear outline of our proof.
Step 1:
For each stopping time interval [ ζ j , ζ j +1 ) , no jump would occur after the initial point at time ζ j and the diffusion matrix Σ and (cid:101) Σ keep the same, thus we can apply the generalized Girsanov theorem to get the Randon-Nikodym derivative for d P G /d P (cid:101) G .
Step 2:
In order to combine the the two density of d N S /d N (cid:101) S and d P G /d P (cid:101) G , we need the independent property of the two measures on the same time interval, then we directly get the density following from Gikhman & Skorokhod (1980)[VII, Section 6, Theorem 2].
Different from the work mentioned above, we will ﬁrst write all the density on each time interval [ iη, ( i + 1) η ) to incorporate the independent requirement mentioned above.
Notice that the relative change of density for d N S /d N (cid:101) S would only depends on the left end point, since the jump intensity would change its values at the initial value of interval [ iη, ( i + 1) η ) , which is a standard idea to deal with generator matrix Q depending on the initial value instead of a constant matrix case.
(See Yin & Zhu (2010)[Section2.7] for similar treatments).
Step 3:
In general, the stopping time interval could contain several time interval with length η , however the jump intensity should only depend on the left end point for each time interval [ iη, ( i + 1) η ) .
Based on the above set up, we now derive the Radon-Nikodym derivative.
First notice that, on each period [ ζ j , ζ j +1 ) , the matrix Σ is ﬁxed and is evaluated at Σ( α ( ζ j )) , which is the same for Σ( (cid:101) α ( ζ j )) .
In particular, Σ( α ( ζ j ))
=
Σ( (cid:101) α ( ζ j )) is a constant diagonal matrix.
According to our deﬁnition dν T =
d P G × d N S and dµ T =
d P (cid:101) G × d N (cid:101) S , we write the Radon-Nikodym derivative on each of the time interval [ iη, ( i + 1) η ) and concatenate them together.
We consider the swapping of the diffusion matrix ﬁrst where a similar construction can be found in Yin & Zhu (2010)[Formula (2.40)], we get the following Radon-Nikodym derivative, for any ε > 0 ,
[Equation]
Next, we show the density for d P G /d P (cid:101) G as below.
On each interval [ ζ j , ζ j +1 ) , given initial value ( β j , (cid:101) β j ) , the matrix Σ( α ( ζ j )) and Σ( (cid:101) α ( ζ j )) are always the same, since no jump would happen.
In particular, in this continuous case the integral on [ ζ j , ζ j +1 ) and [ ζ j , ζ j +1 ] are the same.
Thus we have the following Radon-Nikodym derivative
[Equation]
[Equation]
Notice that matrix Σ is diagonal square matrix, thus we have Σ = Σ ∗ .
Recall that W is a PBrownian motion, assuming there is no jump in the dynamic for β , then according to the Girsanov theorem (see an example in Theorem 8.6.6 and Example 8.6.9 (Øksendal, 2003)) with RadonNikodym derivative d P G /d P , we have the P G -Brownian motion, denoted as W G , which follows
[Equation]
This fact holds true on each of the time interval [ ζ j , ζ j +1 ] .
Multiplying the two density d P G / P (cid:101) G and d N S /d N (cid:101) S , we complete the proof.
Remark 1 Notice that, if we keep the constant diffusion matrix without jump, then the RandonNikodym derivative dµ T /dν T has been used in the stochastic gradient descent setting, for example
Raginsky et al.
However, the notation of the Brownian motion has been used freely, we try to make it consistent in the current setting.
Namely, for constant diffusion matrix Σ , we have
[Equation]
(39) where W G is a P G -Brownian motion as shown in equation 38, not a P -Brownian motion W .
Remark
The density dµ T dν T that we derived above is so far the best we can do.
If one would like to use the continuous time control α ( t ) with continuous jump intensity S ( β ( t )) instead of jumping at the initial point with a ﬁxed rate, then we can not even write the Randon-Nikodym derivative anymore, since α ( t ) and (cid:101) α ( t ) will deﬁne different stopping time, i.e. jump at different time and µ T is not absolutely continuous with respect to ν T .
Based on the above lemma, we further get the following estimates.
Lemma C
Given a large enough batch size n or a small enough m and η , we have the bound of the KL divergence of D KL ( µ T | ν T ) as below,
[Equation]
with
[Equation]
Proof
By the very deﬁnition of the KL-divergence, we have
[Equation]
We shall keep the convention below and denote E ν T ,β = E ν T [ ·| ( β , (cid:101) β ) =
( β, (cid:101) β )] , where β
= ( β (1) , β (2) )
∈ R 2 d and (cid:101) β
= ( (cid:101) β (1) , (cid:101) β (2) )
∈ R 2 d denotes the values at each time iη , i = 0 , 1 , · · · , k .
Plugging Lemma C1 in the above equation and we unify the notation by using time intervals of the type [ iη, ( i + 1) η ] .
To be precise, we get
[Equation]
[Equation]
The above equality follows from the fact that each time interval [ ζ j , ζ j +1 ] always contain exactly some sub-interval [ iη, ( i + 1) η ] .
Namely, we have [ ζ j , ζ j +1 ] =
[ iη, ( i + 1) η ] ∪ [( j + 1) η, ( j + 2) η ] ∪ · · · ∪ [ lη, ( l + 1) η ] , for some i, l ∈ { 0 , 1 , · · · , k } .
In particular, the matrix Σ keep the same on each interval [ iη, ( i + 1) η ] , for some i ∈ { 0 , 1 , · · · , k } .
Similarly, we expand the Radon-Nokodym derivative for d N (cid:101)
S d N S on the time interval of length η .
Based on our deﬁnition of jump intensity, we get
[Equation]
(cid:33) . Without loss of generality, we shall only consider the sum (cid:80) k − 1 i =0 and skip the interval [ kη, T ] .
Notice that on each time interval [ iη, ( i + 1) η ) , the control α ( iη ) and (cid:101) α
( iη ) are ﬁxed, thus the two component of the measure dν T, β are independent.
Taking into account the fact that W G is P G -Brownian motion, thus we apply the martingale property and arrive at
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation] [Table] [Equation] [Table]
[Equation]
We then estimates the three terms I , J , K in order as below.
Estimate of I : Due to the fact that every interval [ iη, ( i + 1) η ) ⊂ [ ζ j , ζ j +1 ) for some j ∈ { 0 , 1 , · · · , N ( T ) } , we know that the control α and (cid:101) α are the same in the interval [ iη, ( i + 1) η ] and the diffusion matrix Σ is just constant matrix.
Thus, we know that matrix Σ − 1 ( (cid:101) α ( iη )) =
Σ − 1 ( α ( iη )) , which takes one of the form from (Σ − 1 (0) , Σ − 1 (1)) :
= (cid:40)(cid:32)
[Equation]
[Equation]
(cid:33)(cid:41) .
If Σ − 1 ( α ( iη )) =
Σ − 1 (0) , we get
[Equation]
Here ∇ G ( β ) :
= (cid:18) ∇ L ( β (1) ) ∇ L ( β (2) ) (cid:19) and ∇ (cid:101) G ( β ) :
= (cid:18) ∇ (cid:101)
L ( β (1) ) ∇ (cid:101) L ( β (2) )
(cid:19) .
The other matrix form of Σ − 1 (1) will result in the same estimates.
We thus get
I
[Equation]
On each ﬁxed interval, for t ∈ [ kη, ( k + 1) η ) , we have P G -Brownian motion and P (cid:101) G -Brownian motion (see examples in Theorem 8.6.6 and Example 8.6.9 (Øksendal, 2003)),
[Equation]
Plugging the P G (and P (cid:101) G )
-Brownian motions to the original dynamics (34) and (35), we have d β t [Equation] t
[Equation]
d W Gt [Equation] and d (cid:101) β t = Σ( α t ) d W (cid:101) Gt . On each interval [ iη, ( i +1) η ) , Σ( α t ) is a constant matrix, thus we know that the probability distribution of { β t } t ∈ [ kη, ( k +1) η ) and { (cid:101) β t } t ∈ [ kη, ( k +1) η ) are the same and we denote as L ( β t ) =
L ( (cid:101) β t ) .
The difference is that β t is driven by P G -Brownian motion and (cid:101) β t is driven by P (cid:101) G -Brownian motion, which implies that, for t ∈ [ iη, ( i + 1) η ) , we have
[Equation]
Thus, we have the following estimates,
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
We now estimate the two terms I
and I 2 separately.
Notice that, following our notation of P (cid:101) GBrownian motion, for t ∈ [ iη, ( i + 1) η ) , we have
[Equation]
which implies that (recall that dµ T
= d P (cid:101) G × N (cid:101) S and Σ ∈
[Equation]
We thus conclude that,
[Equation] As for the term I
[Equation]
according to Assumption [Equation] we obtain that
[Equation]
Now [Equation] we just need to estimate E µ T , (cid:101) β
[ (cid:107) (cid:101) β kη (cid:107) 2 ] † .
On each interval [ iη, ( i + 1) η ] , under the measure dµ T, (cid:101) β , we have
[Equation]
which implies that
[Equation]
The last equality follows from the independence of (cid:101) β kη and W (cid:101) G ( k +1)
η − W (cid:101) G kη , and W (cid:101) G is a P (cid:101) GBrownian motion.
By induction, we get
[Equation]
We conclude that,
[Equation]
which implies that
[Equation]
Estimate J :
According to our deﬁnition of the swapping probability, we have, for each i ,
[Equation]
which means | (cid:101) S ( (cid:101) β iη ) − S ( β iη )
| ≤ 1 .
Denote C τ = | 1 τ (1) − 1 τ (2) | , we have
[Equation]
Applying Taylor expansion for the exponential function at C τ
( L ( β (1) kη ) − L ( β (2) kη )) , we have
[Equation]
[Table]
where the last inequality follows from S ( β (1) iη [Table] β (2) iη ) ≤ 1 .
Combining Lemma B
[Table]
we thus get the following estimates,
[Equation]
where the last inequality follows from the Jensen’s inequality and the last order holds given a large enough batch size n or a small enough m and η
[Equation]
Estimate K : We now estimate the last term K , we have
[Equation]
Combining Lemma B1 again, we conclude with
[Equation]
Combining the estimates of I , J , and K , we complete the proof.
Remark 3 After the change of measure, the expectation is under the new measure P G (or P (cid:101) G ) instead of the Wiener measure P .
In the estimate of term
I , similar L 2 estimates of the term
E µ T , (cid:101) β
[ (cid:107) (cid:101) β
( i +1)
η (cid:107) 2 ] has been obtained in Raginsky et al.
(2017)[Proof of Lemma 7] when there is no swap.
The difference is we write the dynamic of (cid:101) β
( i +1)
η with respect to the P (cid:101) G -Brownian motion W (cid:101) G instead of the P -Brownian motion W .
In principle, W under P (cid:101) G is not a Brownian motion.
We then extend the distance of relative entropy D KL ( µ T | ν T ) to the Wasserstein distance
W 2 ( µ T , ν T ) via a weighted transportation-cost inequality of Bolley & Villani (2005).
Theorem 2 Given a large enough batch size n or a small enough m and η , we have
[Equation]
Proof
Before we proceed, we ﬁrst show in Lemma.
D5 that ν T has a bounded second moment; the L 2 upper bound of µ T is majorly proved in Lemma.
C2 (Chen et al., 2019) except that the slight difference is that the constant in the RHS of (C.38) Chen et al.
(2019) is changed to account for the stochastic noise.
Then applying Corollary 2.3 in Bolley & Villani (2005), we can upper bound the two Borel probability measures µ T and ν T with ﬁnite second moments as follows
[Equation]
where C ν = 2 inf λ> 0 (cid:16) 1 λ (cid:16) 32 + log (cid:82) R d e λ (cid:107) w (cid:107) 2 ν ( dw ) (cid:17)(cid:17) 1 / 2 .
Applying Lemma D6, we have
[Equation]
Combining Lemma. C2 and (cid:112) N ( T ) ≤ N ( T ) and taking η ≤ 1 , kη > 1 , and λ = 1 , we have W
[Equation]
(cid:101) [Equation]
Φ [Equation] √ [Equation]
, where (cid:101) Φ i =
Φ i + √
Φ i for i ∈ { 0 , 1 , 2 } .
In what follows, we have
[Equation]
( T ) , where Ψ i = (cid:0) 12 + 8 (cid:0) κ
+ 2 b + 4 dτ (2) (cid:1)(cid:1) (cid:101)
Φ i for i ∈ { 0 , 1 , 2 } .
By the orders of Φ 0 , Φ 1 and Φ 2 deﬁned in Lemma.
C2, we have
[Equation]
where N ( T ) kη can be interpreted as the average swapping rate from time 0 to T and is of order O (1) .
Taking square root to both sides of the above inequality lead to the desired result (43).
D P ROOF OF T ECHNICAL L EMMAS
Lemma D1 (Local Lipschitz continuity)
Given a d -dimensional centered ball U of radius R , L ( · ) is D R -Lipschitz continuous in that | L ( x i | β 1 ) − L ( x i | β 2 ) | ≤ D R N (cid:107) β 1 − β 2 (cid:107) for ∀ β 1 , β 2 ∈ U and any i ∈ { 1 , 2 , · · · , N } , where D R = CR + max i ∈{ 1 , 2 , ··· ,N } N (cid:107)∇ L ( x i | β (cid:63) )
(cid:107) + Cba .
Proof
For any β 1 , β 2 ∈ U , there exists β 3 ∈ U that satisﬁes the mean-value theorem such that
[Equation]
Moreover, by Lemma D2, we have
[Equation]
Lemma D
Under the smoothness and dissipativity assumptions 1, 2, for any β ∈ R d , it follows that
[Equation]
where Q = max i ∈{ 1 , 2 , ··· ,N } N (cid:107)∇ L ( x i | β (cid:63) )
(cid:107) + bCa .
Proof
According to the dissipativity assumption, we have
[Equation]
where β (cid:63) is a minimizer of ∇ L ( · ) such that ∇ L ( β (cid:63) )
= 0 .
In what follows, we have (cid:107) β (cid:63) (cid:107) ≤ ba .
Combining the triangle inequality and the smoothness assumption 1, we have
[Equation]
Setting C N = CN as in (11) and Q = max i ∈{ 1 , 2 , ··· ,N } (cid:107)∇ L ( x i | β (cid:63) )
(cid:107) + bCa completes the proof.
The following lemma is majorly adapted from Lemma C.2 of Chen et al.
(2019), except that the corresponding constant in the RHS of (C.38) is slightly changed to account for the stochastic noise.
A similar technique has been established in Lemma 3 of Raginsky et al.
Lemma D3 (Uniform L 2 bounds on replica exchange SGLD)
Under the smoothness and dissipativity assumptions 1, 2.
Given a small enough learning rate η ∈ (0 , 1 ∨ aC 2 ) , there exists a positive constant Ψ d,τ (2) ,C,a,b < ∞ such that sup k ≥ 1 E [ (cid:107) β k (cid:107) 2 ] < Ψ d,τ (2) ,C,a,b .
Lemma D4 (Exponential dependence on the variance)
Assume S is a log-normal distribution with mean u − 12 σ 2 and variance σ 2 on the log scale.
Then E [min(1 , S )]
= O ( e u − σ 2 8 ) , which is exponentially smaller given a large variance σ 2 .
Proof
For a log-normal distribution S with mean u − 12 σ 2 and variance σ 2 on the log scale, the probability density f S ( S ) follows that
[Equation]
In what follows, we have
[Equation]
[Equation]
By change of variable y = log S − u + 12 σ 2 σ where S = e σy + u − 12 σ
and y = − uσ + σ 2 given S = 1 , it follows that
E [min(1 , S )]
[Equation]
where the last equality follows from the change of variable z
= σ − y and the second last inequality follows from the exponential tail bound of the standard Gaussian variable P ( y > (cid:15) )
≤ e − (cid:15) [Equation]
Lemma D5 (Uniform L 2 bound on replica exchange Langevin diffusion) For all η ∈ (0 , 1 ∧ a 4 C 2 ) , we have that
[Equation]
Proof
Consider L t ( β t )
=
(cid:107) β t (cid:107) 2 , where β t = ( β (1) t , β (2) t )
∈ R 2 d .
The proof is marjorly adapted from Lemma 3 in Raginsky et al.
(2017), except that the generalized It ˆ o formula (formula 2.7 in page 29 of Yin & Zhu (2010)) is used to handle the jump operator, which follows that
[Equation]
where ∇ G ( β ) := (cid:18) ∇ L ( β (1) )
∇ L ( β (2) )
(cid:19) and M 1 ( t ) and M 2 ( t ) are two martingales deﬁned in formula 2.
in Yin & Zhu (2010)).
Due to the deﬁnition of L t ( β t ) , we have L t ( β (1) t , β (2) t ) =
L t ( β (2) t , β (1) t ) , which implies that the Jump-inducing drift actually disappears.
Taking expectations and applying the margingale property of the Itˆo integral, we have the almost the same upper bound as Lemmain Raginsky et al.
Combining E [ (cid:107) β 0 (cid:107) 2 ] ≤ log E [ e (cid:107) β 0 (cid:107) 2 ] completes the proof.
Lemma D6 (Exponential integrability of replica exchange Langevin diffusion)
For all τ ≤ 2 a , it follows that
[Equation]
Proof
The proof is marjorly adapted from Lemma 4 in Raginsky et al.
The only difference is that the generalized Itˆo formula (formula 2.7 in Yin & Zhu (2010)) is used again as in Lemma
D5.
Consider L ( t, β t ) =
e (cid:107) β t (cid:107) 2 , where β
=
( β (1) t , β (2) t )
∈ R 2 d .
Due to the special structure that
L ( t, β t ) is invariant under the swaps of ( β (1) t , β (2) t ) , the generator of L ( t, β t ) with swaps is the same as the one without swaps.
Therefore, the desired result follows directly by repeating the steps from Lemma 4 in Raginsky et al.
[Table]
if k mod m = 0 then
[Table]
k , where c k is an estimate for − Cov (cid:0) [Table] |
[Table] Var (cid:0) L (cid:0) [Table]
m [Table] (cid:1)(cid:1) .
end if Bias-reduced swaps Swap β (1)
k +1 and β (2)
k
+1 if u < (cid:101) S
η,m,n , where u ∼ Unif [0 , 1] , and (cid:101) S η,m,n follows
[Table]
until k = k max .
Output: { β (1) i T } (cid:98) k max / T [Table] i = , where T is the thinning factor.
E M ORE E MPIRICAL S TUDY ON I MAGE C LASSIFICATION E. T RAINING COST
The batch size of n = 512 almost doubles the training time and memory, which becomes too costly in larger experiments.
A frequent update of control variates using m = 50 is even more time-consuming and is not acceptable in practice.
The choice of m gives rise to a tradeoff between computational cost and variance reduction.
As such, we choose m = 392 , which still obtains signiﬁcant reductions of the variance at the cost of 40% increase on the training time.
Note that when we set m = 2000 , the training cost is only increased by 8% while the variance reduction can be still at most 6 times on CIFAR10 and 10 times on CIFAR100.
E. A DAPTIVE COEFFICIENT
We study the correlation coefﬁcient of the noise from the current parameter β ( h ) k , where h ∈ { 1 , 2 } , and the control variate β
( h ) m (cid:98)
km (cid:99) .
As shown in Fig.5, the correlation coefﬁcients are only around -0.
due to the large learning rate in the early period.
This implies that VR-reSGHMC may overuse the noise from the control variates and thus fails to fully exploit the potential in variance reduction.
In spirit to the adaptive variance, we try the adaptive correlation coefﬁcients to capture the pattern of the time-varying correlation coefﬁcients and present it in Algorithm 2.
As a result, we can further improve the performance of variance reduction by as much as 40% on
CIFAR10 and 30% on CIFAR100 in the ﬁrst 200 epochs.
As the training continues and the learning rate decreases, the correlation coefﬁcient is becoming closer to -1.
In the late period, there is still 10% improvement compared to the standard VR-reSGHMC.
In a nut shell
[Figure]
we can try adaptive coefﬁcients in the early period when the absolute value of the correlation is lower than 0.5 or just use the vanilla replica exchange stochastic gradient Monte Carlo to avoid the computations of variance reduction
[Figure]
lllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllll
[Figure]
llllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllll
[Figure]
llllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllll
[Figure]
a r i
[Figure]
t
i
[Figure]
o rr e l a t
i
[Figure]
c
[Figure]
ff i c i
[Figure]
l ll llll lll
l l l l ll ll l l llll ll lllllll l l l l lll llll l l l lll ll ll lll ll lllll
[Figure]
l lllll l l l llll lll
lll l l lllllllllllllllllll
[Figure]
c
[Figure]
F M ORE E MPIRICAL S TUDY ON U NCERTAINTY Q UANTIFICATION
To avoid sacriﬁcing the prediction power for the known classes, we also include the uncertainty estimate on CIFAR10 using the Brier score (BS) † and compare it with the estimates on SVHN.
The optimal BS scores on the seen CIFAR10 dataset and the unseen SVHN dataset are 0 and 0.1, respectively.
As shown in Table.2, we see that the scores before calibration in the seen CIFAR10 is much lower than the ones in the unseen SVHN.
This implies that all the models perform quite well in terms of what it knows, although cSGHMC are slightly better than the alternatives.
To alleviate this issue, we propose to calibrate the predictive probability through the temperature scaling (Guo et al., 2017) and obtain much better results.
Regarding the BS score on the unseen dataset, we see that MSGD still performs the worst for frequently making over-conﬁdent predictions; SGHMC performs better but is far away from satisfying.
reSGHMC obtains much better performance by allowing interactions between different chains.
However, the large correction term affects the efﬁciency of the swaps signiﬁcantly.
In the end, our proposed algorithm increases the efﬁciency of the swaps via variance reduction and further improves the highly-optimized BS score based on reSGHMC from 0.29 to 0.27, which is much closer to the ideal 0.1.
Note that the accurate uncertainty estimates of cVR-reSGHMC on the seen dataset is still maintained.
Together with the lowest BS score in the unseen SVHN dataset, cVR-reSGHMC shows its strength in uncertainty quantiﬁcation.
T ABLE 2: U NCERTAINTY ESTIMATES ON SVHN USING CIFAR10 MODELS .
[Table]
G M ODIFIED E XAMPLE 5.
We revisit Example 5.1, and re-run the procedures with temperature τ (1) = 1 . 0 .
In Fig.
6, we present trace plots and kernel density estimates (KDE) of samples generated from VR-reSGLD, reSGLD, and SGLD.
In particular, we run VR-reSGLD with m = 40 , τ (1)
= 1 , τ (2) = 500 , η = 1 e − 5 , and
F = 1 ; reSGLD with the same hyper-parameters as VR-reSGLD except for F = 500 ; and SGLD with η = 1 e − 5 and τ = 1 .
Note that here, we run reSGLD with a greater F than in Example 5.1 in order to prohibit the drastic reduction of the swapping rate which is caused by the pickier target density.
As in Example 5.1, for the ground truth, we run replica exchange Langevin dynamics
[Figure]
with long enough iterations.
In Figs 6(a) and 6(b), we observe that, even though the distribution of interest has a pickier density, our proposed algorithm VR-reSGLD was able to detect both modes and acceptably jump between them.
On the other hand, the competitor algorithm SGLD was trapped in the ﬁrst mode visited and never escaped.
reSGLD was able to jump some times between modes only after considering a substantial factor F = 500 which, according to the theory, introduces bias.