MISSO: M INIMIZATION BY I NCREMENTAL S TOCHASTIC S URROGATE O PTIMIZATION FOR L ARGE S CALE N ONCONVEX AND N ONSMOOTH P ROBLEMS
Anonymous authors
Paper under double-blind review
A BSTRACT
Many constrained, nonconvex and nonsmooth optimization problems can be tackled using the majorization-minimization (MM) method which alternates between constructing a surrogate function which upper bounds the objective function, and then minimizing this surrogate.
For problems which minimize a ﬁnite sum of functions, a stochastic version of the MM method selects a batch of functions at random at each iteration and optimizes the accumulated surrogate.
However, in many cases of interest such as variational inference for latent variable models, the surrogate functions are expressed as an expectation.
In this contribution, we propose a doubly stochastic MM method based on Monte Carlo approximation of these stochastic surrogates.
We establish asymptotic and non-asymptotic convergence of our scheme in a constrained, nonconvex, nonsmooth optimization setting.
We apply our new framework for inference of logistic regression model with missing data and for variational inference of Bayesian variants of LeNetand Resnet-18 on respectively the MNIST and CIFAR-10 datasets.
I NTRODUCTION
We consider the constrained minimization problem of a ﬁnite sum of functions:
[Equation]
where Θ is a convex, compact, and closed subset of R p , and for any i ∈ (cid:74) 1 , n (cid:75) , the function L i : R p
→ R is bounded from below and is (possibly) nonconvex and nonsmooth.
To tackle the optimization problem (1), a popular approach is to apply the majorization-minimization (MM) method which iteratively minimizes a majorizing surrogate function.
A large number of existing procedures fall into this general framework, for instance gradient-based or proximal methods or the Expectation-Maximization (EM) algorithm
(McLachlan & Krishnan, 2008) and some variational Bayes inference techniques (Jordan et al., 1999); see for example (Razaviyayn et al., 2013) and (Lange, 2016) and the references therein.
When the number of terms n in (1) is large, the vanilla MM method may be intractable because it requires to construct a surrogate function for all the n terms L i at each iteration.
Here, a remedy is to apply the Minimization by Incremental Surrogate Optimization (MISO) method proposed by Mairal (2015), where the surrogate functions are updated incrementally.
The MISO method can be interpreted as a combination of MM and ideas which have emerged for variance reduction in stochastic gradient methods (Schmidt et al., 2017).
An extended analysis of MISO has been proposed in (Qian et al., 2019).
The success of the MISO method rests upon the efﬁcient minimization of surrogates such as convex functions, see (Mairal, 2015, Section 2.3).
A notable application of MISO-like algorithms is described in (Mensch et al., 2017) where the authors builds upon the stochastic majorizationminimization framework of Mairal (2015) to introduce a method for sparse matrix factorization.
Yet, in many applications of interest, the natural surrogate functions are intractable, yet they are deﬁned as expectation of tractable functions.
For instance, this is the case for inference in latent variable models via maximum likelihood (McLachlan & Krishnan, 2008).
Another application is variational inference (Ghahramani, 2015), in which the goal is to approximate the posterior distribution of parameters given the observations; see for example (Neal, 2012; Blundell et al., 2015; Polson et al., 2017; Rezende et al., 2014; Li & Gal, 2017).
This paper ﬁlls the gap in the literature by proposing a method called Minimization by Incremental
Stochastic Surrogate Optimization (MISSO) , designed for the nonconvex and nonsmooth ﬁnite sum optimization, with a ﬁnite-time convergence guarantee.
Our work aims at formulating a generic class of incremental stochastic surrogate methods for nonconvex optimization and building the theory to understand its behavior.
In particular, we provide convergence guarantees for stochastic EM and Variational Inference-type methods, under mild conditions.
In summary, our contributions are:
• we propose a unifying framework of analysis for incremental stochastic surrogate optimization when the surrogates are deﬁned as expectations of tractable functions.
The proposed MISSO method is built on the Monte Carlo integration of the intractable surrogate function, i.e., a doubly stochastic surrogate optimization scheme.
• we present an incremental update of the commonly used variational inference and Monte Carlo EM methods as special cases of our newly introduced framework.
The analysis of those two algorithms is thus conducted under this unifying framework of analysis.
• we establish both asymptotic and non-asymptotic convergence for the MISSO method.
In particular, the MISSO method converges almost surely to a stationary point and in O ( n/(cid:15) ) iterations to an (cid:15) -stationary point, see Theorem 1.
• in essence, we relax the class of surrogate functions used in MISO (Mairal, 2015) and allow for intractable surrogates that can only be evaluated by Monte-Carlo approximations.
Working at the crossroads of Optimization and Sampling constitutes what we believe to be the novelty and the technicality of our framework and theoretical results.
In Section 2, we review the techniques for incremental minimization of ﬁnite sum functions based on the MM principle; speciﬁcally, we review the MISO method (Mairal, 2015), and present a class of surrogate functions expressed as an expectation over a latent space.
The MISSO method is then introduced for the latter class of intractable surrogate functions requiring approximation.
In Section 3, we provide the asymptotic and non-asymptotic convergence analysis for the MISSO method (and of the MISO (Mairal, 2015) one as a special case).
Section 4 presents numerical applications including parameter inference for logistic regression with missing data and variational inference for two types of Bayesian neural networks.
The proofs of theoretical results are reported as Supplement.
Notations.
We denote (cid:74) 1 , n (cid:75)
= { 1 , . . . , n } .
Unless otherwise speciﬁed, (cid:107) · (cid:107) denotes the standard
Euclidean norm and (cid:104)· | ·(cid:105) is the inner product in the Euclidean space.
For any function f : Θ → R , f (cid:48) ( θ , d ) is the directional derivative of f at θ along the direction d , i.e.,
[Equation]
The directional derivative is assumed to exist for the functions introduced throughout this paper.
I NCREMENTAL M INIMIZATION OF F INITE S UM N ONCONVEX F UNCTIONS
The objective function in (1) is composed of a ﬁnite sum of possibly nonsmooth and nonconvex functions.
A popular approach here is to apply the MM method, which tackles (1) through alternating between two steps — (i) minimizing a surrogate function which upper bounds the original objective function; and (ii) updating the surrogate function to tighten the upper bound.
As mentioned in the introduction, the MISO method (Mairal, 2015) is developed as an iterative scheme that only updates the surrogate functions partially at each iteration.
Formally, for any i ∈ (cid:74) 1 , n (cid:75) , we consider a surrogate function (cid:98) L i ( θ ; θ ) which satisﬁes the assumptions ( H1 , H2 ):
H1.
For all i ∈ (cid:74) 1 , n (cid:75) and θ ∈ Θ , (cid:98) L i ( θ ; θ ) is convex w.r.t.
θ , and it holds
[Equation]
where the equality holds when θ = θ .
H2.
For any θ i ∈ Θ , i ∈ (cid:74) 1 , n (cid:75) and some (cid:15) > 0 , the difference function (cid:98) e
( θ ; { θ i } ni =1 ) :
=n (cid:80) ni =1 (cid:98)
L i ( θ ; θ i ) − L ( θ ) is deﬁned for all θ ∈ Θ (cid:15) and differentiable for all θ ∈ Θ , where Θ (cid:15) =
{ θ ∈ R d , inf θ (cid:48)
∈ Θ (cid:107) θ − θ (cid:48) (cid:107) <
(cid:15) } is an (cid:15) -neighborhood set of Θ .
Moreover, for some constant
L , the gradient satisﬁes
[Equation]
[Equation] [Equation] [Equation] [Equation]
[Equation]
[Equation] i
[Equation]
i k
A
[-θ ∈ Θ 1 n ni =
-][Equation]
[Equation]
We remark that H1 is a common assumption used for surrogate functions, see (Mairal, 2015,
Section 2.3).
H2 can be satisﬁed when the difference function (cid:98) e ( θ ; { θ i } ni =1 ) is L -smooth, i.e., (cid:98) e is differentiable on Θ and its gradient ∇ (cid:98) e is LLipschitz, ∀ θ ∈ Θ .
H2 can be implied by applying (Razaviyayn et al., 2013, Proposition 1).
The inequality (3) implies (cid:98) L i ( θ ; θ ) ≥
L i ( θ ) >
−∞ for any θ ∈ Θ .
The MISO method is an incremental version of the MM method, as summarized by Algorithm 1, which shows that the MISO method maintains an iteratively updated set of upper-bounding surrogate functions {A ki ( θ ) } ni =1 and updates the iterate via minimizing the average of the surrogate functions.
Particularly, only one out of the n surrogate functions is updated at each iteration [cf.
Line 5] and the sum function 1 n (cid:80) ni =
A k +1 i
( θ ) is designed to be ‘easy to optimize’, which, for example, can be a sum of quadratic functions.
As such, the MISO method is suitable for large-scale optimization as the computation cost per iteration is independent of n .
Under H1, H2, it was shown that the MISO method converges almost surely to a stationary point of (1) (Mairal, 2015, Prop. 3.1).
We now consider the case when the surrogate functions (cid:98) L i ( θ ; θ ) are intractable.
Let Z be a measurable set, p i : Z × Θ → R + a probability density function, r i : Θ × Θ × Z
→ R a measurable function and µ i a σ -ﬁnite measure.
We consider surrogate functions which satisfy H1, H2 and that can be expressed as an expectation, i.e. :
[Equation]
Plugging (5) into the MISO method is not feasible since the update step in Step 6 involves a minimization of an expectation.
Several motivating examples of (1) are given in Section 2.
In this paper, we propose the Minimization by Incremental Stochastic Surrogate Optimization (MISSO) method which replaces the expectation in (5) by Monte Carlo integration and then optimizes the objective function (1) in an incremental manner.
Denote by M ∈ N the Monte Carlo batch size and let { z m ∈ Z } Mm =1 be a set of samples.
These samples can be drawn (Case 1) i.i.d.
from the distribution p i ( · ; θ ) or (Case 2) from a Markov chain with stationary distribution p i ( · ; θ ) ; see Section 3 for illustrations.
To this end, we deﬁne the stochastic surrogate as follows:
[Equation]
and we summarize the proposed MISSO method in Algorithm 2.
Compared to the MISO method, there is a crucial difference in that the MISSO method involves two types of randomness.
The ﬁrst level of randomness comes from the selection of i k in Line 5.
The second level of randomness stems from the set of Monte Carlo approximated functions (cid:101)
A k i ( θ ) used in lieu of A k i ( θ ) in Line 6 when optimizing for the next iterate θ ( k ) .
We now discuss two applications of the MISSO method.
Example 1: Maximum Likelihood Estimation for Latent Variable Model.
Latent variable models (Bishop, 2006) are constructed by introducing unobserved (latent) variables which help explain the observed data.
We consider n independent observations (( y i , z i ) , i ∈ (cid:74) n (cid:75) ) where y i is observed and z i is latent.
In this incomplete data framework, deﬁne { f i ( z i , θ ) , θ ∈ Θ } to be the complete [Equation] [Equation] [Equation]
[Equation]
data likelihood models, i.e., the joint likelihood of the observations and latent variables.
Let
[Equation]
denote the incomplete data likelihood, i.e., the marginal likelihood of the observations y i .
For ease of notations, the dependence on the observations is made implicit.
The maximum likelihood (ML) estimation problem sets the individual objective function L i ( θ ) to be the i -th negated
incomplete data log-likelihood L i ( θ ) :
= − log g
i ( θ ) .
Assume, without loss of generality, that g
i ( θ ) (cid:54) =
for all θ ∈ Θ .
We deﬁne by p
i ( z i , θ ) :
= f i ( z i , θ ) /g i ( θ ) the conditional distribution of the latent variable z
i given the observations
y i .
A surrogate function (cid:98) L i ( θ ; θ ) satisfying H1 can be obtained through writing f i ( z i , θ ) =
f i ( z i , θ ) p
i ( z i , θ ) p i ( z i , θ ) and applying the Jensen inequality:
[Equation]
We note that H2 can also be veriﬁed for common distribution models.
We can apply the MISSO method following the above speciﬁcation of r i ( θ ; θ , z i ) and
p i
( z i , θ ) .
Example 2: Variational Inference.
Let (( x i , y i ) , i ∈ (cid:74) 1 , n (cid:75) ) be i.i.d.
input-output pairs and w ∈
W ⊆ R d be a latent variable.
When conditioned on the input data x = ( x i , i ∈ (cid:74) 1 , n (cid:75) ) , the joint distribution of y =
( y i , i ∈ (cid:74) 1 , n (cid:75) ) and w is given by: [Equation]
[Equation]
Our goal is to compute the posterior distribution p ( w | [Equation]
, x ) . In most cases, the posterior distribution p ( w | y, x ) is intractable and is approximated using a family of parametric distributions, { q ( w, θ ) , θ ∈ Θ } .
The variational inference (VI) problem (Blei et al., 2017) boils down to minimizing the Kullback-Leibler (KL) divergence between q ( w, θ ) and the posterior distribution p ( w | y, x ) : [Equation]
[Equation]
Using (8), we decompose L ( θ ) =
n − 1 (cid:80) [-ni =
-]n i =
1 L i ( θ ) + const .
where:
[Equation]
Directly optimizing the ﬁnite sum objective function in (9) can be difﬁcult.
First, with n (cid:29) 1 , evaluating the objective function L ( θ ) requires a full pass over the entire dataset.
Second, for some complex models, the expectations in (10) can be intractable even if we assume a simple parametric model for q ( w ; θ ) .
Assume that L i is L -smooth.
We apply the MISSO method with a quadratic surrogate function deﬁned as:
[Equation]
It is easily checked that the quadratic function (cid:98) L i ( θ ; θ ) satisﬁes H1, H2.
To compute the gradient ∇L i ( θ ) , we apply the re-parametrization technique suggested in (Paisley et al., 2012; Kingma &
Welling, 2014; Blundell et al., 2015).
Let t : R d × Θ (cid:55)→ R d be a differentiable function w.r.t.
θ ∈ Θ which is designed such that the law of w = t ( z, θ ) is q ( · , θ ) , where z ∼ N d (0 , I ) .
By (Blundell et al., 2015, Proposition 1), the gradient of − r i ( · ) in (10) is:
[Equation]
where for each z ∈ R d , J t θ ( z, θ ) is the Jacobian of the function t ( z, · ) with respect to θ evaluated at θ .
In addition, for most cases, the term ∇ d ( θ ) can be evaluated in closed form as the gradient of the
KL between the prior distribution π ( · ) and the variational candidate q ( · , θ ) .
[Equation]
Finally, using (11) and (13), the surrogate function (6) is given by (cid:101)
L i ( θ ; θ , { z m } Mm =1 ) :=
M − 1 (cid:80) Mm =1 r
i ( θ ; θ , z m ) where { z m } Mm =1 are i.i.d samples drawn from N (0 , I ) .
C ONVERGENCE A NALYSIS
We now provide asymptotic and non-asymptotic convergence results of our method.
Assume:
H3.
For all i ∈ (cid:74) 1 , n (cid:75) , θ ∈ Θ , z i ∈ Z , r i ( · ; θ , z i ) is convex on Θ and is lower bounded.
We are particularly interested in the constrained optimization setting where Θ is a bounded set.
To this end, we control the supremum norm of the MC approximation, introduced in (6), as:
H4.
For the samples { z i,m } Mm =1 , there exist ﬁnite constants C r and C gr such that
[Equation]
M M (cid:88) m = (cid:98) L (cid:48) i ( θ , θ − θ ; θ ) − r (cid:48) i ( θ , θ − θ ; θ , z i,m ) (cid:107) θ − θ (cid:107) (cid:12)(cid:12)(cid:12)(cid:12)(cid:12) 2   for all i ∈ (cid:74) 1 , n (cid:75) , and we denoted by E θ [ · ] the expectation w.r.t.
a Markov chain { z i,m } Mm =1 with initial distribution ξ i ( · ; θ ) , transition kernel Π i, θ , and stationary distribution p i ( · ; θ ) .
Some intuitions behind the controlling terms: It is common in statistical and optimization problems, to deal with the manipulation and the control of random variables indexed by sets with an inﬁnite number of elements.
Here, the controlled random variable is an image of a continuous function deﬁned as r i ( θ ; θ , z i,m ) − (cid:98)
L i ( θ ; θ ) for all z ∈ Z and for ﬁxed ( θ , θ ) ∈
Θ 2 .
To characterize such control, we will have recourse to the notion of metric entropy (or bracketing number) as developed in (Van der Vaart, 2000; Vershynin, 2018; Wainwright, 2019).
A collection of results from those references gives intuition behind our assumption H4, which is classical in empirical processes.
In (Vershynin, 2018, Theorem 8.2.3), the authors recall the uniform law of large numbers:
[Equation]
where F is a class of L -Lipschitz functions.
Moreover, in (Vershynin, 2018, Theorem 8.1.3 ) and (Wainwright, 2019, Theorem 5.22), the application of the Dudley inequality yields:
[Equation]
where N ( F , (cid:107) · (cid:107) ∞ , ε ) is the bracketing number and (cid:15) denotes the level of approximation (the bracketing number goes to inﬁnity when (cid:15) → 0 ).
Finally, in (Van der Vaart, 2000, p.271, Example),
N ( F , (cid:107) · (cid:107) ∞ , ε ) is bounded from above for a class of parametric functions F = f θ : θ ∈ Θ :
N ( F , (cid:107) · (cid:107) ∞ , ε ) ≤ K (cid:18) diam Θ ε (cid:19) d , for all 0 < ε < diam Θ .
The authors acknowledge that those bounds are a dramatic manifestation of the curse of dimensionality happening when sampling is needed.
Nevertheless, the dependence on the dimension highly depends on the class of surrogate functions F used in our scheme, as smaller bounds on these controlling terms can be derived for simpler class of functions, such as quadratic functions.
Stationarity measure.
As problem (1) is a constrained optimization task, we consider the following stationarity measure:
[Equation]
where g + ( θ ) := max { 0 , g ( θ ) } , g − ( θ ) := − min { 0 , g ( θ ) } denote the positive and negative part of g ( θ ) , respectively.
Note that θ is a stationary point if and only if g − ( θ ) =
(Fletcher et al., 2002).
Furthermore, suppose that the sequence { θ ( k ) } k ≥ 0 has a limit point θ that is a stationary point, then one has lim k →∞ g − ( θ ( k ) )
= 0 .
Thus, the sequence { θ ( k ) } k ≥ 0 is said to satisfy an asymptotic stationary point condition .
This is equivalent to (Mairal, 2015, Deﬁnition 2.4).
To facilitate our analysis, we deﬁne τ ki as the iteration index where the i -th function is last accessed in the MISSO method prior to iteration k ,
τ k +1 i k
= k for instance.
We deﬁne:
[Equation]
We ﬁrst establish a non-asymptotic convergence rate for the MISSO method:
Theorem 1.
Under H1-H4.
For any K max ∈ N , let K be an independent discrete
r.v. drawn uniformly from { 0 , ..., K max − 1 } and deﬁne the following quantity:
[Equation]
Then we have following non-asymptotic bounds:
[Equation]
Note that ∆ ( K max ) is ﬁnite for any K max ∈ N .
Iteration Complexity of MISSO.
As expected, the MISSO method converges to a stationary point of (1) asymptotically and at a sublinear rate E [ g ( K ) − ] ≤ O ( (cid:112) ∆ ( K max ) /K max ) .
In other terms, MISSO requires O ( nL/(cid:15) ) iterations to reach an (cid:15) -stationary point when the suboptimality condition, that characterizes stationarity, is E (cid:2) (cid:107) g − ( θ ( K ) )
(cid:107) 2 (cid:3) .
Note that this stationarity criterion are similar to the usual quantity used in stochastic nonconvex optimization, i.e., E (cid:2) (cid:107)∇L ( θ ( K ) )
(cid:107) 2 (cid:3) .
In fact, when the optimization problem (1) is unconstrained, i.e., Θ = R p , then E (cid:2) g ( θ ( K ) ) (cid:3) = E (cid:2) ∇L ( θ ( K ) ) (cid:3) . Sample Complexity of MISSO. Regarding the sample complexity of our method, setting M ( k ) =
k 2 /n 2 , as a non-decreasing sequence of integers satisfying (cid:80) ∞
k
=
{+M − 1 / 2 ( k ) < ∞ , in order to keep ∆ ( K max ) (cid:16) nL , then the MISSO method requires (cid:80) nL/(cid:15)k =
+}
k 2 /n 2 = nL 3 /(cid:15) 3 samples to reach an (cid:15) -stationary point.
Furthermore,we remark that the MISO method can be analyzed in Theorem 1 as a special case of the MISSO method satisfying C r = C gr = 0 .
In this case, while the asymptotic convergence is well known from (Mairal, 2015)
[cf. H4], Eq.
(16) gives a non-asymptotic rate of E [ g ( K ) − ] ≤ O ( (cid:112) nL/K max ) which is new to our best knowledge.
Next, we show that under an additional assumption on the sequence of batch size M ( k ) , the MISSO method converges almost surely to a stationary point:
Theorem 2.
Under H1-H4.
In addition, assume that { M ( k ) } k ≥ 0 is a non-decreasing sequence of integers which satisﬁes (cid:80) ∞
k =
M − 1 / 2 ( k ) < ∞ .
Then:
the negative part of the stationarity measure converges a.s. to zero, i e., lim k →∞ g − ( θ ( k ) )
a.s. = 0 .
the objective value L ( θ ( k ) ) converges a.s. to a ﬁnite number L , i.e., lim k →∞ L ( θ ( k ) )
a.s.
= L .
In particular, the ﬁrst result above shows that the sequence { θ ( k ) } k ≥ 0 produced by the MISSO method satisﬁes an asymptotic stationary point condition .
N UMERICAL E XPERIMENTS 4. B INARY LOGISTIC REGRESSION WITH MISSING VALUES
This application follows Example 1 described in Section 2.
We consider a binary regression setup, (( y i , z i ) ,
i ∈ (cid:74) n (cid:75) )
where y i ∈ { 0 , 1 } is a binary response and z i
= ( z i,j ∈ R , j ∈ (cid:74) p (cid:75) ) is a covariate vector.
The vector of covariates z i =
[ z i, mis , z i, obs ] is not fully observed where we denote by z i, mis the missing values and z i, obs the observed covariate.
It is assumed that ( z i , i ∈ (cid:74) n (cid:75) ) are i.i.d.
and marginally distributed according to N ( β , Ω ) where β ∈ R p and Ω is a positive deﬁnite p × p matrix.
We deﬁne the conditional distribution of the observations y i given z i =
( z i, mis , z i, obs ) as:
[Equation]
[Equation] where for u ∈ R , S ( u ) = 1 / (1+e − u ) , δ = ( δ 0 , · · · , δ p ) are the logistic parameters and ¯ z i =
(1 , z i ) .
Here, θ = ( δ , β , Ω ) is the parameter to estimate.
For i ∈ (cid:74) n (cid:75) , the complete log-likelihood reads:
[Equation]
Fitting a logistic regression model on the TraumaBase dataset: We apply the MISSO method to ﬁt a logistic regression model on the TraumaBase ( http://traumabase.eu ) dataset, which consists of data collected from 15 trauma centers in France, covering measurements on patients from the initial to last stage of trauma.
This dataset includes information from the ﬁrst stage of the trauma, namely initial observations on the patient’s accident site to the last stage being intense care at the hospital and counts more than 200 variables measured for more than 7 000 patients.
Since the dataset considered is heterogeneous – coming from multiple sources with frequently missed entries – we apply the latent data model described in (17) to predict the risk of a severe hemorrhage which is one of the main cause of death after a major trauma.
Similar to (Jiang et al., 2018), we select p = 16 inﬂuential quantitative measurements, on n = 6384 patients.
For the Monte Carlo sampling of z i, mis , required while running MISSO, we run a
Metropolis-Hastings algorithm with the target distribution p ( ·| z
i, obs , y i ; θ ( k ) ) .
We compare in Figure 1 the convergence behavior of the estimated parameters δ and β using
SAEM (Delyon et al., 1999) (with stepsize γ k = 1 /k α
where α = 0 .
6 after tuning)
, MCEM (Wei [Figure] Tanner, 1990) and the proposed MISSO method.
For the MISSO method, we set the batch size to M ( k ) =
10 + k
and we examine with selecting different number of functions in Line 5 in the method – the default settings with 1 (MISSO), 10% (MISSO10) and 50% (MISSO50) minibatches per iteration.
From Figure 1, the MISSO method converges to a static value with less number of epochs than the MCEM, SAEM methods.
It is worth noting that the difference among the MISSO runs for different number of selected functions demonstrates a variance-cost tradeoff.
Though wall clock times are similar for all methods, they are reported in the appendix for completeness.
4. T RAINING B AYESIAN CNN USING MISSO
This application follows Example 2 described in Section 2.
We use variational inference and the
ELBO loss (10) to ﬁt Bayesian Neural Networks on different datasets.
At iteration k , minimizing the sum of stochastic surrogates deﬁned as in (6) and (13) yields the following MISSO update — step (i) pick a function index
i k uniformly on (cid:74) n (cid:75) ; step (ii) sample a Monte Carlo batch { z ( k ) m } M ( k ) m = from N (0 , I ) ; and step (iii) update the parameters, with ˜
w = t ( θ ( k − 1) , z ( k ) m ) , as
[Equation]
k = − 1 M ( k )
M ( k ) (cid:88) m = ∇ w log p ( y i k
| x i k , ˜ w ) + ∇ µ (cid:96) d
( θ ( k − 1) ) , where ˆ µ ( τ k ) (cid:96) = 1 n (cid:80) ni
=1 µ ( τ ki ) (cid:96) and d ( θ ) = n − 1 (cid:80) d(cid:96) =
1 (cid:0) − log( σ ) + ( σ 2 + µ 2 (cid:96) ) /
2 − 1 / 2 (cid:1) .
Bayesian LeNet-5 on MNIST (LeCun et al., 1998):
We apply the MISSO method to ﬁt a Bayesian variant of LeNet-5 (LeCun et al., 1998).
We train this network on the MNIST dataset (LeCun, 1998).
The training set is composed of n = 55 000 handwritten digits, 28 × 28 images.
Each image is labelled with its corresponding number (from zero to nine).
Under the prior distribution π , see (8), the weights are assumed independent and identically distributed according to N (0 , 1) .
We also assume that q ( · ; θ ) ≡ N ( µ, σ 2 I ) .
The variational posterior parameters are thus θ =
( µ, σ ) where µ = ( µ (cid:96) , (cid:96) ∈
(cid:74) d (cid:75) ) where d is the number of weights in the neural network.
We use the re-parametrization as w = t ( θ , z ) = µ
+ σz with z ∼ N (0 , I ) .
Bayesian ResNet-18 (He et al., 2016) on CIFAR-10 (Krizhevsky et al., 2012):
We train here the
Bayesian variant of the ResNet-18 neural network introduced in (He et al., 2016) on CIFAR-10.
The latter dataset is composed of n = 60 000 handwritten digits, 32 × 32 colour images in 10 classes, with 6 000 images per class.
As in the previous example, the weights are assumed independent and identically distributed according to N (0 , I ) .
Standard hyperparameters values found in the literature, such as the annealing constant or the number of MC samples, were used for the benchmark methods.
For efﬁciency purpose and lower variance, the Flipout estimator (Wen et al., 2018) is used.
Experiment Results: We compare the convergence of the Monte Carlo variants of the following state of the art optimization algorithms — the ADAM (Kingma & Ba, 2015), the Momentum (Sutskever et al., 2013) and the SAG (Schmidt et al., 2017) methods versus the Bayes by Backprop (BBB) (Blundell et al., 2015) and our proposed MISSO method.
For all these methods, the loss function (10) and its gradients were computed by Monte Carlo integration based on the reparametrization described above.
The mini-batch of indices and MC samples are respectively set to 128 and M ( k ) =
k .
The learning rates are set to 10 − 3 for LeNet-5 and 10 − 4 for Resnet-18.
Figure 2(a) shows the convergence of the negated evidence lower bound against the number of passes over data (one pass represents an epoch).
As observed, the proposed MISSO method outperforms
Bayes by Backprop and Momentum, while similar convergence rates are observed with the MISSO,
ADAM and SAG methods for our experiment on MNIST dataset using a Bayesian variant of LeNet5.
On the other hand, the experiment conducted on CIFAR-10 (Figure 2(b)) using a much larger network, i.e., a Bayesian variant of ResNet-18 showcases the need of a well-tuned adaptive methods to reach lower training loss (and also faster).
Our MISSO method is similar to the Monte Carlo variant of ADAM but slower than Adagrad optimizer.
Recall that the purpose of this paper is to provide a common class of optimizers, such as VI, in order to study their convergence behaviors, and not to introduce a novel method outperforming the baselines methods.
We report wall clock times for all methods in the appendix for completeness.
C ONCLUSION
We present a unifying framework for minimizing a nonconvex and nonsmooth ﬁnite-sum objective function using incremental surrogates when the latter functions are expressed as an expectation and are intractable.
Our approach covers a large class of nonconvex applications in machine learning such as logistic regression with missing values and variational inference.
We provide both ﬁnitetime and asymptotic guarantees of our incremental stochastic surrogate optimization technique and illustrate our ﬁndings training a binary logistic regression with missing covariates to predict hemorrhagic shock and Bayesian variants of two Convolutional Neural Networks on benchmark datasets.
A P ROOFS OF THE T HEORETICAL R ESULTS A. P ROOF OF T HEOREM
Theorem.
Under H1-H4.
For any K max ∈ N
[Equation]
let K be an independent discrete r
[Equation]
v. drawn uniformly from { 0 , ..., K max − 1 } and deﬁne the following quantity:
[Equation]
Then we have following non-asymptotic bounds:
[Equation]
and
[Equation] [Equation]
[Equation]
Proof We begin by recalling the deﬁnition
[Equation]
Notice that
[Equation]
Furthermore, we recall that
[Equation]
Due to H2, we have
[Equation]
To prove the ﬁrst bound in (16), using the optimality of θ ( k +1) , one has
[Equation]
k
[Equation] [Equation]
≤
[Equation] [Equation]
Let F k be the ﬁltration of random variables up to iteration k , i.e., { i (cid:96) − 1 , { z ( (cid:96) − 1)
i
[Equation]
−
[Equation]
m } M (
[Equation]
− 1) m
=1 , θ ( (cid:96) ) } k(cid:96)
=1 .
We observe that the conditional expectation evaluates to
[Equation]
where the last inequality is due to H
[Equation]
Moreover,
[Equation]
Taking the conditional expectations on both sides of (19) and re-arranging terms give:
[Equation]
Proceeding from (20), we observe the following lower bound for the left hand side
[Equation]
where (a) is due to (cid:98) e ( k ) ( θ ( k ) )
=
[cf. H1], (b) is due to (18) and we have deﬁned the summation in the last equality as − δ ( k )
( θ ( k ) ) .
Substituting the above into (20) yields
[Equation] [Equation]
Observe the following upper bound on the total expectations:
[Equation]
which is due to H4.
It yields
[Equation]
Finally
[Equation]
for any K max ∈ N
[Equation]
we let K be a discrete r
[Equation]
v. that is uniformly drawn from { 0 , 1 , ..., K max − 1 } .
Using H4 and taking total expectations lead to
[Equation]
For all i ∈ (cid:74) 1 , n (cid:75) , the index i is selected with a probability equal to 1 n when conditioned independently on the past.
We observe:
[Equation]
Taking the sum yields:
[Equation]
where the last inequality is due to upper bounding the geometric series.
Plugging this back into (22) yields
[Equation]
L
[Equation]
K max
[Equation]
θ
[Equation]
K max
[Equation]
This concludes our proof for the ﬁrst inequality in (16).
To prove the second inequality of (16)
[Equation]
we deﬁne the shorthand notations g ( k ) :
= g ( θ ( k ) ) , g ( k ) − := − min { 0 , g ( k ) } , g ( k ) + := max { 0 , g ( k ) } .
We observe that
[Equation]
where the last inequality is due to the Cauchy-Schwarz inequality and we have deﬁned (cid:98) L (cid:48)
i ( θ , d ; θ ( τ ki ) )
as the directional derivative of (cid:98) L i ( · ; θ ( τ ki ) ) at θ along the direction d .
Moreover, for any θ ∈ Θ , n
[Equation]
where the inequality is due to the optimality of θ ( k ) and the convexity of (cid:101) L ( k ) ( θ ) [cf. H3].
Denoting a scaled version of the above term as:
[Equation]
We have
[Equation] [Equation]
Since g ( k ) = g ( k ) + − g ( k ) − and g ( k ) + g ( k ) − = 0 , this implies
[Equation]
Consider the above inequality when k = K ,
i
[Equation]
e., the random index
[Equation]
and taking total expectations on both sides gives
[Equation]
We note that
[Equation]
where the ﬁrst inequality is due to the convexity of ( · ) 2 and the Jensen’s inequality
[Equation]
and
[Equation]
where (a) is due to H4 and (b) is due to (24).
This implies
E [ g ( K ) − ] ≤ (cid:115)
[Equation]
and concludes the proof of the theorem.
A. P ROOF OF T HEOREM
Theorem.
Under H1-H4.
In addition, assume that { M ( k ) } k ≥ 0 is a non-decreasing sequence of integers which satisﬁes (cid:80) ∞
k =
M − 1 / 2 ( k ) < ∞ .
Then:
the negative part of the stationarity measure converges a.s. to zero, i.e., lim k →∞ g − ( θ ( k ) )
a.s. = 0 .
the objective value L ( θ ( k ) ) converges a.s. to a ﬁnite number L ,
i
e., lim k →∞ L ( θ ( k ) )
a.s.
= L .
Proof We apply the following auxiliary lemma which proof can be found in Appendix A.3 for the readability of the current proof:
Lemma 1.
Let ( V k ) k ≥ 0 be a non negative sequence of random variables such that E [ V 0 ] < ∞ .
Let ( X k ) k ≥ 0 a non negative sequence of random variables and ( E k ) k ≥ 0 be a sequence of random variables such that (cid:80) ∞
k =
E [ | E k | ] < ∞ .
If for any k ≥ 1 :
[Equation]
[Equation]
then:
(i) for all k ≥ 0 , E
[ V k ] < ∞ and the sequence ( V k ) k ≥ 0 converges a.s. to a ﬁnite limit V ∞ .
(ii) the sequence ( E [ V k ]) k ≥ 0 converges and lim k →∞ E
[ V k ] =
E [ V ∞ ] .
(iii) the series (cid:80) ∞ k =
X k converges almost surely and (cid:80) ∞
k =
E [ X k ] < ∞ .
We proceed from (19) by re-arranging terms and observing that
[Equation]
Our idea is to apply Lemma 1.
Under H1, the ﬁnite sum of surrogate functions (cid:98) L ( k ) ( θ ) , deﬁned in (15), is lower bounded by a constant c k > −∞ for any θ .
To this end, we observe that
[Equation]
(28) is a non-negative random variable.
Secondly, under H
[Equation]
the following random variable is non-negative
[Equation]
Thirdly, we deﬁne
[Equation]
Note that from the deﬁnitions (28), (29), (30), we have V k +1 ≤ V k − X k
+ E k for any k ≥ 1 .
Under H4, we observe that
[Equation] [Equation]
Therefore,
[Equation]
Using (24) and the assumption on the sequence { M ( k ) } k ≥
[Equation]
we obtain that
[Equation]
Therefore, the conclusions in Lemma 1 hold.
Precisely, we have (cid:80) ∞ k =
X k < ∞ and (cid:80) ∞
k =
E [ X k ] < ∞ almost surely.
Note that this implies ∞ >
[Equation]
Since (cid:98) e ( k ) ( θ ( k ) ) ≥ 0 , the above implies
[Equation]
and subsequently applying (18), we have lim k →∞ (cid:107) (cid:98) e ( k ) ( θ ( k ) )
(cid:107) = 0 almost surely.
Finally, it follows from (18) and (26) that
[Equation]
where the last equality holds almost surely due to the fact that (cid:80) ∞ k
=
[Equation]
E [sup θ ∈ Θ | (cid:15) ( k ) ( θ ) | ] < ∞ .
This concludes the asymptotic convergence of the MISSO method.
Finally, we prove that L ( θ ( k ) ) converges almost surely.
As a consequence of Lemma 1, it is clear that { V k } k ≥ 0 converges almost surely and so is { (cid:98) L ( k ) ( θ ( k ) ) } k ≥ 0 , i.e., we have lim k →∞ (cid:98) L ( k )
( θ ( k ) )
=
L .
Applying (31) implies that
[Equation]
This shows that L ( θ ( k ) ) converges almost surely to L .
A. P ROOF OF L EMMA
Lemma.
Let ( V k ) k ≥ 0 be a non negative sequence of random variables such that E [ V 0 ] < ∞ .
Let ( X k ) k ≥ 0 a non negative sequence of random variables and ( E k ) k ≥ 0 be a sequence of random variables such that (cid:80) ∞
k =
E [ | E k | ] < ∞ .
If for any k ≥ 1 :
[Equation]
− 1 − X k − 1 + −
then:
(i) for all k ≥ 0 , E
[ V k ] < ∞ and the sequence ( V k ) k ≥ 0 converges a.s. to a ﬁnite limit V ∞ .
(ii) the sequence ( E [ V k ]) k ≥ 0 converges and lim k →∞ E
[ V k ] =
E [ V ∞ ] .
(iii) the series (cid:80) ∞ k =
X k converges almost surely and (cid:80) ∞
k =
E [ X k ] < ∞ .
Proof We ﬁrst show that for all k ≥ 0 , E [ V k ] < ∞ .
Note indeed that:
[Equation]
showing that E [ V k ] ≤ E
[ V 0 ] + E (cid:104)(cid:80) kj =1 E j (cid:105) <
∞ .
Since 0 ≤ X k ≤ V k − 1 − V k + E k we also obtain for all k ≥ 0 , E [ X k ] < ∞ .
Moreover, since
E (cid:104)(cid:80) ∞
j =1 | E j | (cid:105) < ∞ , the series (cid:80) ∞
j =1 E j converges
a.s.
We may therefore deﬁne:
[Equation]
Note that E [ | W k | ] ≤ E [ V k ] + E (cid:104)(cid:80) ∞
j = k +1 | E j | (cid:105) <
∞ .
For all k ≥ 1 , we get:
[Equation] [Equation]
Hence the sequences ( W k ) k ≥ 0 and ( E [ W k ]) k ≥ 0 are non increasing.
Since for all k ≥ 0 , W k ≥ − (cid:80) ∞
j =1 | E j | > −∞ and E [ W k ] ≥ − (cid:80) ∞
j =1 E [ | E j | ] > −∞ , the (random) sequence ( W k ) k
≥converges a.s. to a limit W ∞ and the (deterministic) sequence ( E [ W k ]) k ≥ 0 converges to a limit w ∞ .
Since | W k | ≤ V
+ (cid:80) ∞
j =1 | E j | , the Fatou lemma implies that:
[Equation]
showing that the random variable W ∞ is integrable.
In the sequel, set U k (cid:44) W 0 − W k .
By construction we have for all k ≥ 0 , U k ≥ 0 , U k ≤
U k +1 and E
[ U k ] ≤ E [ | W 0 | ] + E [ | W k | ] < ∞ and by the monotone convergence theorem, we get:
[Equation] [Equation]
Finally, we have:
[Equation]
and E [
[Equation] [Equation]
showing that E [ W ∞ ] = w ∞ and concluding the proof of (ii).
Moreover, using (35) we have that
W k ≤ W k − 1 − X k which yields:
[Equation]
an concludes the proof of the lemma.
B P RACTICAL D ETAILS FOR THE B INARY L OGISTIC R EGRESSION ON THE T RAUMABASE B. T RAUMABASE DATASET QUANTITATIVE VARIABLES
The list of the 16 quantitative variables we use in our experiments are as follows — age, weight, height, BMI (Body Mass Index), the Glasgow Coma Scale, the Glasgow Coma Scale motor component, the minimum systolic blood pressure, the minimum diastolic blood pressure, the maximum number of heart rate (or pulse) per unit time (usually a minute), the systolic blood pressure at arrival of ambulance, the diastolic blood pressure at arrival of ambulance, the heart rate at arrival of ambulance, the capillary Hemoglobin concentration, the oxygen saturation, the ﬂuid expansion colloids, the ﬂuid expansion cristalloids, the pulse pressure for the minimum value of diastolic and systolic blood pressure, the pulse pressure at arrival of ambulance.
B. M ETROPOLIS -H ASTINGS ALGORITHM
During the simulation step of the MISSO method, the sampling from the target distribution π ( z i, mis ; θ ) :
= p ( z i, mis | z i, obs , y i ; θ ) is performed using a Metropolis-Hastings (MH) algorithm (Meyn & Tweedie, 2012) with proposal distribution q ( z i, mis ; δ ) :
= p ( z i, mis | z i, obs ; δ ) where θ = ( β, Ω) and δ = ( ξ, Σ) .
The parameters of the Gaussian conditional distribution of z i, mis | z
i, obs read:
[Equation]
where we have used the Schur Complement of Ω obs,obs in Ω and noted β mis (resp.
β obs ) the missing (resp.
observed) elements of β .
The MH algorithm is summarized in Algorithm 3.
Algorithm
Calculate the ratio r =
[Equation] [Equation]
[Table] [Table]
[Table] [Table] [Table] [Table]
[Table]
) 6: if u < r then 7:
Accept z
i, mis ,m 8: else 9: z i, mis ,m ← z i, mis ,m − 1 10: end if 11: end for 12: Output: z i, mis ,M
B. MISSO U PDATE
Choice of surrogate function for MISO:
We recall the MISO deterministic surrogate deﬁned in (7):
[Equation]
where θ =
( δ, β, Ω) and θ = (¯ δ, ¯ β, ¯Ω) .
We adapt it to our missing covariates problem and decompose the surrogate function deﬁned above into an observed and a missing part.
Surrogate function decomposition We adapt it to our missing covariates problem and decompose the term depending on θ , while ¯ θ is ﬁxed, in two following parts leading to
[Equation]
−
[Equation]
The mean β and the covariance Ω of the latent structure can be estimated minimizing the sum of
MISSO surrogates ˜ L (2)
i ( β, Ω , θ , { z m } Mm =1 ) , deﬁned as MC approximation of ˆ L (2) i ( β, Ω , θ ) , for all i ∈ (cid:74) n (cid:75) , in closed-form expression.
We thus keep the surrogate ˆ L (2) i ( β, Ω , θ ) as it is, and consider the following quadratic approximation of ˆ L (1) i ( δ, θ ) to estimate the vector of logistic parameters δ :
[Equation]
Recall that:
[Equation]
where ˙ S ( u ) is the derivative of S ( u ) .
Note that ˙ S ( u ) ≤ 1 / 4 and since, for all i ∈ (cid:74) n (cid:75) , the p × p matrix
z i z (cid:62) i is semi-deﬁnite positive we can assume that:
L
For
all i ∈ (cid:74) n (cid:75) and (cid:15) > 0 , there exist, for all z
i ∈
Z , a positive deﬁnite matrix H i ( z i ) :
= 14 ( z i z (cid:62) i + (cid:15)I d ) such that for all δ ∈ R p , − z i z (cid:62) i ˙ S
( δ (cid:62) z i ) ≤
H i ( z i ) .
Then, we use, for all i ∈ (cid:74) n (cid:75) , the following surrogate function to estimate δ :
[Equation]
41) where:
[Equation]
Finally, at iteration k , the total surrogate is:
[Equation]
where for all
i ∈ (cid:74) n (cid:75) :
[Equation]
Minimizing the total surrogate (42) boils down to performing a quasi-Newton step.
It is perhaps sensible to apply some diagonal loading which is perfectly compatible with the surrogate interpretation we just gave.
The logistic parameters are estimated as follows:
[Equation]
where ˜ L (1) i ( δ, θ ( τ ki ) , { z i,m } M ( τki ) m
=1 ) is the MC approximation of the MISO surrogate deﬁned in (41) and which leads to the following quasi-Newton step:
[Equation]
with ˜ D ( k ) = 1 n (cid:80) ni =1 ˜ D
[Equation]
ki [Equation] i and ˜ H ( k ) = 1 n (cid:80) ni
=1 ˜ H ( τ ki ) i .
MISSO updates: At the k -th iteration, and after the initialization, for all i ∈ (cid:74) n (cid:75) , of the latent variables ( z (0) i ) , the MISSO algorithm consists in picking an index
i k uniformly on (cid:74) n (cid:75) , completing the observations by sampling a Monte Carlo batch { z ( k )
i k , mis ,m } M ( k ) m
=1 of missing values from the conditional distribution
p ( z i k , mis | z
i k , obs , y i k ; θ ( k − 1) ) using an MCMC sampler and computing the estimated parameters as follows:
[Equation]
(43) where z ( k ) i,m = ( z ( k ) i, mis ,m , z i, obs ) is composed of a simulated and an observed part, ˜ D ( k ) = 1 n (cid:80) ni
=1 ˜ D ( τ ki ) i , ˜ H ( k ) = 1 n (cid:80) ni
=1 ˜ H ( τ ki ) i and w ( k ) i,m =
z ( k ) i,m ( z ( k ) i,m ) (cid:62) − β ( k ) ( β ( k ) ) (cid:62) .
Besides, ˜ L (1) i ( β, Ω , θ , { z m } Mm =1 ) and ˜ L (2) i ( β, Ω , θ , { z m } Mm =1 ) are deﬁned as MC approximation of ˆ L (1) i ( β, Ω , θ ) and ˆ L (2) i ( β, Ω , θ ) , for all
i ∈ (cid:74) n (cid:75) as components of the surrogate function (40).
B. W ALL CLOCK TIME
We provide Table 1, the running time for each method, plotted in Figure 1, employed to train a logistic regression with missing values on the TraumaBase dataset ( p = 16 inﬂuential quantitative measurements, on n = 6384 patients). The running times are sensibly the same since for each method the computation complexity per epoch is similar.
We remark a slight delay using the MISSO method with a batch size of 1 , as our code implemented in R, is not totally optimized and parallelized.
Yet, when the batch size tends to 100% , we retrieve the duration of MCEM, which is consistent with the fact that MISSO with a full batch update boils down to the MCEM algorithm.
SAEM
[Table]
C P RACTICAL D ETAILS FOR THE I NCREMENTAL V ARIATIONAL I NFERENCE C. N EURAL N ETWORKS A RCHITECTURE
Bayesian LeNet-5 Architecture: We describe in Table 2 the architecture of the Convolutional
Neural Network introduced in (LeCun et al., 1998) and trained on MNIST:
[Table]
Bayesian ResNet-18 Architecture: We describe in Table 3 the architecture of the Resnet-18 we train on CIFAR-10: [Table]
[Table]
A LGORITHMS UPDATES
First, we initialize the means µ (0) (cid:96) for (cid:96) ∈ (cid:74) d (cid:75) and variance estimates σ (0) .
At iteration k , minimizing the sum of stochastic surrogates deﬁned as in (6) and (13) yields the following MISSO update — step (i) pick a function index
i k uniformly on (cid:74) n (cid:75) ; step (ii) sample a Monte Carlo batch { z ( k ) m } M ( k ) m = from N (0 , I ) ; and step (iii) update the parameters as
[Equation] [Equation] (cid:88)
[Equation]
(cid:88) i = [Equation]
[Equation]
(44) where we deﬁne the following gradient terms for all i ∈ (cid:74) 1 , n (cid:75) :
[Equation]
Note that our analysis in the main text does require the parameter to be in a compact set.
For the current estimation problem considered, this can be enforced in practice by restricting the parameters in a ball.
In our simulation for the BNNs example, we did not implement the algorithms that stick closely to the compactness requirement for illustrative purposes.
However, we observe empirically that the parameters are always bounded.
The update rules can be easily modiﬁed to respect the requirement.
For the considered VI problem, we recall the surrogate functions (11) are quadratic and indeed a simple projection step sufﬁces to ensure boundedness of the iterates.
For all benchmark algorithms, we pick, at iteration k , a function index
i k uniformly on (cid:74) n (cid:75) and sample a Monte Carlo batch { z ( k ) m } M ( k ) m
=1 from the standard Gaussian distribution.
The updates of the parameters µ (cid:96) for all (cid:96) ∈ (cid:74) d (cid:75) and σ break down as follows:
Monte Carlo
[Equation]
where ˆ δ ( k ) µ (cid:96) ,i =
ˆ δ ( k − 1) µ (cid:96) ,i and ˆ δ ( k ) σ,i =
ˆ δ ( k − 1) σ,i for i (cid:54)
=
i k
and are deﬁned by (45) for i
=
i k .
The learning rate is set to γ = 10 − 3 .
Bayes By Backprop update : Set
[Equation]
where the learning rate γ = 10 − 3 .
Monte Carlo Momentum update : Set
[Equation]
and
[Equation]
where
[Equation]
and ˆ
[Equation]
where α and γ , respectively the momentum and the learning rates, are set to 10 − 3 .
Monte Carlo ADAM update : Set
[Equation]
where
[Equation]
and ˆ m ( k ) σ = m ( k [Equation]
[Equation]
m
[Equation]
k
[Equation]
k
[Equation]
The hyperparameters are set as follows: γ = 10 − 3 , ρ 1 = 0 .
9 , ρ 2 = 0 .
999 , (cid:15) =
10 − 8 .
C.
W ALL CLOCK TIME
We provide Table 4, the running time for each method, plotted in Figure 2, used to train a Bayesian variant of LeNet-5 on MNIST.
The incremental method as MISSO and MC-SAG displays a similar wall clock time, despite being a bit worse given (a) the initialization that requires to compute a vector of n gradients kept in memory and updated through the iterations and (b) the average operation for each parameters update to compute the aggregated drift term, see (44).
[Table]