A Statistical Online Inference Approach in Averaged Stochastic Approximation
Abstract
In this paper we propose a general framework to perform statistical online inference in a class of constant step size stochastic approximation (SA) problems, including the well-known stochastic gradient descent (SGD) and Q-learning.
Regarding a constant step size SA procedure as a time-homogeneous Markov chain, we establish a functional central limit theorem (FCLT) for it under weaker conditions, and then construct confidence intervals for parameters via random scaling.
To leverage the
FCLT results in the Markov chain setting, an alternative condition that is more applicable for SA problems is established.
We conduct experiments to perform inference with both random scaling and other traditional inference methods, and finds that the former has a more accurate and robust performance.
Introduction
Stochastic approximation (SA) is a class of fixed-point algorithms that enjoys a wide range of applications and research [23, 24, 21, 14].
In general, with the goal of solving an underlying deterministic equation z ( θ ∗ ) = 0 , SA algorithms perform iterative updates based on random approximations of z ( θ ) .
An SA procedure has the following form
[Equation]
where η > 0 is the constant step size and Z t
= Z ( θ t , ξ t +1 ) is a random realization satisfying E ξ t +1 Z ( θ t , ξ t +1 )
=
z ( θ t ) .
Although SA algorithms are initially proposed for solving optimization problems, a large variety of modern machine learning problems such as TD learning [25] and Qlearning [28] can also be solved via stochastic approximation, and hence studies of such problems in an SA perspective have emerged in recent years [6, 15, 20, 18].
The celebrated Polyak-Ruppert averaged estimator [22] is often used to stabilize and accelerate SA algorithms.
Instead of using the last iterate of an SA procedure, the estimator takes an average over all iterates.
It is well-known and extensively studied that if the step size η in (1) is replaced by suitably decaying step sizes, the estimator converges almost surely to the ground-truth solution θ ∗ , and a central limit theorem (CLT) with optimal covariance can be established for the averaged iterates.
Examples include [5], which proposes the batch-means inference method for SGD-type estimator; and [4], which gives the very first asymptotic results and a statistical inference framework for adaptively collected data.
Recently, this result has been extended to a functional central limit theorem (FCLT) in particular settings, such as SGD [16], Local SGD [17], and Q-learning [18].
In these works, however, additional conditions such as (2 + δ ) -th bounded moments of noises are required to obtain such stronger FCLT result, and proofs are based on an elaborated decomposition on the partial sum, which leads to some cumbersome calculation.
This motivates us to consider whether the same results can be established in a simpler way.
In many real applications, constant step size SA algorithms are often used for convenience, and in this case the SA algorithm can be seen as a time-homogeneous Markov chain because each iteration of the
SA procedure is only based on the current state.
Such perspective is also adopted by [9], which gives some nice theoretical results on the constant step size SGD.
Meanwhile, FCLT results are well-established in the Markov chain setting [19, 11, 26, 3, 7, 8].
Due to the fine structure of such
Markov chains, the FCLT may be established with weak conditions, i.e., with only second bounded moments of noises.
Based on the established FCLT, statistical inference via the random scaling method can be implemented.
First proposed in [13] for time series regression and recently adapted in the inference of
SGD parameters [16], random scaling utilizes the whole trajectory of the partial sum of iterates to construct a covariance-like matrix, and studentizes the averaged iterates using this matrix.
This results in a pivotal statistic whose limit distribution does not depend on any other redundant parameter that needs to be further estimated.
Without the need of the additional estimation step, so random scaling turns out to be more robust and efficient compared with traditional methods, such as spectral variance and batch-means [10].
The remainder of this paper is organized as follows.
In Section 2 we propose a general statistical inference framework that includes the establishment of the FCLT and random scaling.
In Section 3, we present applications of the framework in three modern problems including linear stochastic approximation (LSA), stochastic gradient descent (SGD) and Q-learning.
Generally, the PolyakRuppert averaged estimator does not converge exactly to the ground-truth solution θ ∗ , but we give bounds on the bias to show such bias diminishes as the step size η converges to zero.
Numerical experiments are given in Section 4 to illustrate the accuracy and efficiency of our inference framework.
A Statistical Online Inference Framework
In this section, we introduce a general statistical inference framework for a class of constant step size stochastic approximation (SA) problems.
The inference procedure contains three main steps: establishing a functional central limit theorem (FCLT) for the parameters to be estimated, constructing a pivotal statistic via random scaling based on the previously established FCLT, and providing a problem-dependent estimate on the bias between the ground-truth solution and the expectation of the stationary distribution to which the SA process converges.
2.1 Functional Central Limit Theorem
We first state a FCLT established under a Markov chain setting.
Let ( X n ) n ≥ 0 be an ergodic Markov chain on the measurable space ( S, S ) , with transition probability P and stationary distribution π .
For any measurable function g ∈ L 2 π , define the operator Pg ( x )
=
(cid:82) g ( y ) P ( x, d y ) =
E [ g ( X 1 ) | X
= x ] .
By P x or P π we denote the probability law of ( X n ) n ≥ 0 starting from x or having π as its initial distribution; E x or E π the corresponding expectations.
Given f ∈ L 2 π , we define the additive functionals S n ( f ) = (cid:80) n − 1 k =0 f
( X k ) and Y n ( t )
=
n − 12 S
⌊ nt ⌋ ( f ) .
In [8], the authors proved a FCLT for a univariate functional of ( X n ) n ≥ 0 started at a point x .
With slight modification, we can obtain the following multivariate functional version.
Theorem 2.1.
Suppose f : S →
R m satisfies (cid:82) ∥
f ( x ) ∥ 2 π ( d x ) < ∞ and (cid:82) f ( x ) π ( d x ) = 0 .
If there exists 0 < α < 12 such that
[Equation]
then Σ f := lim n →∞ 1 n E π ( S n ( f ) S n ( f ) ⊤ ) exists, and for π -almost every point x ∈ S the sequence n − 12 S n ( f ) converges in distribution, under the probability measure P x , to the d -dimensional
Gaussian distribution N ( 0 , Σ f ) .
Furthermore, the process ( Y n ( t ))
≤ t ≤ 1 converges weakly to ( Σ 12 f B ( t ))
≤ t ≤ 1 on the Skorokhod space D [0 , 1] , where B =
( B ( t ))
t ≥ 0 is the d -dimensional standard Brownian motion.
See Section A.1 for the detailed discussion and proof.
In the sequel, we use d −→ to denote “converge in distribution” and ⇒ to denote “converge weakly in D [0 , 1] ”.
Note that an SA procedure of the form (1) can be seen as a time-homogeneous Markov chain, and therefore in order to establish a FCLT for the parameter sequence ( θ t ) t ≥ 0 , it suffices to prove the existence of its stationary distribution π , and verify the conditions in Theorem 2.1 for f = Id − E π θ .
The condition (2) is a bit harder to verify, so we attempt to ensure it with a stronger Wasserstein contraction condition.
For p, q ≥ 1 , we define the
Wasserstein distance W p,q induced from the L p ( R d ) space by
[Equation]
where Γ( µ, ν ) is the set of all couplings of µ and ν ; and denote P p,q as its corresponding Wasserstein space.
We have the following proposition.
Proposition 2.1.
Suppose for some p, q ≥ 1 , there exists a constant γ ∈ (0 , 1) such that for any µ, ν ∈ P p,q , the Wasserstein contraction
[Equation]
holds.
Suppose in addition that W p,q ( δ ( · ) , π ) ∈ L 2 π , where δ ( · ) is the Dirac measure and π is the stationary distribution.
Then for any f : R d → R m satisfying (cid:82) ∥
[Equation] ( x
[Equation]
d
[Equation]
<
[Equation]
f [Equation] =and
[Equation]
for all x , y ∈ R
[Equation]
for some constant Lip p ( f ) (i.e., each component of f is Lip p ( f ) -Lipschitz in the ∥ · ∥ p norm), the condition (2) holds.
The proof of Proposition 2.1 is deferred to Section A.2.
As we will see later, many SA problems (e.g., linear SA, SGD and Q-learning) possess such a Wasserstein contraction property, and hence the corresponding FCLT holds.
2.2 Random Scaling
Once the FCLT for an SA problem is established, we can use the random scaling method to carry out inference.
For the sequence ( θ t ) t ≥ 0 which is supposed to converge to its stationary distribution π and satisfy the FCLT
[Equation]
we studentize it with a matrix (cid:98) V n constructed as follows
[Equation]
where ¯ θ n := 1 n (cid:80) nk
=
θ k is the Polyak-Ruppert averaged estimator for the parameter θ .
Such studentization leads to a pivotal statistic whose asymptotic distribution does not depend on the parameters of the problem.
Proposition 2.2.
Suppose the sequence ( θ t ) t ≥ 0 converges to its stationary distribution π and its
FCLT (4) holds.
For a matrix R ∈ R ℓ × d of rank ℓ ≤ d , we have
[Equation]
where B ℓ is a ℓ -dimensional standard Brownian motion and (cid:98) V n is defined as in (5) .
Proposition 2.2 follows directly from the continuous mapping theorem.
Statistical inference for E π θ can be carried out via Proposition 2.2.
For example, by setting R =
I
d we can construct a (1 − α ) asymptotic ellipsoidal confidence region centered at ¯ θ n for estimating the whole vector E π θ .
More often, we set R =
e j (the j -th unit vector) to construct a (1 − α ) asymptotic confidence interval for estimating the j -th component of the expectation ( E π θ ) j , so as to avoid heavy computation from the matrix inverse ( R (cid:98) V n R ⊤ ) − 1 .
Corollary 2.1.
Under the same assumptions of Proposition 2.2, we have that
[Equation]
−
[Equation]
where (cid:98) V n,jj is the j -th diagonal element of (cid:98) V n , and q α 2 is the (1 − α/ 2) -th quantile of the following random variable
[Equation]
with ( B 1 ( t ))
t ≥
a one-dimensional standard Brownian motion.
[Table]
The random variable t ∗ is mixed normal with its distribution symmetric around zero.
For easy reference, its critical values are listed in Table 1.[-
-]
It is crucial to explain why inference via random scaling takes advantage over traditional inference procedures (e.g., the spectral variance method and the batch-means method that will be introduced hereafter) in our SA problem.
Traditional estimators are mostly constructed on top of the CLT
[Equation]
[Equation]
where Σ = lim
[Equation]
(cid:80) nk
=1 ( θ k − E π θ ))
⊤ (cid:105) is the asymptotic covariance.
Because Σ is unknown, one attempts to first construct an estimate (cid:98) Σ for Σ based on parameter samples ( θ t ) t ≥ 0 and then establish a confidence interval based on (7) with Σ replaced by (cid:98) Σ .
As a consequence, the estimation error in (cid:98) Σ may cause significant impact on the inference efficiency.
Meanwhile, during the estimation of Σ , additional hyper-parameters are likely to occur, and different choices of such hyper-parameters may affect the inference performance.
Random scaling, on the other hand, studentize the sum 1 √ n (cid:80) nk
=1 ( θ k − E π θ )
to obtain a pivotal statistic.
It does not need the additional estimation step and additional hyper-parameters, resulting in its higher accuracy over traditional estimators.
2.3 Problem-Dependent Bias
The random scaling method mentioned above has already provided an efficient way to carry out inference on ( E π θ ) j (or more generally, any linear combination of the expectation E π θ ).
However, in most nonlinear SA problems there is often a gap between the expectation E π θ and the ground-truth solution θ ∗ .
This issue will be further illustrated by the examples discussed in Section 3.
Therefore, it is theoretically not valid to carry out inference on θ ∗ with a confidence interval constructed following
Corollary 2.1.
However, in the following applications we try to upper bound the bias ∥
E π θ − θ ∗ ∥ with a quantity of order O ( η α ) for some α > 0 .
When the step size is small, such bias is negligible and thus the original inference procedure may still be reasonable.
Case Studies
In Section 2, we have established a general statistical inference framework for constant step size stochastic approximation problems.
We now illustrate several specific applications that are wellknown in optimization and reinforcement learning.
3.1 Linear Stochastic Approximation (LSA)
The first example considers solving a linear system of the form Aθ = b , where A ∈ R d × d and b ∈ R d .
We assume that A is invertible, and there is a unique solution θ ∗ to the linear equation.
Suppose we can observe a sequence of random variables of the form { ( A t , b t ) } t ≥ 1 , assumed to be independent and identically distributed, and unbiased in the sense that E ( A t |F t − 1 )
=
A and E ( b t |F t − 1 ) =
b , where F t − 1 denotes the σ -field generated by { ( A k , b k ) } t − 1 k =1 .
For a given initial vector θ 0 , LSA is formulated as
[Equation]
where η > 0 is the constant step size.
Before presenting our results, we make some assumptions.
Assumption 3.1.
The matrix − A ∈ R d
×
d is Hurwitz, i.e., λ := min 1 ≤ i ≤ d ℜ ( λ i ( A )) > 0 where λ i ( A ) is the i -th eigenvalue of A and ℜ ( λ i ( A )) is its corresponding real part.
Assumption 3.2 (Bounded second moment of noise) .
Denote Ξ t =
A t − A and ξ t = b t − b .
There exist v 2 A > 0 and v 2 b > 0 such that E ∥
Ξ t u ∥ 2 ≤ v 2 A and E | ξ ⊤ t u | 2 ≤ v 2 b for any t ≥ 1 and u ∈ S
d − 1 .
Moreover, Ξ t and ξ t are uncorrelated.
Theorem 3.1.
Suppose Assumptions 3.1 and 3.2 hold for the LSA procedure (8) .
Then there exists a constant η 0 > 0 such that for any η ∈ (0 , η 0 ) ,
the Markov process ( θ t ) t ≥ 0 defined by (8) has a unique stationary distribution π η
for θ ∼ π η , we have E π η
θ
=
θ ∗ ; 3.
the following FCLT holds
[Equation]
where ( B ( t ))
t ≥ 0 is the d -dimensional standard Brownian motion and Σ π η = lim n →∞ n cov π η (¯ θ n ) is the asymptotic covariance matrix.
Remark 3.1.
If we further suppose that { A t } t ≥ 1 and { b t } t ≥ 1 have i.i.d.
entries, then the asymptotic covariance matrix Σ π η has an explicit form (see Section A.3 for details).
This matches the asymptotic results in [20], where the same assumptions are needed and a central limit theorem (CLT) is established, based on an additional requirement that the noises Ξ t and ξ t have bounded (2 + δ ) -th moments for some δ > 0 .
Theorem 3.1 extends this CLT result into a functional version even without such an additional moment requirement.
Note that in the LSA problem, the expectation of the stationary distribution E π η θ exactly matches the unique solution θ ∗ , so that the problem-dependent bias in this case vanishes.
The established FCLT allows us to perform statistical inference directly on θ ∗ via Corollary 2.1.
3.2 Stochastic Gradient Descent (SGD)
Next, we consider the inference problem for a minimizer of an objective function, define by
[Equation]
where f ( θ ) :
=
E [ f ( θ , ξ )] and f ( θ , ξ ) is an unbiased random variable for the objective function.
The constant step size SGD iteratively updates the parameter in the following form
[Equation]
In this case, we make the following assumptions to derive our results.
Assumption 3.3.
f ( · ) is m -strongly convex, and for each ξ , f ( · , ξ ) is L -smooth.
Assumption 3.4 (Regularized gradient noise) .
Define ε ( θ , ξ ) = ∇ f ( θ , ξ ) − ∇ f ( θ ) as the gradient noise at θ , and S =
E [ ε ( θ ∗ , ξ )
ε ( θ ∗ , ξ ) ⊤ ] .
There exists some C > 0 such that
[Equation]
Theorem 3.2.
Suppose Assumptions 3.3 and 3.4 hold for the SGD iterates (9) .
Then there exists a constant η 0 > 0 such that for any η ∈ (0 , η 0 ) ,
the Markov process ( θ t ) t ≥ 0 defined by (9) has a unique stationary distribution π η ; 2.
for θ ∼ π η , we have E π η ∥ θ − θ ∗ ∥ 2 ≤ ηm E π η ∥ ε ( θ , ξ ) ∥ 2 = O ( η ) ; 3.
The following FCLT holds
[Equation]
where ( B ( t ))
t ≥ 0 is the d -dimensional standard Brownian motion and Σ π η = lim n →∞ n cov π η (¯ θ n ) is the asymptotic covariance matrix.
Theorem 3.2 provides all ingredients for performing inference on E π η θ .
In addition, the bias can be bounded via
[Equation]
that is, it is of order O ( η 1 / 2 ) .
3.3 Q-Learning
Finally we consider an application in Q-learning.
An infinite-horizon MDP is represented by
M = ( S , A , γ, P, R, r ) , where S is the state space and A is the action space, γ ∈ (0 , 1) is the discount factor, P : S
×
A → ∆( S ) represents the probability transition kernel, R : S
×
A → [0 , ∞ ) is the random reward, and r : S
×
A → [0 , ∞ ) is the expectation of the reward.
For a given deterministic policy π : S → A , the expected long-term reward is measured by a value function V π and a Q-function Q π defined as follows:
[Equation]
and Q π
[Equation]
s
[Equation]
a ) =
E τ
[Equation]
π
[Equation]
s
[Equation]
(cid:35) , for any state-action pair ( s, a ) ∈ S × A .
Here τ = { ( s t , a t ) } t ≥ 0 is a trajectory of the MDP induced by the policy π and the expectation E τ ∼ π ( · ) is taken with respect to the randomness of the trajectory τ .
The optimal value function V ∗ and optimal Q-function Q ∗ are defined as V ∗ ( s ) = max π V π ( s ) and Q ∗ ( s, a ) = max π Q π ( s, a ) , respectively.
It is well known that Q ∗ is the unique fixed point of the Bellman operator T , i.e., T ( Q ∗ ) =
Q ∗ , where
[Equation]
Now assume access to a generative model.
In particular, in each iteration t ≥ 0 , we collect independent samples of the reward r t ( s, a ) and the next state s t ( s, a ) ∼ P ( ·| s, a ) for every stateaction pair ( s, a ) ∈ S × A .
The constant step size Q-learning maintains a Q-function estimate,
Q t : S ×
A → R , for all t ≥ 0 and updates its entries by the following update rule
[Equation]
12) where η ∈ (0 , 1] is the step size and (cid:98) T t is the empirical Bellman operator constructed by samples collected in the t -th iteration:
[Equation]
Clearly, (cid:98) T t is an unbiased estimate of the Bellman operator T .
Assumption 3.5 (Uniformly bounded reward) .
The random reward R is non-negative and uniformly bounded, i.e., 0 ≤ R ( s, a ) ≤ 1 almost surely for all ( s, a ) ∈ S × A .
Theorem 3.3.
Suppose Assumption 3.5 holds for the Q-learning update rule (12) , and assume access to a generative model for each state-action pair ( s, a ) ∈ S × A .
Let Q ∈ R |S|×|A| be the matrix form of the Q-function Q ( s, a ) .
Then there exists a constant η 0 > 0 such that for any η ∈ (0 , η 0 ) ,
the Markov process ( Q t ) t ≥ 0 defined by (12) has a unique stationary distribution Q η ; 2. for Q ∼ Q η , we have E Q η ∥ Q − Q ∗ ∥ ∞
=
O ( η 1 / 2 ) ; 3.
The following FCLT holds
[Equation]
where ( B ( t ))
t ≥ 0 is the d -dimensional standard Brownian motion and Σ Q η
[Equation] lim n →∞ n cov Q η
( ¯ Q n ) is the asymptotic covariance matrix.
Experiments
In this section we conduct the empirical analysis of the random scaling method via Monte Carlo experiments.
Two traditional inference methods, namely, the spectral variance method and the batchmeans method, are also examined for comparison.
These two methods are popular in estimating the asymptotic variance of a Markov chain [10], and confidence intervals can be constructed based on the CLT (7) with Σ replaced by an estimator (cid:98) Σ .
From now on, we represent the concerned parameter as β for clarity.
The spectral variance estimator is based on the fact that Σ = (cid:80) + ∞ s =
−∞ cov π η ( β t , β t + s ) .
It is constructed as
[Equation]
where w n ( · ) is the lag window, b n is the truncation point, and
[Equation]
which is a consistent estimator of the lag s autocovariance γ ( s ) :
= cov π η ( β t , β t + s ) .
Under mild conditions on w n ( · ) and b n , (cid:98) Σ SV is a consistent estimator of Σ [10].
In our experiments we choose w n ( · ) to be the Tukey-Hanning window w n ( k ) = (cid:0) 12 + 12 cos ( π | k | /b n ) (cid:1) 1 {| k | <b n } with b n = n 3 / 4 .
The batch-means estimator is another consistent estimator of Σ .
It first divides a total of n = a n b n iterates into a n batches, and calculate the mean of each batch ¯ β k :
k
+1 := 1 b n (cid:80) b n i =1 β kb n + i for k = 0 , . . .
, a n − 1 .
The batch-means estimator is then defined as
[Equation]
When a n and b n are allowed to increase as n increases (e.g., by setting a n = b n = ⌊√ n ⌋ ), (cid:98) Σ BM is a consistent estimator of (cid:98) Σ [12].
In our experiments we choose a n = b n = ⌊√ n ⌋ .
In the following, we consider the constant step size SGD on two baseline models: linear regression and logistic regression.
Additional experiments for Q-learning on Grid World are included in
Appendix D. All experiments are run on Intel Xeon 14-core CPUs.
The performance of the spectral variance method, batch-means method, and random scaling method are compared in three aspects: the coverage rates for E π η β and β ∗ , and the lengths of confidence intervals.
The nominal coverage rate is chosen as 95% .
For brevity, we focus on the first coefficient β 1 hereafter.
The implementation code is available in https://github.com/bangoz/sa-inference .
Linear Regression The data are generated from
[Equation]
where x t is a d -dimensional covariate following the multivariate normal distribution N (0 , I d ) , ε t is the noise from N (0 , 1) , and β ∗ is equi-spaced on the interval [0 , 1] .
The dimension of x is set as d = 5 , 20 .
The constant learning rate is set as η = 0 .
The initial value of β 0 is set as zero.
The simulation results are based on 1,000 replications.
Our experimental design is the same as that of [29, 16], except that our learning rates η are smaller than the initial learning rates in their settings, which are chosen to be η 0 = 0 .
This is inevitable because constant step size SGD requires smaller learning rates to ensure convergence.
Note that SGD for linear regression is also a case of LSA, so the expectation of the stationary distribution is equal to the true parameter, i.e., E π η β = β ∗ .
Table 2 summarizes partial simulation results for d = 5 .
Full results for d = 5 , 20 can be found in
Appendix C. Overall, the random scaling method is satisfactory in that its coverage rates are the closest to the nominal one (95%) across all experimental settings.
Nevertheless, the average lengths of confidence intervals (CI) constructed via random scaling are always larger than that of both the spectral variance method and the batch-means method.
The batch-means method, on the contrary, shows the lowest coverage rates and the smallest average CI lengths, and its performance varies hugely when we choose different step sizes.
When η = 0 .
01 it performs the worst, deviating from the nominal rate about 10% even at n = 40 ,
(when d = 5 ) or at n = 100 , 000 (when d = 20 ).
Comparing our experiment results with that of [16], where the authors use SGD with decaying step sizes of the form η t = η 0 t − a , we find that average CI lengths in our settings are a bit larger.
Such observation is also reasonable, because SGD with particular decaying step sizes is shown to converge to the true parameter almost surely, and so does its corresponding Polyak-Ruppert averaged estimator [22]; but in the case of constant step sizes, the iterates converge to a non-degenerate stationary distribution, and such non-degeneration adds to additional fluctuation of the Polyak-Ruppert averaged estimator.
Logistic Regression
The data are generated from
P (
[Equation]
All the settings are the same as in linear regression.
Note that for logistic regression, the expectation of the stationary distribution is not necessarily equal to the true parameter, i.e., E π η β ̸ = β ∗ .
However, since the difference is diminishing as η → 0 , the confidence intervals may still be useful for inference on β ∗ .
Therefore, we include the coverage rates for β ∗ 1 in our results.
Table 3 summarizes partial simulation results for d = 5 .
Full results can be found in Appendix C.
These results are similar to those in linear regression.
Coverage rates for β ∗ 1 are generally smaller than coverage rates for E π η β ∗ 1 as expected, and their differences become larger as the step size gets larger.
Nevertheless, coverage rates for β ∗ 1 based on random scaling deviates within 5% of the nominal one in most settings, which indicates that such confidence intervals are still reasonable when the step size is small.
The batch-means method is still not robust to different step sizes, because the coverage rates even fall below 60% when η = 0 .
Conclusion
In this paper we have studied statistical inference on a class of constant step size stochastic approximation (SA) problems.
Based on a FCLT in the context of Markov chains, we have established the corresponding FCLTs for LSA, SGD, and Q-learning.
A sufficient condition for the FCLT has been stated and proved for ease of application.
Various inference methods including the random scaling method, spectral variance method and batch-means method have been conducted in experiments, based on the established FCLTs.
We have shown that random scaling performs well in terms of coverage rates, and is also robust against different step sizes.
There are also limitations and issues that need to be further investigated.
First, except in trivial cases, iterates converge to a stationary distribution whose expectation does not equal to the true parameter, and such systematic bias prohibits efficient inference procedures for the true parameter we are concerned about.
It remains open whether there exist practical ways to mitigate the impact of such bias.
Probabilistic tools such as the large deviation theory might help construct valid confidence intervals for the true parameter.
Second, recent works considering SA problems with decaying step sizes, such as [16, 17, 18], prove the FCLT by manually dividing the estimator into several terms, applying martingale CLT to one term, and uniformly bounding other terms. It is
[Table] [Table] interesting to consider whether such proofs can be simplified by regarding these SA problems as time-inhomogeneous Markov chains and applying techniques in the Markov chain setting.
This work has been supported by the National Key Research and Development Project of China (No. 2020AAA0104400) and the National Natural Science Foundation of China (No. 12271011)
Checklist
For all authors... (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?
[Yes] (b) Did you describe the limitations of your work?
[Yes] See Section 5.
(c) Did you discuss any potential negative societal impacts of your work?
[No] Our work is theoretical and has no potential negative social impacts.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes] 2.
If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
[Yes] (b) Did you include complete proofs of all theoretical results?
[Yes] 3.
If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[Yes] (b) Did you specify all the training details (e
g., data splits, hyperparameters, how they were chosen)?
[Yes] (c) Did you report error bars (e
g., with respect to the random seed after running experiments multiple times)?
[Yes]
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
[Yes] 4.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
[N/A] (b) Did you mention the license of the assets?
[N/A] (c) Did you include any new assets either in the supplemental material or as a URL?
[N/A] (d) Did you discuss whether and how consent was obtained from people whose data you’re using/curating?
[N/A] (e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content?
[N/A] 5.
If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?
[N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
[N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
[N/A]