Mean Estimation in High-Dimensional Binary Markov Gaussian Mixture Models
Abstract
We consider a high-dimensional mean estimation problem over a binary hidden
Markov model, which illuminates the interplay between memory in data, sample size, dimension, and signal strength in statistical inference.
In this model, an estimator observes n samples of a d -dimensional parameter vector θ ∗ ∈ R d , multiplied by a random sign S i ( 1 ≤ i ≤ n ), and corrupted by isotropic standard Gaussian noise.
The sequence of signs { S i } i ∈ [ n ] ∈ {− 1 , 1 } n is drawn from a stationary homogeneous Markov chain with flip probability δ ∈ [0 , 1 / 2] .
As δ varies, this model smoothly interpolates two well-studied models: the Gaussian
Location Model for which δ = 0 and the Gaussian Mixture Model for which δ = 1 / 2 .
Assuming that the estimator knows δ , we establish a nearly minimax optimal (up to logarithmic factors) estimation error rate, as a function of ∥ θ ∗ ∥ , δ, d, n .
We then provide an upper bound to the case of estimating δ , assuming a (possibly inaccurate) knowledge of θ ∗ .
The bound is proved to be tight when θ ∗ is an accurately known constant.
These results are then combined to an algorithm which estimates θ ∗ with δ unknown a priori, and theoretical guarantees on its error are stated.
Introduction
Memory between data samples is ubiquitous in practical applications, as data collected from networks or sampled time series inevitably inherit spatial or temporal statistical dependencies.
Numerous examples arise in imaging, meteorology, health care, finance, social science, and so on [Glaeser et al., 1996, Bertrand et al., 2000, Sacerdote, 2001, Duflo and Saez, 2003, Christakis and Fowler, 2013].
In principle, prior knowledge of the existence of such memory can be used to improve the performance of statistical estimators compared to the performance obtained in memoryless models.
Development and analysis of statistical inference algorithms for models with memory has been extensively explored from the algorithmic perspective, and computationally efficient algorithms such as Baum-Welch and message-passing were developed [Ephraim and Merhav, 2002, van Handel, 2008, MacKay, 2003,
Wainwright and Jordan, 2008].
It was also extensively studied from the theoretical perspective in the classical, fixed-dimensional and asymptotic regime, e.g., Györfi et al.
[2002, Chapter 27 and references therein].
However, much less is understood about the high dimensional, non-asymptotic regime, which is of paramount importance in modern applications, and the focus of current extensive research [Vershynin, 2018, Wainwright, 2019].
As we exemplify in this paper, the error in such estimation problems depends in an intricate way on the interplay between the number of samples, the dimension of the vector parameters to be estimated, the noise level (signal-to-noise ratio), and the level of memory between the samples.
An ever popular and fundamental model is the Gaussian mixture model, in which memory exists between samples whenever the latent variables determining the component of each sample are dependent.
Numerous recent papers, e.g., Balakrishnan et al.
[2017], Xu et al.
[2016], Klusowski and Brinda [2016], Jin et al.
[2016], Dwivedi et al.
[2020b], Dwivedi et al.
[2018], Dwivedi et al.
[2020a], Zhao et al.
[2018], Yan et al.
[2017], Weinberger and Bresler [2022] have focused on highdimensional memoryless models, and analyzed computationally efficient estimation algorithms, most notably the expectation maximization (EM) algorithm.
Specifically, the seminal Balakrishnan et al.
[2017] has provided theoretical guarantees for EM in memoryless models, though without proving minimax optimality, which was subsequently established in Wu and Zhou [2019].
As a notable exception, Yang et al.
[2015] has generalized the analysis of Balakrishnan et al.
[2017] to a hidden
Markov model (HMM) that has memory, yet again, without determining how the minimax error rate depends on the number of samples, dimension, noise level, and the amount of memory.
In this paper, we address the question of precise characterization of the minimax error rate in terms of these parameters, in the context of a high-dimensional Gaussian HMM.
We next turn to formally define this model and the estimation problem, describe known results, and then present our contributions.
Our obtained results illuminate the opportunities and challenges associated with optimal inference in high dimensional models with memory.
1.1 Problem formulation
Let S n 0 :=
( S 0 , S 1 , . . .
, S n ) be the following homogeneous binary symmetric Markov chain, S i ∈ {− 1 , 1 } , P [ S 0 = 1] = 1 / 2 and S i = (cid:26) S i [Equation]
[Equation]
for i ∈ [ n ] := { 1 , . . . , n } , and where δ ∈ [0 , 1] is the flip probability of the binary Markov chain.
We also denote ρ := 1 − 2 δ ∈ [ − 1 , 1] which is the correlation between adjacent samples ρ = E
[ S i S
i +1 ] .
At each time point i ∈ [ n ] , a sample of a d -dimensional Gaussian mixture model is observed
[Equation]
where Z i ∼ N (0 , I d ) is an i.i.d.
sequence, independent of S n 0 , and where θ ∗ ∈ R d , d ≥ 1 .
At its two extremes, this model degenerates to one of two fundamental models, which are well studied.
When δ = 0 , the memory length is infinite, and the sign S
= S 1 · · · = · · · = S n is fixed.
Thus, up to this sign ambiguity, the model (2) is the standard Gaussian location model (GLM), which is essentially a memoryless model (and exactly so if S 0 is known).
When δ = 12 , the signs S n 0 are i.i.d.
and have no memory at all.
The model (2) is then a Gaussian mixture model (GMM) with two symmetric components, which is also a memoryless model.
In all other cases, 0 < δ < 12 (or 12 < δ < 1 ), the model is a simple version of a HMM.
The inference problem we consider in this paper is the estimation of θ ∗ ∈ R d , under the loss function
[Equation]
that is, the Euclidean distance error under a possible sign ambiguity.
1 An intermediate goal (or an additional problem) is to estimate δ , under the regular absolute error loss function | ˆ δ − δ | .
The fundamental limits of this estimation problem will be gauged by the local minimax rate, which is the maximal decrease rate of the loss possible for any estimator, given n samples, at dimension d ≥ 2 , for signal strength ∥ θ ∗ ∥ = t , and under flip probability δ .
Specifically, for d ≥ 2 it is defined as
[Equation]
For general d ≥ 1 , the global minimax rate is defined with the condition ∥
θ ∗ ∥ = t replaced by ∥ θ ∗ ∥≤ t (this condition trivializes the estimator for d = 1 ).
1.2 Known minimax estimation errors rates for GLM and GMM
Before delving into models with memory ( 0 < δ < 1 / 2 ), we review known results on the minimax rates in memoryless high dimensional Gaussian models – the GLM ( δ = 0 ) and the GMM ( δ = 1 / 2 ).
We refer the reader to note that the minimax error rate in these models may undergo two possible phase transitions – one as t increases, and the other one as d increases.
We also remark that the regime of interest is that of low-separation ( t ≲ 1 ), in which accurately detecting the components is impossible, yet parameter estimation with vanishing loss is possible.
At the first extreme, the local minimax rate for the GLM ( δ = 0 ) is the usual parametric error rate
[Equation]
This is achieved by the trivial estimator ˆ θ = 0 if t ≤ (cid:113) dn and the simple empirical average estimator ˆ θ = 1 n (cid:80) ni
=1 X i if t ≥ (cid:113) dn (see [Wu, 2017, Sec.
III-9]).
The rate Θ( (cid:113) dn ) is then the global minimax rate, i.e., the largest error over t =
∥ θ ∗ ∥ > 0 .
This model does not have a phase transition with dimension.
At the other extreme, the GMM model ( δ = 12 ) undergoes a phase transition at the dimension d = n .
At low dimension, d ≤ n , the minimax rate was neatly shown in [Wu and Zhou, 2019, Appendix B] to be
[Equation]
whereas at high dimension d ≥ n , it is as for the Gaussian location model in (5), i.e., M GMM ( n, d, t ) =
M GLM ( n, d, t ) .
Hence, at this high dimensional regime, the loss does not vanishes by increasing the signal strength t , or by increasing the number of samples n .
The loss in (6) is achieved by the trivial estimator ˆ θ = 0 if t ≤ (cid:0) dn (cid:1) 1 / 4 and by an estimator given by a properly scaled and shifted principal component of the empirical covariance matrix of X n 1 :=
( X 1 , X 2 , . . .
, X n ) , if t ≥ (cid:0) dn (cid:1) 1 / 4 .
For the
GMM at low dimension, d ≤ n , the global minimax rate is ( dn ) 1 / 4 , which is worse than the minimax rate of the Gaussian location model (cid:113) dn .
Therefore, the GMM has worse estimation performance compared to the GLM from three aspects:
First, at low dimension, d ≤ n , it has a larger global minimax rate ( dn ) 1 / 4 compared to the parametric error rate of the GLM, (cid:113) dn ; Second, at low dimension, d ≤ n , parametric error rate is achieved only for constant separation t ≥ 1 ; Third, the transition to the high dimension regime occurs at d = n .
As is intuitively appealing from a “data-processing” reasoning, a Markov model with flip probability δ ′ should allow for lower estimation error of θ ∗ compared to a Markov model with δ > δ ′ .
Indeed, and as a specific simple example, any Markov model with δ < 12 can be easily transformed to a GMM model by randomizing the signs of each of the samples by an independent Rademacher variable.
Thus we may deduce, e.g., that since at high dimension ( d ≥ n ) the GLM and the GMM have the same minimax rates, the minimax rates for d ≥ n are in fact as in (5) for any δ ∈ [0 , 1] .
We thus henceforth exclusively focus on the regime d ≤ n .
As we show, it is generally true that the improvement in estimation error when δ is reduced is less profound as the dimension increases.
1.3 Contributions
We first consider the case in which δ is known to the estimator of θ ∗ .
For this case, we analyze the loss of an estimator that is based on a computation of the principal component of a properly chosen empirical covariance matrix.
We show (Theorem 1) that, at low dimension, d ≤ δn , it achieves a local minimax rate of [Equation]
 
[Equation]
and at high dimension, d ≥ δn , it is as for the Gaussian location model in (5).
The rate of this estimator is then further shown to be asymptotically optimal (up to a logarithmic factor) via a minimax lower bound (Theorem 2).
Evidently, the loss in (7) smoothly interpolates the rates of the GLM in (5) and the GMM in (6).
Moreover, it is evident that the loss is improved with the decrease of δ from all three aspects previously mentioned.
First, at low dimension, d ≤ δn , the global minimax rate is
(cid:0) δdn (cid:1) 1 / 4 (obtained by equating the first and second cases in (7)).
Hence, reducing the flip probability from δ = Θ(1) to δ = dn smoothly reduces the global minimax rate from the parametric error rate Θ( (cid:113) dn ) of the GLM to the Θ( (cid:0) dn (cid:1) 1 / 4 ) of the GMM.
Second, at low dimension, d ≤ δn , the minimal signal strength required to obtain parametric error rate is t =
Θ( √ δ ) (obtained from the third case in (7)).
This, again, improves by decreasing δ , and matches the extremes t =
Θ( (cid:113) dn ) of GLM and t = Θ(1) of GMM.
Third, the transition to the high dimension regime occurs at d = δn , which is again better with lower δ , and matches the transition point of the GMM given by d = n .
This lower transition point allows us to achieve the error rate of the GLM for any signal strength, even in a regime in which the loss O ( (cid:113) dn ) vanishes with n → ∞ (unlike for GMM, δ = 12 ); specifically, this occurs whenever d ≥ δn yet d = o ( n ) .
Beyond the formal proof, Appendix A provides a heuristic justification for why the minimax error is naturally expected to scale as in (7).
Second, as a step towards the removal of the assumption that δ is known to the estimator, we consider the complementary problem of estimating δ whenever an estimate θ ♯ of θ ∗ is available (which can be either exact θ ♯ = θ ∗ , or inaccurate θ ♯ ̸
= θ ∗ ).
We propose a simple estimator for δ , and analyze its error in case of a mismatch (Theorem 4).
We then specify this result to the matched case θ ♯ = θ ∗ and show that in the non-trivial regime ( ∥ θ ∗ ∥ ≲ 1 ) its error rate is ˜ O ( 1 ∥ θ ∗ ∥ 2 (cid:113) 1 n ) .
We then proceed to show an impossibility lower bound of Ω( (cid:113) 1 n ) (Proposition 6) for this error rate.
The precise dependence of the estimation error rate of δ on ∥ θ ∗ ∥ is therefore not precisely determined, and we discuss the challenges in settling this matter.
Third, we consider the case in which the estimator of θ ∗ has no prior knowledge of δ .
We propose a three-step algorithm for this case (Algorithm 1).
First, a (possibly) gross estimate ˆ θ ( A ) of θ ∗ is computed based on third of the samples, assuming the worst case of δ = 12 .
Then, an estimate ˆ δ ( B ) of δ is computed using another third of the samples, assuming the estimate ˆ θ ( A ) .
Finally, a refined estimate ˆ θ ( C ) of θ ∗ is obtained by (essentially) assuming that δ is ˆ δ ( B ) .
At each of the steps above, the algorithm may stop and decide to return its current estimate when it determines that no further improvement is possible by moving on to the next steps.
We analyze the loss of this algorithm (Theorem 7), and show that this algorithm is capable of partially achieving the gains associated with the case of known δ .
Our technique of partitioning the samples to blocks according to the dependence structure of the
Markov chain renders the possibility of understanding the minimax rate of more general models, such as mixtures with multiple components
[Doss et al., 2020] and memory, Ising models [Daskalakis et al., 2019], Boltzmann machines [Bresler et al., 2019], Markov random fields, etc.
Our findings in the mean estimation setting also stand in contrast to the data wastage phenomenon observed in the linear regression setting in the prior work Bresler et al.
1.4 Additional related work
Both the GLM [Johnstone, 2002, Tsybakov, 2008] and GMM [Lindsay, 1995, McLachlan et al., 2019] are classic models which were well-explored from numerous perspectives.
In the last few years, there is a surge of interest in the non-asymptotic performance analysis of computationally efficient estimation algorithms for this estimation task.
For example, Moitra and Valiant [2010], Kalai et al.
[2010], Anandkumar et al.
[2014], Hardt and Price [2015], Wu and Yang [2020] have analyzed methodof-moments-based algorithms, and various other papers considered the EM algorithm
[Balakrishnan et al., 2017, Xu et al., 2016, Jin et al., 2016, Klusowski and Brinda, 2016, Weinberger and Bresler, 2022, Dwivedi et al., 2020b,a, 2018, Zhao et al., 2018, Yan et al., 2017].
Specifically, the local minimax rate for GMM in (6) was determined in [Wu and Zhou, 2019] as a benchmark for the operation of the EM algorithm.
The model (2) is a simple instance of a HMM [Ephraim and Merhav, 2002, van Handel, 2008] in high dimensions.
Parameter estimation in such models is practically performed via the Baum-Welch algorithm
[Baum et al., 1970], which is a computationally efficient version of EM for HMMs.
To the best of our knowledge, there were hardly any attempts to characterize the minimax rates in such models, with the exception of [Aiylam, 2018] and [Yang et al., 2015], previously mentioned.
In
[Aiylam, 2018], a local version of the Baum-Welch algorithm was proposed, and vanishing error of the convergence of the estimate to the true parameter was established for both a population version as well as a finite-sample version.
In [Yang et al., 2015], general bounds on the performance of
Baum-Welch algorithm were specified to the Gaussian model with Markov signs (2) considered here.
It was shown that the Baum-Welch algorithm achieves a parametric error rate, and converges in a finite number of iterations
[Yang et al., 2015, Corollary 2].
However, the qualifying condition for the estimation error bound of [Yang et al., 2015] is that t =
∥
θ ∗ ∥ ≳ log 1 1 − (1 − 2 δ ) 2 , that is, a non-trivial separation when δ is constant, which further blows up as δ ↓ 0 .
By contrast, in this paper, our goal is to characterize the estimation error in the regime of δ and t =
∥ θ ∗ ∥ in which the minimax rate is affected by these parameters, and this requires analyzing vanishing δ and t .
More broadly, there is a growing interest in advancing the quantitative understanding of the performance of statistical learning and inference with dependent data.
Bresler et al.
[2020] studied linear regression with Markovian covariates and characterized the minimax error rate in terms of the mixing time of the Markov chain.
A stochastic gradient descent-style algorithm adapted to the
Markov setting was shown to be minimax optimal.
Statistical estimation problems including linear and logistic regression with more general network dependencies among response variables were studied by Daskalakis et al.
[2019] and Kandiros et al.
Learnability and generalization bounds were derived by Dagan et al.
[2019] for dependent data satisfying the so-called Dobrushin’s condition.
1.5 Notation conventions
For a vector v ∈ R d , ∥ v ∥ is the Euclidean norm.
For a positivedefinite matrix A , λ max ( A ) and v max ( A ) are the maximal eigenvalue and the associated eigenvector (of unit norm) of A .
Unless otherwise stated, the constants involved in Bachmann-Landau notation are numerical, and do not depend on the parameters ( n, d, δ, t ) .
It holds that a ≳ b (resp.
a ≲ b ) if there exists a constant c >(resp.
C > 0 ) such that a ≥ cb (resp.
a ≤ Cb ).
If a ≲ b and a ≳ b .
then a ≍ b .
Integer constraints (ceiling and floor) on large quantities that do not affect the results are omitted for brevity.
For a real numbers a, b the shorthand notation a ∨ b := max { a, b } , a ∧ b :=
min { a, b } and ( a ) + := a ∨ 0 is used.
A sequence of objects X 1 , · · · , X n is denoted by X n 1 . Expectation, variance and probability are denoted by E , V and P , respectively.
Equality in distribution of random variables X
and Y is denoted by X d = Y .
All logarithms log are to the base e .
Mean estimation for a known flip probability
In this section, we consider the problem of estimating θ ∗ whenever δ is exactly known to the estimator.
In that case, it may be assumed w.l.o.g. that δ ∈ [0 , 12 ] , as otherwise one may negate each of the even samples to obtain an equivalent model with δ replaced with 1 − δ .
Hence also ρ ∈ [0 , 1] .
We next describe an estimator for this task, state a bound on its performance, and then show that it matches (up to a logarithmic factor) an impossibility lower bound.
The estimator operationally interpolates and therefore simultaneously generalizes the empirical average estimator (30) and the (properly scaled) principal component estimator (32) analyzed in [Wu and Zhou, 2019, Appendix B].
It degenerates to the latter estimators if δ ↓ 0 or δ ↑ 12 .
Specifically, the estimator partitions the sample into blocks of equal length k each (which will later be set to k = 18 δ , according to the mixing time of the Markov chain S n 0 ).
Let ℓ denote the number of blocks respectively, so that kℓ = n .
Let I i
= { ( i − 1) k
+ 1 , ( i − 1) k
+ 2 , · · · , ik } denote the indices of the i th block.
Further, let { R i } i ∈ [ ℓ ] be an i.i.d.
Rademacher sequence ( R i ∼ Uniform {− 1 , 1 } ), and let
[Equation]
denote the average of the samples in the i th block (randomized with a sign R i ), where
[Equation]
is the gain (average of the signs) of the i th block, and
[Equation]
is the average noise of the i th block.
Due to the sign randomization, it holds that { S i } i ∈
[ ℓ ] is an i.i.d.
sequence.
Since { Z i } i ∈ [ ℓ ] is also an i.i.d.
sequence, then so is { X i } i ∈ [ ℓ ] .
For notational simplicity we will omit the block index i of a generic block.
For block length k , we denote by
[Equation]
the second moment of the gain S .
Note that ξ k ∈ [ 1 k , 1] for any δ ∈ [0 , 12 ] , and, in particular, it is always positive.
For a sequence of samples X n 1 = ( X 1 , . . .
, X n ) , we define by ˆΣ n,k ( X n 1 ) the empirical covariance matrix of the averaged samples over blocks { X i } i ∈ [ ℓ ] , that is
[Equation]
whose population average is Σ n,k
( θ ∗ ) , where
[Equation]
We note that θ is the principal component of Σ n,k ( θ ) , that is, λ max (Σ n,k ( θ )) =
ξ k ∥ θ ∥ 2 + 1 k and the corresponding eigenvector is v max (Σ n,k
( θ ))
= θ .
We thus consider the following estimator for θ ∗ , from a sequence X n 1 , and with a block length of k [Equation]
[Equation]
The estimator is thus constructed from two types of averages: First, a coherent average of the samples at each block, to obtain ℓ block-samples X i with gain S
i and noise variance reduced by a factor of k .
Second, an incoherent average of the “square” of the ℓ block-samples X i X ⊤ i , which resolves the remaining sign ambiguity between blocks.
This balance two extreme cases: If δ = 0 , then this reduces the problem to the GLM (with a sign ambiguity) and k = n is an optimal choice.
If δ = 12 , then this reduces the problem to the GMM, in which coherent averaging is non-beneficial and k
=is rate optimal.
Generally, the optimal choice of the block length k is proportional to the mixing time of the Markov chain Θ( 1 δ ) .
This choice assures that the random gain S is ± 1 with a (constant) high probability.
In fact, an elementary, yet crucial, part of the analysis establishes that the random gain ¯ S has constant variance for this choice of block length (see Lemma 9 in Appendix B.1).
On the other hand, if k = Ω( 1 δ ) , then the random gain S will not be ± 1 (or not even bounded away from zero) with high probability, and such choice is never efficient.
Specifically, we consider the estimator in (14) with k = [-
-]1 8 δ .
The above estimation procedure is depicted in Figure 2 in Appendix B.1.
Let us denote
[Equation]
which will actually be the global minimax rate.
Theorem 1.
Assume that δ ≥ 1 n and d ≤ n , and set ˆ θ ≡ ˆ θ cov ( X n 1 ; k ) with k = 18 δ .
Then, there exist numerical constants c 0 , c 1 , c 2 > 0 such that for every θ ∗ ∈ R d
E (cid:104) loss (ˆ θ, θ ∗ ) (cid:105) ≤ c 0 · (cid:40) β ( n, d, δ ) , ∥
[Equation]
( n, d, δ ) (cid:113)
d n
[Equation]
and
[Equation]
[Equation]
with probability larger than 1 − c 2 n .
Theorem 1 is proved in Appendix B.1.
Evidently, Theorem 1 implies that the upper bound on the minimax rate stated in (7) above holds in low dimension, d ≤ δn , whereas M ( n, d, δ, t ) ≍
M GLM ( n, d, t ) holds in high dimension d ≥ δn .
We remark that the condition δ ≥ 1 n is mild as otherwise the model (2) is essentially equivalent to GLM.
See Remark 8 in Appendix B.1.
Numerical validation of the performance of the estimator ˆ θ cov ( X n 1 ; k ) is shown in Appendix F.
We next consider an impossibility result.
As we have seen, at high dimension, d ≥ δn , the minimax error rates achieved are the same as for the Gaussian location model, and thus clearly cannot be improved.
We thus next focus on the low dimensional regime d ≤ δn .
Theorem 2.
Assume that 2 ≤ d ≤ δn and n ≥ 128 d .
Then the local minimax rate is bounded as
[Equation]
Hence, the minimax rates achieved by the estimator in Theorem 1 are nearly asymptotically optimal, up to a (cid:112) log( n ) factor.
The full proof of Theorem 2 together with a summary of the main ideas used in the proof is presented in Appendix B.2.
Remark 3 (Relaxation of the noise distribution assumption) .
For the sake of clarity of exposition, we have assumed in Theorems 1 and 2 that the noise samples { Z i } ni =1 are i.i.d.
isotropic Gaussians.
There are two straightforward relaxations of this assumption.
First, our minimax upper bound (Theorem 1) can be proved to any subGaussian noise distribution, simply because all the concentration bounds for Gaussian random variables used in the proof admit subGaussian analogues.
The impossibility result (converse, Theorem 2) trivially holds for subGaussian noise since Gaussians are special case of subGaussians.
Second, the isotropic assumption can be relaxed to anisotropic noise with known covariance Σ by simple standardization: If Z i ∼ N (0 d , Σ) are i.i.d.
for some known Σ ≻ 0 , then the estimator will multiply the samples by Σ − 1 / 2 and reduce the problem back to the isotropic setting.
After applying the estimator we propose for the isotropic case, the estimator will obtain its final estimate by multiplying its isotropic estimate by Σ 1 / 2 .
The loss of the estimator will then be gauged by the Mahalanobis distance, parameterized by Σ .
We refer the reader to Appendix G for a discussion on more challenging directions in which the isotropic Gaussian assumption can be relaxed.
Flip probability estimation for a given estimator of θ ∗
In this section, we consider the problem of estimating δ whenever θ ∗ is approximately known to be θ ♯ .
We propose a simple estimator, and then discuss the importance of the accuracy of θ ∗ .
We then derive an impossibility result for the matched case, θ
♯
= θ ∗ .
First note that an estimator for δ can be easily obtained from an estimator for ρ , with essentially the same error rate, via ˆ δ = 12 (1 − ˆ ρ ) .
Thus we focus on estimating ρ .
Assume for simplicity that n is even.
Observing that E [ X ⊤ 2 i
X
i +1 ]
= ρ ∥ θ ∗ ∥ 2 , we propose the following natural estimator for ρ , which replaces the population average with empirical average:
[Equation]
That is, the estimator is based on evaluating the correlation of each of two adjacent samples
X 2 i and
X 2 i − 1 .
We first state a general bound on the estimation error of this estimator.
We then consider the case in which θ ∗ is known, and show how the estimation error is improved in this case
Theorem 4.
Assume that d ≤ n .
Let θ ∗ ∈ R d and let θ ♯ be an estimate of θ ∗ .
Set ˆ ρ ≡ ˆ ρ corr ( X n 1 ; θ ♯ ) and ˆ δ = 12 (1 − ˆ ρ ) .
Then, it holds with probability 1 − 8 n that
[Equation]
The proof of Theorem 4 appears in Appendix C.1.
Note that Theorem 4 states a high-probability bound, suitable to its usage later on in Section 4.
A bound on the expectation of the error can be obtained by the standard method of integrating tails.
The effect of knowledge of θ ∗
If θ ∗ is known up to a sign, i.e., θ ♯ = ± θ ∗ , then for the purpose of ρ (or equivalently δ ) estimation, the model (2) can be reduced to a one-dimensional model by rotational invariance of isotropic Gaussian (See additional details in Appendix C.2).
It then immediately follows from Theorem 4 that:
Corollary 5.
Assume that d ≤ n , ∥ θ ∗ ∥≤ 1 and θ ♯ = ± θ ∗ .
Let U n 1 be defined in as U i :=
∥ θ ∗ ∥· S i + W i where W i ∼ N (0 , 1) i.i.d., ˆ ρ ≡ ˆ ρ corr ( U n 1 ; θ ♯ ) and ˆ δ = 12 (1 − ˆ ρ ) .
Then it holds with probability 1 − 8 n that
[Equation]
[Equation]
Numerical validation of the performance of the estimator ˆ δ corr ( X n 1 ; θ ♯ )
= 12 (1 − ˆ ρ corr ( X n 1 ; θ ♯ )) in the mismatched (Theorem 4) and matched (Corollary 5) cases is provided in Appendix F.
We next consider an impossibility lower bound.
Proposition 6.
Suppose that θ ♯ = θ ∗ and ∥ θ ∗ ∥≤ 1 √ 2 .
Then
[Equation]
where the infimum is over any estimator ˆ δ ( U n 1 ) based on the model U i
=
∥ θ ∗ ∥ S i + W i where each W i is i.i.d.
N (0 , 1) .
The proof of Proposition 6 is presented in Appendix C.2.
According to Corollary 5 and Proposition 6, in estimating δ with a known θ ∗ , though the dependence Θ( 1 √ n ) of the minimax error rate on the sample size is shown to be nearly optimal, it is unclear what the optimal dependence on the signal strength should be.
This is left as an interesting open question and we discuss the challenges associated with this problem in Appendix C.2.
Mean estimation under an unknown flip probability
As we have seen, if an estimator for θ ∗ knows the value of δ , and if both δ ≤ 12 and d ≤ δn hold, then the estimator can achieve improved error rates over the GMM case ( δ = 12 ).
In this section, we assume that both θ ∗ and δ are unknown, and so the estimator is required to estimate δ in order to use this knowledge for an estimator of θ ∗ .
We propose an estimation procedure of three steps based on sample splitting of 3 n samples.
We mention at the outset that the regime in which improvement is possible will be for low signal strength ∥ θ ∗ ∥ ≲ 1 (low separation between the components), and up to a dimension which depends on δ .
Of course the estimation procedure does not know ( θ ∗ , δ ) in advance, and so it is required to identify if ( θ ∗ , δ ) are in this regime during its operation.
We now begin with an overview of the steps of the estimation algorithm.
At Step A, the algorithm estimates θ ∗ based on X n 1 assuming a Gaussian mixture model ( δ = 12 ) to obtain an estimate ˆ θ ( A ) .
Then, based on ∥ ˆ θ ( A ) ∥ , the algorithm decides whether improvement is potentially possible had δ was known.
There are two cases.
The first case is that ∥ ˆ θ ( A ) ∥ is too low, and then its estimate is not sufficiently accurate to be used in the next steps.
Essentially, this happens when the norm is below the global minimax rate ( dn ) 1 / 4 , and the estimation error of the norm on the same scale as the norm of ∥ θ ∗ ∥ .
A trivial estimator of ˆ θ = 0 is then optimal in terms of error rates.
It can be already noted at this step that while the global minimax rate for the known δ case is ( δdn ) 1 / 4 , here the algorithm already stops and estimates ˆ θ = 0 even if just ∥ θ ∗ ∥ ≲ ( dn ) 1 / 4 , leading to larger global minimax rate.
The second case is that ∥ θ ∗ ∥ is larger than a constant.
In this case, the estimation based on a GMM already achieves the optimal parametric O ( (cid:113) dn ) error rate of the Gaussian location model, and so no further estimation steps are necessary.
Otherwise, an improvement in the estimation is possible.
The algorithm proceeds to Step B, and uses X 2 nn +1 to obtain an estimate ˆ δ ( B ) of δ based on the mismatched θ ♯ ≡ ˆ θ ( A ) .
Then, based on the estimate ˆ δ ( B ) the algorithm decides whether the accuracy of ˆ δ ( B ) is sufficient to be used in an refined estimation of θ ∗ .
If the accuracy of ˆ δ ( B ) is not good enough, then the algorithm outputs the estimate from Step A, that is ˆ θ ( A ) .
Otherwise, it proceeds to Step C, in which θ ∗ is re-estimated using X 3 n 2 n +1 , based on a mismatched choice of k , that is k
≍ 1ˆ δ ( B ) instead of k = 18 δ .
Intuitively, the estimated value ˆ δ ( B ) should be larger than δ so the resulting block size k ≍ 1ˆ δ ( B ) will be such that the gain in the block is still close to 1 with high probability.
On the other hand, it is desired that ˆ δ ( B ) will be on the same scale as δ so that the estimation rate (304) (see also (7)) – which now essentially holds with ˆ δ ( B ) instead of δ – would be as small as possible.
Thus, if the algorithm has assured in Step B that ˆ δ ( B ) ≍ δ , then at Step C it will achieve the error rate indicated in (304).
mixture model:
[Equation]
3: if ∥ ˆ θ ( A ) ∥≤ 2 λ θ · log( n ) · ( dn ) 1 / 4 then 4: return ˆ θ guaranteed 5: else if then 6: return ˆ θ = θ A ) 7: end if 8: step B: Estimate δ assuming a mismatched mean value ˆ θ ( A ) :
[Equation]
9: if ˆ δ ( B ) ≤
[Equation]
λ δ λ θ
[Equation] [Equation] ∥ ˆ [Equation]
[Equation] dn then 10: return ˆ θ = ˆ θ ( A ) ▷
No further improvement can be guaranteed 11: end if 12: step C Estimate ∗ assuming a mismatched probability ˆ δ B )
[Equation]
13: return ˆ θ = ˆ θ ( C ) .
The formal description of the estimation algorithm is provided in Algorithm 1.
We remark that refining the estimation of δ can be easily incorporated as a fourth step of this algorithm, but we do not present this in order to keep the statement of the result simple.
The error of the estimator output by Algorithm 1 is as follows:
Theorem 7.
There exist constants c 1 , c 2 ≥ 0 and λ θ , λ δ ≥ 1 such that if d ≤ n 4 λ
[Equation]
θ log 2 ( n ) ∧ 16 then the output ˆ θ of Algorithm 1 satisfies for any θ ∗ ∈ R d , with probability 1 − O ( 1 n ) :
If d ≤
[Equation]
then
[Equation]
If d ≥ [-
-]1 64 λ
[Equation]
δ
[Equation]
n then loss (ˆ θ, θ
[Equation]
) ≤ c
[Equation]
log n
[Equation]
Theorem 7 implies that up to logarithmic factors, the error rates of the known δ case are recovered for low enough dimension d ≲ δ 4 n and ∥ θ ∗ ∥ ≳
(cid:0) dδ 2 n (cid:1) 1 / 4 .
The analysis of Algorithm 1 appears in
Appendix D. Numerical validation of the performance of Algorithm 1 can be found in Appendix F.
The impact of unknown δ on the estimation error of θ ∗
Comparing Theorems 2 (known δ ) and(unknown δ ) reveals the deterioration in the estimation of θ ∗ due to the lack of knowledge of δ from the three aspects mentioned in Section 1.3 (ignoring logarithmic factors):
First, the global minimax rate is O ( dn ) 1 / 4 as for the GMM, instead of the rate O ( δdn ) 1 / 4 for the Markov model case with known δ .
Second, at the regime ( dn ) 1 / 4 ≲ ∥ θ ∗ ∥ ≲ ( dδ 2 n ) 1 / 4 the error rate is O ( 1 ∥ θ ∗ ∥ (cid:113) dn ) instead of the lower
O ( 1 ∥ θ ∗ ∥ (cid:113) δdn ) .
Third, the algorithm is only effective when the dimension is as low as d ≲ δ 4 n .
For higher dimensions, the rates of the GMM are achieved, which can be achieved even without the knowledge of δ .
Conclusion and future work
In this paper, we have considered an elementary, yet fundamental, high-dimensional model with memory.
We have obtained a sharp bound on the minimax rate of estimation in case the underlying statistical dependency (flip probability) is known, and proposed a three-step estimation algorithm when it is unknown.
This has revealed the gains possible in estimation rates due to the memory between the samples, and smoothly interpolated between the extreme cases of GLM and GMM.
An interesting open problem is to either characterize the optimality of the algorithm or improving in the unknown δ case, which requires understanding optimal estimation of the flip probability.
Naturally, as the model considered in this paper is basic, there is an ample of possibilities to generalize this model.
These include, a larger number of components in the mixture, statistical dependency with a more complicated graphical structure between the data samples, existence of nuisance parameters such as the noise variance, sharp finite-sample/finite-iteration analysis of specific practical algorithms such as Baum-Welch, location-scale model with anisotropic noise, heavy-tailed noise, and so on.
Acknowledgments and Disclosure of Funding
Part of this work was done when YZ was a postdoc at Technion where he received funding from the European Union’s Horizon 2020 research and innovation programme under grant agreement No 682203-ERC-[Inf-Speed-Tradeoff].
The work of of NW was supported in part by the Israel Science
Foundation (ISF) under Grant 1782/22.
NW is grateful to Guy Bresler for introducing him to this problem, for the initial ideas that led to this research, and for many helpful discussions on the topic.
Checklist
For all authors... (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?
[Yes] (b) Did you describe the limitations of your work?
[Yes] (c) Did you discuss any potential negative societal impacts of your work?
[N/A] (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes] 2.
If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
[Yes] (b) Did you include complete proofs of all theoretical results?
[Yes] 3.
If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[N/A] (b) Did you specify all the training details (e g., data splits, hyperparameters, how they were chosen)?
[N/A] (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[N/A] (d) Did you include the total amount of compute and the type of resources used (e
g., type of GPUs, internal cluster, or cloud provider)?
[N/A] 4.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
[N/A] (b) Did you mention the license of the assets?
[N/A] (c) Did you include any new assets either in the supplemental material or as a URL?
[N/A] (d) Did you discuss whether
and how consent was obtained from people whose data you’re using/curating?
[N/A] (e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content?
[N/A] 5.
If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?
[N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
[N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
[N/A]