Mean Estimation in High-Dimensional Binary Markov Gaussian Mixture Models
Anonymous Author(s)
Af ﬁ liation
Address email
Abstract
We consider a high-dimensional mean estimation problem over a binary hidden
Markov model, which illuminates the interplay between memory in data, samplesize, dimension, and signal strength in statistical inference.
In this model, anestimator observes n samples of a d -dimensional parameter vector θ ∗ ∈ R d ,multiplied by a random sign S i ( 1 ≤ i ≤ n ), and corrupted by isotropic standard
Gaussian noise.
The sequence of signs { S i } i ∈ [ n ] ∈ { − 1 , 1 } n is drawn from astationary homogeneous Markov chain with ﬂ ip probability δ ∈ [0 , 1 / 2] .
As δvaries, this model smoothly interpolates two well-studied models: the Gaussian
Location Model for which δ = 0 and the Gaussian Mixture Model for which δ =1 / 2 .
Assuming that the estimator knows δ , we establish a nearly minimax optimal(up to logarithmic factors) estimation error rate, as a function of � θ ∗ � , δ , d, n .
Wethen provide an upper bound to the case of estimating δ , assuming a (possiblyinaccurate) knowledge of θ ∗ .
The bound is proved to be tight when θ ∗ is anaccurately known constant.
These results are then combined to an algorithm whichestimates θ ∗ with δ unknown a priori, and theoretical guarantees on its error arestated.
Introduction
To what extent does memory between samples from a high dimensional model affect the possibleestimation error?
In general, this estimation error depends in an intricate way on the interplaybetween the number of samples, the dimension of the vector parameters to be estimated, the noiselevel (signal-to-noise ratio) and the amount of memory between the samples.
In this paper, we studythis question in the context of an elementary hidden Markov high-dimensional Gaussian model, withthe goal of establishing a tight characterization of the estimation error in terms of this trade-off.
Wenext turn to formally de ﬁ ne this model and the estimation problem, describe known results, and thenpresent our contributions.
1.1 Problem formulation
Let S n 0 :=
( S 0 , S 1 , . . .
, S n ) be the following homogeneous binary symmetric Markov chain, S i ∈{ − 1 , 1 } , P [ S 0 = 1] = 1 / 2 and
[Equation]
for i ∈ [ n ] := { 1 , . . . , n } , and where δ ∈ [0 , 1] is the ﬂ ip probability of the binary Markov chain.
Wealso denote ρ := 1 − 2 δ ∈ [ − 1 , 1] which is the correlation between adjacent samples ρ = E [ S i S
i +1 ] .
At each time point i ∈ [ n ] , a sample of a d -dimensional Gaussian mixture model is observed
[Equation]
where Z i ∼ N (0 , I d ) is an i.i.d.
sequence, independent of S n 0 , and where θ ∗ ∈ R d , d ≥ 1 .
At its 32 two extremes, this model degenerates to one of two fundamental models.
When δ = 0 , the memorylength is in ﬁ nite, and the sign S
= S 1 · · · = · · · = S n is ﬁ xed.
Thus, up to this sign ambiguity, the 34 model (2) is the standard Gaussian location model (GLM), which is essentially a memoryless model(and exactly so if S 0 is known).
When δ = 12 , the signs S n 0 are i.i.d.
and have no memory at all.
Themodel (2) is then a Gaussian mixture model (GMM) with two symmetric components, which is alsoa memoryless model.
In all other cases, 0 < δ < 12 (or 12 < δ < 1 ), the model is a simple version ofa hidden Markov model (HMM).
The inference problem we consider in this paper is the estimation of θ ∗ ∈ R d , under the loss function
[Equation]
that is, the Euclidean distance error under a possible sign ambiguity.
1 An intermediate goal (or anadditional problem) is to estimate δ , under the regular absolute error loss function | ˆ δ − δ | .
The fundamental limits of this estimation problem will be gauged by the local minimax rate, which isthe maximal decrease rate of the loss possible for any estimator, given n samples, at dimension d ≥ 2 ,for signal strength � θ ∗ � = t , and under ﬂ ip probability δ .
Speci ﬁ cally, for d ≥ 2 it is de ﬁ ned as
[Equation]
For general d ≥ 1 , the global minimax rate is de ﬁ ned with the condition � θ ∗ � = t replaced by� θ ∗ �≤ t (this condition trivializes the estimator for d = 1 ).
1.2 Known results for GLM and GMM
For the high dimensional Gaussian models we consider in this paper, there are two possible phasetransitions – one as t increases, and the other one as d increases.
Typically, the regime of maintheoretical interest is the low-separation regime ( t � 1 ), in which separating with high accuracybetween the components is impossible (in general), yet parameter estimation with vanishing loss isstill possible.
At the ﬁ rst extreme, the local minimax rate for the GLM ( δ = 0 ) is the usual parametric error rate
[Equation]
[Equation]
This is achieved by the trivial estimator ˆ θ = 0 if t ≤ � dn and the simple empirical average estimatorˆ θ =
[Equation]
n � ni = 1 X i
if t ≥ � dn .
The rate Θ ( � dn ) is then the global minimax rate, i.e., the largest errorover t = � θ ∗ � > 0 .
This model does not have a phase transition with dimension.
At the other extreme, 57 the GMM model ( δ = 12 ) undergoes a phase transition at the dimension d = n .
At low dimension,d ≤ n , the minimax rate was neatly shown in [Wu and Zhou, 2019, Appendix B] to be
[Equation]
[Equation] whereas at high dimension d ≥ n , it is as for the Gaussian location model in (5), i.e., M GMM ( n, d, t ) =
M GLM ( n, d, t ) .
However, we note that at this high dimensional regime, the loss is also lower boundedby a constant even if the signal strength t is unbounded.
The loss in (6) is achieved by the trivialestimator ˆ θ = {+
 if+} t ≤ � dn � 1 / 4 and by an estimator given by a properly scaled and shifted principalcomponent of the empirical covariance matrix of X n 1 :=
( X 1 , X 2 , . . .
, X n ) ,
if t ≥ � dn � 1 / 4 .
For the GMM at low dimension, d ≤ n , the global minimax rate is ( dn ) 1 / 4 , which is worse than the minimaxrate of the Gaussian location model � dn .
Therefore, the GMM has worse estimation performance compared to the GLM from three aspects:
First, at low dimension, d ≤ n , it has a larger global minimax rate ( dn ) 1 / 4 compared to the parametricerror rate of the GLM, � dn ; Second, at low dimension, d ≤ n , parametric error rate is achieved onlyfor constant separation t ≥ 1 ; Third, the transition to the high dimension regime occurs at d = n .
As is intuitively appealing from a “data-processing” reasoning, a Markov model with ﬂ ip probabilityδ � should allow for lower estimation error of θ ∗ compared to a Markov model with δ > δ � .
Indeed,and as a speci ﬁ c simple example, any Markov model with δ < 12 can be easily transformed to a GMMmodel by randomizing the signs of each of the samples by an independent Rademacher variable.
Thuswe may easily deduce, for instance, that since at high dimension ( d ≥ n ) the GLM and the GMMhave the same minimax rates, the minimax rates for d ≥ n are in fact as in (5) for any δ ∈ [0 , 1] .
We thus exclusively focus in the rest of the paper on the regime d ≤ n .
As we will show, that is ageneral phenomenon, and the reduction in estimation error when δ is reduced is less profound as thedimension increases.
1.3 Contributions
We ﬁ rst consider the case in which δ is known to the estimator of θ ∗ .
For this case, we show (Theorem1) that an estimator of θ ∗ based on a computation of the principal component of a properly chosenempirical covariance matrix smoothly interpolates the rates of the GLM in (5) and the GMM in (6).
At low dimension, d ≤ δ n , it achieves local minimax rate of
[Equation]
(7) at high dimension d ≥ δ n , it is as for the Gaussian location model in (5).
The rate of this estimator isthen further shown to be asymptotically optimal (up to a logarithmic factor) via a minimax lowerbound (Theorem 2).
Evidently, its performance smoothly interpolates between the performance of the GLM and the GMM, and the loss is improved with the decrease of δ from all three aspects previouslymentioned.
First, at low dimension, d ≤ δ n , the global minimax rate is � δ dn � 1 / 4 .
Thus whenever δ isas low as δ = dn it becomes the parametric error rate Θ ( � dn ) of the GLM; whenever δ = Θ (1) it isthe same as Θ ( � dn � 1 / 4 ) for the GMM; and it smoothly interpolates between these rates for dn ≤ δ ≤ 1 .
Second, at low dimension, d ≤ δ n , parametric error rate is achieved for signal strength as low ast =
Θ ( √ δ ) (which again, matches the extremes t =
Θ ( � dn ) of GLM and t =
Θ (1) of GMM).
Third, the transition to the high dimension occurs at d = δ n , which is again lower than the transitionpoint of the GMM given by d = n .
This lower transition point allows to achieve the error rate of the GLM for any signal strength, even in a regime in which the loss vanishes with n → ∞ (unlike for
GMM, δ = 12 ); speci ﬁ cally, this occurs whenever δ n ≤ d ≤ n .
Beyond the formal proof, Appendix A provides a heuristic justi ﬁ cation for why the minimax error is naturally expected to scale as in (7).
Second, as a step towards the removal of the assumption that δ is known to the estimator, we considerthe complementary problem of estimating δ whenever an estimate θ � of θ ∗ is available (which can beeither exact θ � = θ ∗ , or inaccurate θ � � = θ ∗ ).
We propose a simple estimator for δ , and analyze its 101 error in case of a mismatch (Theorem 3).
We then specify this result to the matched case θ � = θ ∗and show that in the non-trivial regime ( � θ ∗ � � 1 ) its error rate is ˜ O ( 1 � θ ∗ �� 1 n ) .
We then proceedto show an impossibility lower bound of Ω ( � 1 n ) (Proposition 5) for this error rate.
We discuss thechallenges in settling the optimal estimation error rate of δ as a function of � θ ∗ � .
Third, we consider the case in which the estimator of θ ∗ has no prior knowledge of δ .
We proposea three step algorithm for this case (Algorithm 1).
First, a (possibly) gross estimate ˆ θ ( A ) of θ ∗ iscomputed based on third of the samples, assuming the worst case of δ = 12 .
Then, an estimateˆ δ ( B ) of δ is computed using another third of the samples, assuming the estimate ˆ θ ( A ) .
Finally, are ﬁ ned estimate ˆ θ ( C ) of θ ∗ is obtained by (essentially) assuming that δ is ˆ δ ( B ) .
At each of the stepsabove, the algorithm may stop and decide to return its current estimate when it determines that no further improvement is possible by moving on to the next steps.
We analyze the estimation loss ofthis algorithm
(Theorem 6), and show that this algorithm is capable of partially achieving the gainsassociated with the case of known δ .
1.4 Related work
Both the GLM [Johnstone, 2002, Tsybakov, 2008] and GMM [Lindsay, 1995, McLachlan et al.
,2019] are classic models which were well-explored from numerous perspectives.
For GMM, it is wellknown that the maximum likelihood estimator (MLE) is consistent when the mixture components aresuf ﬁ ciently separated [Redner and Walker, 1984].
Optimal error rates for GMM were derived, e.g.,in
[Heinrich and Kahn, 2015], using a minimum distance estimator (between an atomic distributionconvoluted with the Gaussian density and the empirical distribution of the samples).
Nonetheless, the
GMM is a latent variable model, for which maximizing the likelihood w.r.t.
the center parametersis a non-convex optimization problem. In the last few years, there is a surge of interest in the nonasymptotic performance analysis of computationally ef ﬁ cient estimation algorithms for this estimationtask.
For example, Moitra and Valiant [2010], Kalai et al.
[2010], Anandkumar et al.
[2014], Hardtand Price [2015], Wu and Yang [2020] have analyzed method-of-moments-based algorithms, andvarious other papers considered the expectation-maximization (EM) algorithm
[Balakrishnan et al. ,2017, Xu et al., 2016, Jin et al., 2016, Klusowski and Brinda, 2016, Weinberger and Bresler, 2021, Dwivedi et al., 2020b,a, 2018, Zhao et al., 2018, Yan et al., 2017].
Speci ﬁ cally, the local minimaxrate for GMM in (6) was determined in [Wu and Zhou, 2019] as a benchmark for the operation of the EM algorithm.
The model (2) is a simple instance of a HMM [Ephraim and Merhav, 2002, van Handel, 2008] inhigh dimension.
Parameter estimation in such models is practically performed via the Baum-Welchalgorithm [Baum et al., 1970], which is a computationally ef ﬁ cient version of EM for HMMs.
Theconsistency and asymptotic normality of the MLE for this case were established in [Bickel et al. To the best of our knowledge, there were little attempts to characterize the minimax rates insuch models.
In [Aiylam, 2018], a local version of the Baum-Welch algorithm was proposed, andvanishing error of the convergence of the estimate to the true parameter was established for botha population version as well as a ﬁ nite-sample version. In [Yang et al., 2015], the analysis madefor the operation of EM on samples from memoryless latent variable models in [Balakrishnan et al.
{+,2017] was substantially extended to HMM. The results were then speci ﬁ ed to the Gaussian modelwith Markov signs (2) considered here, and it was shown that the Baum-Welch algorithm achievesparametric error rate, and converges in a ﬁ nite number of iterations [Yang et al., 2015, Corollary2]. However, the qualifying condition for this result is that t = � θ ∗ � � log 
 [Equation]+}
δ ) 2 , that is, anon-trivial separation when δ is constant, which further blows up as δ ↓ 0 .
By contrast, in this paper,our goal is to characterize the estimation error in the regime of δ and t = � θ ∗ � in which the minimax 146 rate is affected by these parameters, and this requires analyzing vanishing δ and t .
More broadly, there is a growing interest in advancing the quantitative understanding of theperformance of statistical learning and inference with dependent data.
Bresler et al.
[2020] studiedlinear regression with Markovian covariates and characterized the minimax error rate in terms ofthe mixing time of the Markov chain.
A stochastic gradient descent-style algorithm adapted tothe Markov setting was shown to be minimax optimal.
Statistical estimation problems includinglinear and logistic regression with more general network dependencies among response variableswere studied by Daskalakis et al.
[2019] and Kandiros et al.
Learnability and generalizationbounds were derived by Dagan et al.
[2019] for dependent data satisfying the so-called Dobrushin’scondition.
Finally, we also mention in passing the huge body of work studying reconstruction ofdependence structures and estimation of dependence parameters from graphical samples, which wedo not exhaustively list here.
1.5 Notation conventions
For a vector v ∈ R d , � v � is the Euclidean norm.
For a positivede ﬁ nite matrix A , let λ max ( A ) and 160 v max ( A ) be the maximal eigenvalue and the associated eigenvector (of unit norm) of A .
Unlessotherwise stated, the constants involved in Bachman-Landau notation are numerical constants. Speci ﬁ cally, they do not depend on the problem parameters ( n, d, δ , t ) .
We write a � b (resp.
a � b )if there exists a numerical constant c > 0 (resp.
C > 0 ) such that a ≥ cb (resp.
a ≤ Cb ).
If a � b and 164 a � b .
then a � b .
By assuming that n is suf ﬁ ciently large, we omit for brevity integer constraints (ceiling and ﬂ oor) on large quantities.
We use the shorthand notation a ∨ b := max { a, b } , a ∧ b := 166 min { a, b } . For a real number a , ( a ) + := a ∨ 0 .
A sequence of objects X 1 , · · · , X n is denoted by 167 X n 1 . Expectation, variance and probability are denoted by E , V and P , respectively.
If two randomvariables X and Y share the same distribution, then we write X d = Y .
All logarithms log are to thebase e .
Mean estimation for a known ﬂ ip probability
In this section, we consider the problem of estimating θ ∗ whenever δ is exactly known to the estimator.
In that case, it may be assumed w.l.o.g. that δ ∈ [0 , 12 ] , as otherwise one may negate each of the evensamples to obtain an equivalent model with δ replaced with 1 − δ .
Hence also ρ ∈ [0 , 1] .
We nextdescribe an estimator for this task, state a bound on its performance, and then show that it matches(up to a logarithmic factor) an impossibility result.
The estimator operationally interpolates and therefore simultaneously generalizes the empiricalaverage estimator (30) and the (properly scaled) principal component estimator (32) analyzed in [Wuand Zhou, 2019, Appendix B].
It degenerates to the latter estimators if δ ↓ 0 or δ ↑ 12 .
Speci ﬁ cally,the estimator partitions the sample into blocks of equal length k
each (which will later be set tok = 18 δ , according to the mixing time of the Markov chain S n 0 ).
Let � denote the number of blocksrespectively, so that k � = n .
Let I i
= { ( i − 1) k
+ 1 , ( i − 1) k
+ 2 , · · · , ik } denote the indices of the 182 i th block.
Further, let { R i } i ∈ [ � ] be an i.i.d.
Rademacher sequence ( R i ∼ Uniform { − 1 , 1 } ), and let
X i := [Equation]
denote the average of the samples in the i th block (randomized with a sign R i ), where
[Equation]
is the gain (average of the signs) of the i th block, and
[Equation]
is the average noise of the i th block.
Due to the sign randomization, it holds that { S i } i ∈ [ � ] is an i.i.d.sequence.
Since { Z i } i ∈ [ � ] is also an i.i.d. sequence, then so is { X i } i ∈ [ � ] .
For notational simplicity 187 we will omit the block index i of a generic block.
For block length k , we denote by
[Equation]
the second moment of the gain S .
Note that ξ k ∈ [ 1 k , 1] for any δ ∈ [0 , 12 ] and is in particular alwayspositive.
For a sequence of samples X n 1 = ( X 1 , . . .
, X n ) , we de ﬁ ne by ˆ Σ n,k
( X n 1 ) the empiricalcovariance matrix of the averaged samples over blocks { X i } i ∈ [ � ] , that is
[Equation]
[Equation]
whose population average is Σ n,k
( θ ∗ ) , where
[Equation]
We note that θ is the principal component of Σ n,k ( θ ) , that is, λ max ( Σ n,k ( θ )) =
ξ k � θ � 2 + 1 k and thecorresponding eigenvector is v max ( Σ n,k ( θ ))
= θ .
We thus consider the following estimator for θ ∗ ,from a sequence X n 1 , and with a block length of k
[Equation]
The estimator is thus constructed from two types of averages: First, a coherent average of the samplesat each block, to obtain � block-samples X i with gain S
i and noise variance reduced by a factor ofk .
Second, an incoherent average of the “square” of the � block-samples X i X � i , which resolves theremaining sign ambiguity between blocks.
This can be seen as a balance between two extreme cases: If δ = 0 , then this reduces the problem to the Gaussian location model (with a sign ambiguity) andk = n is an optimal choice.
If δ = 12 , then this reduces the problem to the two-component Gaussianmixture model, in which coherent averaging is non-bene ﬁ cial and k = 1 leads to optimal error rates.
It is thus obvious that the optimal choice of k depends on δ , the ﬂ ip probability of the Markov chain.
Choosing k = Θ ( 1 δ ) , that is, a blcok length proportional to the mixing time of the Markov chain,assures that the random gain S is ± 1 with a (constant) high probability.
In fact, an elemantary, yetcrucuial, part of the analysis establishes that the random gain ¯ S has constant variance for this choiceof block length (see Lemma 8 in Appendix B.1).
On the other hand, if k = Ω ( 1 δ ) , then the randomgain S will not be ± 1 (or not even bounded away from zero) with high probability, and such choiceis never ef ﬁ cient.
Speci ﬁ cally, we consider the estimator in (14) with k =
δ
The above estimationprocedure is depicted in Figure 2 in Appendix B.1.
Let us denote [Equation]
which will actually be the global minimax rate.
Theorem 1.
Assume that δ ≥ 1 n and d ≤ n , and set ˆ θ ≡ ˆ θ cov ( X n 1 ; k ) with k = 18 δ .
Then, there existnumerical constants c 0 , c 1 , c 2 > 0 such that for every θ ∗ ∈ R d
[Equation]
andloss (ˆ θ , θ ∗ ) ≤ [Equation]
with probability larger than 1 − c 2 n .
Theorem 1 will be proved in Appendix B.1.
A simple consequence of Theorem 1 is that theupper bound on the minimax rate stated in (7) above holds in low dimension, d ≤ δ n , whereas
M ( n, d, δ , t ) � M GLM ( n, d, t ) holds in high dimension d ≥ δ n .
We also remark that δ ≥ 1 n is nota restrictive condition since otherwise the ﬂ ip probability is so low that the model (2) is essentiallyequivalent to GLM.
See Remark 7 in Appendix B.1.
Numerical validation of the performance of theestimator ˆ θ cov ( X n 1 ; k ) is shown in Appendix F.
We next consider an impossibility result.
As we have seen, at high dimension d ≥ δ n the minimaxerror rates achieved are the same as for the Gaussian location model, and thus clearly cannot beimproved.
We thus next focus on the low dimensional regime d ≤ δ n .
Theorem 2.
Assume that 2 ≤ d ≤ δ n and n ≥ 128 d .
Then the local minimax rate is bounded as
[Equation]
Hence, the minimax rates achieved by the estimator in Theorem 1 are nearly asymptotically optimal,up to a � log( n ) factor.
The full proof of Theorem 2 together with a summary of the main ideas usedin the proof is presented in Appendix B.2.
Flip probability estimation for a given estimator of θ ∗
In this section, we consider the problem of estimating δ whenever θ ∗ is approximately known to be 231 θ � .
We propose a simple estimator, and then discuss the importance of the accuracy of θ ∗ .
We thenderive an impossibility result for the matched case, θ � = θ ∗ .
An estimator for δ can be easily obtained from an estimator for ρ , via the plug-in ˆ δ = 12 (1 − ˆ ρ ) .
Assuming for simplicity that n is even, a natural estimator for ρ is then
[-X 
 i +1 ]-][Equation]
[Equation]
That is, the estimator is based on evaluating the correlation of each of two adjacent samples
X 2 i and
{+X 2 i − 1 , whose population version is E [ X � 2 i X 
+}
i − 1 ] = ρ � θ ∗ � 2 .
We ﬁ rst state a general bound on the 237 estimation error of this estimator.
We then consider the case in which θ ∗ is known, and show how the 238 estimation error is improved in this case.
Theorem 3.
Assume that d ≤ n .
Let θ ∗ ∈ R d and let θ � be an estimate of θ ∗ .
Set ˆ ρ ≡ ˆ ρ corr ( X n 1 ; θ � )and ˆ δ = 12 (1 − ˆ ρ ) .
Then, it holds with probability 1 − 8 n that
[Equation]
The proof of Theorem 3 appears in Appendix C.1.
Note that Theorem 3 states a high-probabilitybound, suitable to its usage later on in Section 4.
A bound on the expectation of the error can beobtained by standard methods (integrating tails).
The effect of knowledge of θ ∗
If θ ∗ is known up to a sign, i.e., θ � = ± θ ∗ , then for the purpose of ρ 245 (or equivalently δ ) estimation, the model (2) can be reduced to a one-dimensional model by rotationalinvariance of isotropic Gaussian (See additional details in Appendix C.2).
It then immediately followsfrom Theorem 3 that:
Corollary 4.
Assume that d ≤ n , � θ ∗ �≤ 1 and θ � = ± θ ∗ .
Let U n 1 be de ﬁ ned in as U i :=
� θ ∗ � · S i + W i where W i ∼ N (0 , 1) i.i.d. , ˆ ρ ≡ ˆ ρ corr ( U n 1 ; θ � ) and ˆ δ = 12 (1 − ˆ ρ ) .
Then it holds withprobability 1 − 8 n that
[Equation]
Numerical validation of the performance of the estimator ˆ δ corr ( X n 1 ; θ � ) =
[Equation]
1 − ˆ ρ corr ( X n 1 ; θ � )) in themismatched (Theorem 3) and matched (Corollary 4) cases is provided in Appendix F.
We next consider an impossibility lower bound.
Proposition 5.
Suppose that θ � = θ ∗ and � θ ∗ �≤ 1 √ 2 .
Then [Equation] Here the in ﬁ mum is over any estimator ˆ δ ( U n 1 ) based on the model U i =
� θ ∗ � S i + W i where each 256 W i is i.i.d.
N (0 , 1) .
The proof of Proposition 5 is presented in Appendix C.2.
According to Corollary 4 and Proposition5, in estimating δ with a known θ ∗ , though the dependence Θ ( 1 √ n ) of the minimax error rate onthe sample size is shown to be nearly optimal, it is unclear what the optimal dependence on thesignal strength should be.
This is left as an interesting open question and we discuss the challengesassociated with this problem in Appendix C.2.
Mean estimation under an unknown ﬂ ip probability
As we have seen, if an estimator for θ ∗ knows the value of δ , and if both δ ≤ 12 and d ≤ δ n hold,then the estimator can achieve improved error rate over the GMM case ( δ = 12 ).
In this section, we assume that both θ ∗ and δ are unknown, and so the estimator is required to estimate δ in order to usethis knowledge for an estimator of θ ∗ .
We propose an estimation procedure of three steps based onsample splitting of 3 n samples.
We mention at the outset that the regime in which improvement ispossible will be for low signal strength � θ ∗ � � 1 (low separation between the components), and up 269 to a dimension which depends on δ .
Of course the estimation procedure does not know ( θ ∗ , δ ) in 270 advance, and so it is required to identify if ( θ ∗ , δ ) are in this regime during its operation.
We now begin with an overview of the steps of the estimation algorithm.
At Step A, the algorithmestimates θ ∗ based on X n 1 assuming a Gaussian mixture model ( δ = 12 ) to obtain an estimate ˆ θ ( A ) .
Then, based on � ˆ θ ( A ) � , the algorithm decides whether improvement is potentially possible had δwas known.
There are two cases.
The ﬁ rst case is that � ˆ θ ( A ) � is too low, and then its estimate is notsuf ﬁ ciently accurate to be used in the next steps.
Essentially, this happens when the norm is belowthe global minimax rate ( dn ) 1 / 4 , and the estimation error of the norm on the same scale as the normof � θ ∗ � .
A trivial estimator of ˆ θ = 0 is then optimal in terms of error rates.
It can be already noted atthis step that the minimax rate for the known δ case is ( δ dn ) 1 / 4 , whereas here the algorithm stops ansestimates ˆ θ = 0 even if just � θ ∗ � � ( dn ) 1 / 4 , leading to larger global minimax rate.
The second case isthat � θ ∗ � is larger than a constant.
In this case, the estimation based on a GMM already achieves theoptimal parametric O ( � dn ) error rate of the Gaussian location model, and so no further estimationsteps are necessary.
Otherwise, an improvement in the estimation is possible.
The algorithm proceedsto Step B, and uses X 2 nn +1 to obtain an estimate ˆ δ ( B ) of δ based on the mismatched θ � ≡ ˆ θ ( A ) .
Then,based on the estimate ˆ δ ( B ) the algorithm decides whether the accuracy of ˆ δ ( B ) is suf ﬁ cient to be usedin an re ﬁ ned estimation of θ ∗ .
If the accuracy of ˆ δ ( B ) is not good enough, then the algorithm outputsthe estimate from Step A, that is ˆ θ ( A ) .
Otherwise, it proceeds to Step C, in which θ ∗ is re-estimatedusing X 3 n 2 n +1 , based on a mismatched choice of k , that is k � 1ˆ δ ( B ) instead of k = 18 δ .
Intuitively, theestimated value ˆ δ ( B ) should be larger than δ so the resulting block size k � 1ˆ δ ( B ) will be such that thegain in the block is still close to 1 with high probability.
On the other hand, it is desired that ˆ δ ( B )will be on the same scale as δ so that the estimation rate (296) (see also (7)) – which now essentiallyholds with ˆ δ ( B ) instead of δ – would be as small as possible.
Thus, if the algorithm has assured in Step B that ˆ δ ( B ) � δ , then at Step C it will achieve the error rate indicated in (296).
A formal description of the operation of the estimation algorithm is provided in Algorithm 1.
Wenote in passing that re ﬁ ning the estimation of δ can be easily incorporated as a fourth step of thisalgorithm, but we do not present this in order to keep the statement of the result simple.
The error ofthe estimator output by Algorithm 1 is as follows:
Theorem 6.
Assume that d ≤ n 2 λ θ log 2 ( n ) ∧ Then, there exist numerical constants c 1 , c 2 ≥ 0 andλ θ , λ δ ≥ 1 so that the output ˆ θ of the estimation Algorithm 1 satis ﬁ es that for any θ ∗ ∈ R d , with 299 probability− O ( 1 n • If d ≤ 1 64 λ δ λ θ log 2 ( n ) δ 4 n then
loss (ˆ θ , θ ∗ ) ≤ c 1 log n ·  
[Equation]
, �
[Equation]
• If d ≥ 1 64 λ δ λ θ log 2 ( n ) δ 4 n
thenloss
[Equation]
[Equation] [Equation]
[Equation] [Equation]
As can be observed, ignoring logarithmic factors, if the dimension is low enough d � δ 4 n and� θ ∗ � � � d δ 2 n � 1 / 4 then the error rates of the known δ case are recovered.
The analysis of Algorithm 1appears in Appendix D. Numerical validation of the performance of Algorithm 1 can be found in Appendix F.
The impact of lack of knowledge of δ
The deterioration in the estimation of θ ∗ due to the lackof knowledge of δ includes the following (ignoring logarithmic factors).
First, the global minimaxrate is ( dn ) 1 / 4 as for the GMM, instead of the rate ( δ dn ) 1 / 4 for the Markov model case with known δ .
Second, at the regime ( dn ) 1 / 4 � � θ ∗ � � ( d δ 2 n ) 1 / 4 the error rate is O ( 1 � θ ∗ � � dn ) instead of the lower
O ( 1 � θ ∗ � � δ dn ) .
Third, the algorithm is only effective when the dimension is as low as d � δ 4 n .
Forhigher dimensions, the rates of the GMM are achieved, which can be achieved even without theknowledge of δ .
Conclusion and future work
In this paper, we have considered an elementary, yet fundamental, high-dimensional model withmemory.
We have obtained a sharp bound on the minimax rate of estimation in case the underlyingstatistical dependency ( ﬂ ip probability) is known, and proposed a three-step estimation algorithmwhen it is unknown.
This has revealed the gains possible in estimation rates due to the memorybetween the samples, and smoothly interpolated between the extreme cases of GLM and GMM.
Aninteresting open problem is to either characterize the optimality of the algorithm or imporving inthe unknown δ case; this requires understanding optimal estimation of the ﬂ ip probability, and thechallenges associated with this problem are illuminated.
Naturally, as the model considered in this paper is basic, there is an ample of possibilities to generalizethis model.
These include, a larger number of components in the mixture, statistical dependency witha more complicated graphical structure between the data samples, existence of nuisance parameterssuch as the noise variance, sharp ﬁ nite-sample/
ﬁ nite-iteration analysis of speci ﬁ c practical algorithmssuch as Baum-Welch, and so on. https html .
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately re ﬂ ect the paper’scontributions and scope?
[Yes] (b) Did you describe the limitations of your work?
[Yes]
(c) Did you discuss any potential negative societal impacts of your work?
[N/A](d)
Have you read the ethics review guidelines and ensured that your paper conforms tothem?
[Yes]2.
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes](b) Did you include complete proofs of all theoretical results?
[Yes]
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[N/A]
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how theywere chosen)?
[N/A](c)
Did you report error bars (e g., with respect to the random seed after runningexperiments multiple times)?
[N/A](d)
Did you include the total amount of compute and the type of resources used (e
g., typeof GPUs, internal cluster, or cloud provider)?
[N/A]
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[N/A]
(b) Did you mention the license of the assets?
[N/A]
(c) Did you include any new assets either in the supplemental material or as a URL?
[N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[N/A]
(e) Did you discuss whether the data you are using/curating contains personally identi ﬁ able information or offensive content?
[N/A]
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if applicable?
[N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount
spent on participant compensation?
[N/A]