Mean Estimation in High-Dimensional Binary Markov Gaussian Mixture Models
Anonymous Author(s)
Af ﬁ liation
Address email
Abstract
We consider a high-dimensional mean estimation problem over a binary hidden
Markov model, which illuminates the interplay between memory in data, samplesize, dimension, and signal strength in statistical inference.
In this model, anestimator observes n samples of a d -dimensional parameter vector θ ∗ ∈ R d ,multiplied by a random sign S i ( 1 ≤ i ≤ n ), and corrupted by isotropic standard
Gaussian noise.
The sequence of signs { S i } i ∈ [ n ] ∈ { − 1 , 1 } n is drawn from astationary homogeneous Markov chain with ﬂ ip probability δ ∈ [0 , 1 / 2] .
As δvaries, this model smoothly interpolates two well-studied models: the Gaussian
Location Model for which δ = 0 and the Gaussian Mixture Model for which δ =1 / 2 .
Assuming that the estimator knows δ , we establish a nearly minimax optimal(up to logarithmic factors) estimation error rate, as a function of ∥ θ ∗ ∥ , δ , d, n .
Wethen provide an upper bound to the case of estimating δ , assuming a (possiblyinaccurate) knowledge of θ ∗ .
The bound is proved to be tight when θ ∗ is anaccurately known constant.
These results are then combined to an algorithm whichestimates θ ∗ with δ unknown a priori, and theoretical guarantees on its error arestated.
Introduction
Memory between data samples is ubiquitous in practical applications, as data collected from networksor sampled time series inevitably inherit spatial or temporal statistical dependencies.
Numerousexamples arise in imaging, meteorology, health care, ﬁ nance, social science and so on [Glaeser et al.
,1996, Bertrand et al., 2000, Sacerdote, 2001, Du ﬂ o and Saez, 2003, Christakis and Fowler, 2013].
Inprinciple, prior knowledge of the existence of such memory can be used to improve the performanceof statistical estimators compared to memoryless models.
Development and analysis of statisticalinference algorithms for models with memory has been extensively explored from the algorithmicperspective, and computationally ef ﬁ cient algorithms such as Baum-Welch and message-passingwere developed [Ephraim and Merhav, 2002, van Handel, 2008, MacKay, 2003, Wainwright and
Jordan, 2008].
It was also extensively studied from the theoretical perspective in the classical, ﬁ xeddimensional and asymptotic regime, e.g., Györ ﬁ et al. [2002, Chapter 27 and references therein].
However, much less is understood about about the high dimensional, non-asymptotic regime, whichis of paramount importance in modern applications, and the focus of current extensive research
, Wainwright, 2019]. As we exemplify in this paper, the error in such estimationproblems depends in an intricate way on the interplay between the number of samples, the dimensionof the vector parameters to be estimated, the noise level (signal-to-noise ratio), and the level ofmemory between the samples.
An ever popular and fundamental model is the Gaussian mixture model, for which memory existsbetween samples when the latent variables determining the component of each sample are dependent.
[2016], Jin et al. [2016], Dwivedi et al. [2020b], Dwivedi et al. [2018], Dwivedi et al.
[2020a], Zhaoet al. [2018], Yan et al. [2017], Weinberger and Bresler [2022] have focused on high-dimensionalmemoryless models, and analyzed computationally ef ﬁ cient estimation algorithms, most notably theexpectation maximization (EM) algorithm.
Speci ﬁ cally, the seminal Balakrishnan et al. [2017] hasprovided theoretical guarantees for EM in memoryless models, though without proving minimaxoptimality, which was subsequently established in Wu and Zhou [2019].
As a notable exception, Yanget al.
[2015] has generalized the analysis of Balakrishnan et al. [2017] to a hidden Markov model(HMM) that has memory, yet again, without determining how the minimax error rate depends on thenumber of samples, dimension, noise level, and the amount of memory.
In this paper, we address thequestion of precise characterization of the minimax error rate in terms of these parameters, in thecontext of a high-dimensional Gaussian HMM.
Our characterization illuminates the opportunities andchallenges associated with optimal inference in high dimensional models with memory.
We next turnto formally de ﬁ ne this model and the estimation problem, describe known results, and then presentour contributions.
1.1 Problem formulation
Let S n 0 :=
( S 0 , S 1 , . . .
, S n ) be the following homogeneous binary symmetric Markov chain, S i ∈{ − 1 , 1 } , P [ S 0 = 1] = 1 / 2 and
[Equation]
for i ∈ [ n ] := { 1 , . . . , n } , and where δ ∈ [0 , 1] is the ﬂ ip probability of the binary Markov chain.
Wealso denote ρ := 1 − 2 δ ∈ [ − 1 , 1] which is the correlation between adjacent samples ρ = E [ S i S
i +1 ] .
At each time point i ∈ [ n ] , a sample of a d -dimensional Gaussian mixture model is observed
[Equation]
where Z i ∼ N (0 , I d ) is an i.i.d.
sequence, independent of S n 0 , and where θ ∗ ∈ R d , d ≥ 1 .
At its 58 two extremes, this model degenerates to one of two well-studied and fundamental models.
Whenδ = 0 , the memory length is in ﬁ nite, and the sign S
= S 1 · · · = · · · = S n is ﬁ xed.
Thus, up to this 60 sign ambiguity, the model (2) is the standard Gaussian location model (GLM), which is essentially amemoryless model (and exactly so if S 0 is known).
When δ = 12 , the signs S n 0 are i.i.d.
and haveno memory at all.
The model (2) is then a Gaussian mixture model (GMM) with two symmetriccomponents, which is also a memoryless model.
In all other cases, 0 < δ < 12 (or 12 < δ < 1 ), themodel is a simple version of a HMM.
The inference problem we consider in this paper is the estimation of θ ∗ ∈ R d , under the loss function
[Equation]
that is, the Euclidean distance error under a possible sign ambiguity.
1 An intermediate goal (oran additional problem) is to estimate δ , under the regular absolute error loss function | ˆ δ − δ | .
The 68 fundamental limits of this estimation problem will be gauged by the local minimax rate, which is themaximal decrease rate of the loss possible for any estimator, given n samples, at dimension d ≥ 2 ,for signal strength ∥ θ ∗ ∥ = t , and under ﬂ ip probability δ .
Speci ﬁ cally, for d ≥ 2 it is de ﬁ ned as
[Equation]
For general d ≥ 1 , the global minimax rate is de ﬁ ned with the condition ∥
θ ∗ ∥ = t replaced by∥ θ ∗ ∥≤ t (this condition trivializes the estimator for d = 1 ).
1.2 Known minimax estimation errors rates for GLM and GMM
Before delving into models with memory ( 0 < δ < 1 / 2 ), we review known results on the minimaxrates in memoryless high dimensional Gaussian models – the GLM ( δ = 0 , the GMM ( δ = 1 / 2 ).
Werefer the reader to note that the minimax error rate in these models may undergo two possible phasetransitions – one as t increases, and the other one as d increases.
We also remark that the regime of1 Similar bounds can be derived for the squared loss by trivial extensions. interest is that of low-separation ( t ≲ 1 ), in which accurately detecting the components is impossible,yet parameter estimation with vanishing loss is possible.
At the ﬁ rst extreme, the local minimax rate for the GLM ( δ = 0 ) is the usual parametric error rate
[Equation]
This is achieved by the trivial estimator ˆ θ = 0 if t ≤ q dn and the simple empirical average estimatorˆ θ = 1 n
P [Equation]
i if
[Equation]
≥ q dn (see [Wu, 2017, Sec.
III-9]).
The rate Θ ( q dn ) is then the global minimaxrate, i.e., the largest error over t =
∥ θ ∗ ∥ > 0 .
This model does not have a phase transition with 84 dimension.
At the other extreme, the GMM model ( δ = 12 ) undergoes a phase transition at thedimension d = n .
At low dimension, d ≤ n , the minimax rate was neatly shown in [Wu and Zhou,2019, Appendix B] to be
[Equation]
whereas at high dimension d ≥ n , it is as for the Gaussian location model in (5), i.e., M GMM ( n, d, t ) =
M GLM ( n, d, t ) .
However, at this high dimensional regime, the loss is also lower bounded by a constanteven if the signal strength t is unbounded.
The loss in (6) is achieved by the trivial estimator ˆ θ = 0if t ≤ � dn  1 / 4 and by an estimator given by a properly scaled and shifted principal component ofthe empirical covariance matrix of X n 1 :=
( X 1 , X 2 , . . .
, X n ) ,
if t ≥ � dn  1 / 4 .
For the GMM at lowdimension, d ≤ n , the global minimax rate is ( dn ) 1 / 4 , which is worse than the minimax rate of the
Gaussian location model
[Equation]
Therefore, the GMM has worse estimation performance compared to the GLM from three aspects:
First, at low dimension, d ≤ n , it has a larger global minimax rate ( dn ) 1 / 4 compared to the parametricerror rate of the GLM, q dn ; Second, at low dimension, d ≤ n , parametric error rate is achieved onlyfor constant separation t ≥ 1 ; Third, the transition to the high dimension regime occurs at d = n .
As is intuitively appealing from a “data-processing” reasoning, a Markov model with ﬂ ip probabilityδ ′ should allow for lower estimation error of θ ∗ compared to a Markov model with δ > δ ′ .
Indeed,and as a speci ﬁ c simple example, any Markov model with δ < 12 can be easily transformed to a GMMmodel by randomizing the signs of each of the samples by an independent Rademacher variable.
Thuswe may deduce, e.g., that since at high dimension ( d ≥ n ) the GLM and the GMM have the sameminimax rates, the minimax rates for d ≥ n are in fact as in (5) for any δ ∈ [0 , 1] .
We thus henceforthexclusively focus on the regime d ≤ n .
As we show, it is generally true that the improvement inestimation error when δ is reduced is less profound as the dimension increases.
1.3 Contributions
We ﬁ rst consider the case in which δ is known to the estimator of θ ∗ .
For this case, we analyze theloss of an estimator that is based on a computation of the principal component of a properly chosenempirical covariance matrix.
We show (Theorem 1) that, at low dimension, d ≤ δ n , it achieves localminimax rate of
[Equation]
and at high dimension d ≥ δ n , it is as for the Gaussian location model in (5).
The rate of this estimatoris then further shown to be asymptotically optimal (up to a logarithmic factor) via a minimax lowerbound (Theorem 2).
Evidently, the loss in (7) smoothly interpolates the rates of the GLM in (5) and the GMM in (6).
Moreover, it is evident that the loss is improved with the decrease of δ from all threeaspects previously mentioned.
First, at low dimension, d ≤ δ n , the global minimax rate is � δ dn  1 / [-
 (obtained-]4(obtained by equating the ﬁ rst and second cases in (7)).
Hence, reducing the ﬂ ip probability fromδ =
Θ (1) to δ = dn smoothly reduces the global minimax rate from the parametric error rate Θ ( q dn )of the GLM to the Θ ( � dn  1 / 4 ) of the GMM.
Second, at low dimension, d ≤ δ n , the minimal signalstrength required to obtain parametric error rate is t =
Θ ( √ δ ) (obtained from the third case in (7)).
This, again, improves by decreasing δ , and matches the extremes t =
Θ ( q dn ) of GLM and t =
Θ (1)of GMM.
Third, the transition to the high dimension regime occurs at d = δ n , which is again betterwith lower δ , and matches the transition point of the GMM given by d = n .
This lower transitionpoint allows us to achieve the error rate of the GLM for any signal strength, even in a regime in whichthe loss
O ( q dn ) vanishes with n → ∞ (unlike for GMM, δ = 12 ); speci ﬁ cally, this occurs wheneverd ≥ δ n yet d = o ( n ) .
Beyond the formal proof, Appendix A provides a heuristic justi ﬁ cation forwhy the minimax error is naturally expected to scale as in (7).
Second, as a step towards the removal of the assumption that δ is known to the estimator, we considerthe complementary problem of estimating δ whenever an estimate θ ♯ of θ ∗ is available (which can beeither exact θ ♯ = θ ∗ , or inaccurate θ ♯ ̸
= θ ∗ ).
We propose a simple estimator for δ , and analyze its 130 error in case of a mismatch (Theorem 4).
We then specify this result to the matched case θ ♯ = θ ∗ andshow that in the non-trivial regime ( ∥ θ ∗ ∥ ≲ 1 ) its error rate is ˜ O ( 1 ∥ θ ∗ ∥q 1 n ) .
We then proceed to showan impossibility lower bound of Ω ( q 1 n ) (Proposition 6) for this error rate.
The precise dependenceof the estimation error rate of δ on ∥ θ ∗ ∥ is therefore not precisely determined, and we discuss the 134 challenges in settling this matter.
Third, we consider the case in which the estimator of θ ∗ has no prior knowledge of δ .
We proposea three step algorithm for this case (Algorithm 1).
First, a (possibly) gross estimate ˆ θ ( A ) of θ ∗ iscomputed based on third of the samples, assuming the worst case of δ = 12 .
Then, an estimateˆ δ ( B ) of δ is computed using another third of the samples, assuming the estimate ˆ θ ( A ) .
Finally, are ﬁ ned estimate ˆ θ ( C ) of θ ∗ is obtained by (essentially) assuming that δ is ˆ δ ( B ) .
At each of the stepsabove, the algorithm may stop and decide to return its current estimate when it determines that nofurther improvement is possible by moving on to the next steps.
We analyze the estimation loss ofthis algorithm
(Theorem 7), and show that this algorithm is capable of partially achieving the gainsassociated with the case of known δ .
1.4 Additional related work
Both the GLM [Johnstone, 2002, Tsybakov, 2008] and GMM [Lindsay, 1995, McLachlan et al.
,2019] are classic models which were well-explored from numerous perspectives.
In the last fewyears, there is a surge of interest in the non-asymptotic performance analysis of computationallyef ﬁ cient estimation algorithms for this estimation task.
For example, Moitra and Valiant [2010],
Kalai et al.
[2010], Anandkumar et al.
[2014], Hardt and Price [2015], Wu and Yang [2020] haveanalyzed method-of-moments-based algorithms, and various other papers considered the expectationmaximization (EM) algorithm
[Balakrishnan et al., 2017, Xu et al., 2016, Jin et al., 2016, Klusowskiand Brinda, 2016, Weinberger and Bresler, 2022, Dwivedi et al., 2020b,a, 2018, Zhao et al., 2018,
Yan et al., 2017].
Speci ﬁ cally, the local minimax rate for GMM in (6) was determined in [Wu and
Zhou, 2019] as a benchmark for the operation of the EM algorithm.
The model (2) is a simple instance of a HMM [Ephraim and Merhav, 2002, van Handel, 2008] inhigh dimension.
Parameter estimation in such models is practically performed via the Baum-Welchalgorithm [Baum et al., 1970], which is a computationally ef ﬁ cient version of EM for HMMs.
To thebest of our knowledge, there were little attempts to characterize the minimax rates in such modelswith the exception of [Yang et al., 2015] previously mentioned.
However, the qualifying conditionfor the estimation error bound of [Yang et al., 2015] on EM is that t =
∥
θ ∗ ∥ ≳ log 1 1 −
[Equation]
) 2 , that 161 is, a non-trivial separation when δ is constant, which further blows up as δ ↓ 0 .
By contrast, in this 162 paper, our goal is to characterize the estimation error in the regime of δ and t =
∥ θ ∗ ∥ in which theminimax rate is affected by these parameters, and this requires analyzing vanishing δ and t .
More broadly, there is a growing interest in advancing the quantitative understanding of theperformance of statistical learning and inference with dependent data.
Bresler et al.
[2020] studiedlinear regression with Markovian covariates and characterized the minimax error rate in terms ofthe mixing time of the Markov chain.
A stochastic gradient descent-style algorithm adapted to the
Markov setting was shown to be minimax optimal.
Statistical estimation problems including linearand logistic regression with more general network dependencies among response variables werestudied by Daskalakis et al.
[2019] and Kandiros et al.
Learnability and generalization boundswere derived by Dagan et al.
[2019] for dependent data satisfying the so-called Dobrushin’s condition.
1.5 Notation conventions
For a vector v ∈ R d , ∥ v ∥ is the Euclidean norm.
For a positivede ﬁ nite matrix A , λ max ( A ) and 174 v max ( A ) are the maximal eigenvalue and the associated eigenvector (of unit norm) of A .
Unlessotherwise stated, the constants involved in Bachmann-Landau notation are numerical, and do notdepend on the parameters ( n, d, δ , t ) .
It holds that a ≳ b (resp.
a ≲ b ) if there exists a constant c > 0(resp.
C > 0 ) such that a ≥ cb (resp.
a ≤ Cb ).
If a ≲ b and a ≳ b .
then a ≍ b .
Integer constraints(ceiling and ﬂ oor) on large quantities that do not affect the results are omitted for brevity.
For a realnumbers a, b the shorthand notation a ∨ b := max { a, b } , a ∧ b :=
min { a, b } and ( a ) + := a ∨ 0 is 180 used.
A sequence of objects X 1 , · · · , X n is denoted by X n 1 . Expectation, variance and probability 181 are denoted by E , V and P , respectively.
Equality in distribution of random variables X
and Y isdenoted by X d = Y .
All logarithms log are to the base e .
Mean estimation for a known ﬂ ip probability
In this section, we consider the problem of estimating θ ∗ whenever δ is exactly known to the estimator.
In that case, it may be assumed w.l.o.g. that δ ∈ [0 , 12 ] , as otherwise one may negate each of the evensamples to obtain an equivalent model with δ replaced with 1 − δ .
Hence also ρ ∈ [0 , 1] .
We nextdescribe an estimator for this task, state a bound on its performance, and then show that it matches(up to a logarithmic factor) an impossibility result.
The estimator operationally interpolates and therefore simultaneously generalizes the empiricalaverage estimator (21) and the (properly scaled) principal component estimator (23) analyzed in [Wuand Zhou, 2019, Appendix B].
It degenerates to the latter estimators if δ ↓ 0 or δ ↑ 12 .
Speci ﬁ cally,the estimator partitions the sample into blocks of equal length k
each (which will later be set tok = 18 δ , according to the mixing time of the Markov chain S n 0 ).
Let ℓ denote the number of blocksrespectively, so that k ℓ = n .
Let I i
= { ( i − 1) k
+ 1 , ( i − 1) k
+ 2 , · · · , ik } denote the indices 195 of the i th block.
Further, let { R i } i ∈ [ ℓ ] be an i.i.d.
Rademacher sequence ( R i ∼ Uniform { − 1 , 1 } ),and let X i := R i · 1 k
P j ∈ I
i X j = S
i θ ∗
+ Z i denote the average of the samples in the i th block(randomized with a sign R i ), where S i :=
R i · 1 k
P j ∈
I
i S
j is the gain (average of the signs) of thei th block, and Z i := R i · 1 k
P j ∈
I
i Z j ∼ N � 0 , 1 k · I d  is the average noise of the i th block.
Due tothe sign randomization, it holds that { S i } i ∈
[ ℓ ] is an i.i.d.
sequence.
Since { Z i } i ∈ [ ℓ ] is also an i.i.d.sequence, then so is { X i } i ∈ [ ℓ ] .
For notational simplicity we will omit the block index i of a genericblock.
For block length k , we denote by ξ k :=
E [ S 2 ]
=
E [( 1 k
P kj =1 S j ) 2 ] the second moment of thegain S .
Note that ξ k ∈ [ 1 k , 1] for any δ ∈ [0 , 12 ] and is in particular always positive.
For a sequenceof samples X n 1 = ( X 1 , . . .
, X n ) , we de ﬁ ne by ˆ Σ n,k
( X n 1 ) the empirical covariance matrix of theaveraged samples over blocks { X i } i ∈ [ ℓ ] , that is ˆ Σ n,k ( X n 1 ) :
= 1 ℓ
P ℓ i =
1 X i X
⊤ i , whose populationaverage is Σ n,k
( θ ∗ ) , where Σ n,k ( θ ∗ ) :=
E [ X X ⊤ ] = ξ k θ ∗ θ ⊤∗ + 1 k I d .
We note that θ is the principalcomponent of Σ n,k ( θ ) , that is, λ max ( Σ n,k ( θ )) =
ξ k ∥ θ ∥ 2 + 1 k and the corresponding eigenvector is 207 v max ( Σ n,k ( θ ))
= θ .
We thus consider the following estimator for θ ∗ , from a sequence X n 1 , and witha block length of k
[Equation]
The estimator is thus constructed from two types of averages: First, a coherent average of the samplesat each block, to obtain ℓ block-samples X i with gain S
i and noise variance reduced by a factor of k .
Second, an incoherent average of the “square” of the ℓ block-samples X i X ⊤ i , which resolvesthe remaining sign ambiguity between blocks.
This balance two extreme cases: If δ = 0 , then thisreduces the problem to the GLM (with a sign ambiguity) and k = n is an optimal choice.
If δ = 12 ,then this reduces the problem to the GMM, in which coherent averaging is non-bene ﬁ cial and k = 1is rate optimal.
Generally, the optimal choice of the block length k is proportional to the mixing timeof the Markov chain Θ ( 1 δ ) .
This choice assures that the random gain S is ± 1 with a (constant) highprobability.
In fact, an elementary, yet crucial, part of the analysis establishes that the random gain ¯ Shas constant variance for this choice of block length (see Lemma 9 in Appendix B.1).
On the otherhand, if k = Ω ( 1 δ ) , then the random gain S will not be ± 1 (or not even bounded away from zero)with high probability, and such choice is never ef ﬁ cient.
Speci ﬁ cally, we consider the estimator in (8)with k =
δ
The above estimation procedure is depicted in Figure 2 in Appendix B.1.
Let us denote β ( n, d, δ ) :
=
q dn ∨ �
δ dn  1 / 4 , which will actually be the global minimax rate.
Theorem 1.
Assume that δ ≥ 1 n and d ≤ n , and set ˆ θ ≡ ˆ θ cov ( X n 1 ; k ) with k = 18 δ .
Then, there existnumerical constants c 0 , c 1 , c 2 > 0 such that for every θ ∗ ∈ R d
E
[Equation]
and loss (ˆ θ , θ ∗ )
[Equation]
i with probability larger than 1 − c 2 n .
Theorem 1 is proved in Appendix B.1.
Evidently, Theorem 1 implies that the upper bound onthe minimax rate stated in (7) above holds in low dimension, d ≤ δ n , whereas M ( n, d, δ , t ) ≍ 228 M
GLM ( n, d, t ) holds in high dimension d ≥ δ n .
2 We remark that the condition δ ≥ 1 n is mild asotherwise the model (2) is essentially equivalent to GLM.
See Remark 8 in Appendix B.1.
Numericalvalidation of the performance of the estimator ˆ θ cov ( X n 1 ; k ) is shown in Appendix F.
We next consider an impossibility result.
As we have seen, at high dimension, d ≥ δ n , the minimaxerror rates achieved are the same as for the Gaussian location model, and thus clearly cannot beimproved.
We thus next focus on the low dimensional regime d ≤ δ n .
Theorem 2.
Assume that 2 ≤ d ≤ δ n and n ≥ 128 d .
Then the local minimax rate is bounded as
[Equation]
Hence, the minimax rates achieved by the estimator in Theorem 1 are nearly asymptotically optimal,up to a p log( n ) factor.
The full proof of Theorem 2 together with a summary of the main ideas usedin the proof is presented in Appendix B.2.
Remark 3 (Relaxation of the noise distribution assumption) .
For the sake of clarity of exposition, wehave assumed in Theorems 1 and 2 that the noise samples { Z i } ni =1 are i.i.d.
isotropic Gaussians.
There 240 are two straightforward relaxations of this assumption.
First, our minimax upper bound (Theorem 1)can be proved to any subGaussian noise distribution, simply because all the concentration boundsfor Gaussian random variables used in the proof admit subGaussian analogues.
The impossibilityresult (converse, Theorem 2) trivially holds for subGaussian noise since Gaussians are special caseof subGaussians.
Second, the isotropic assumption can be relaxed to anisotropic noise with knowncovariance Σ by simple standardization:
If Z i ∼ N (0 d , Σ ) are i.i.d.
for some known Σ ≻ 0 , then theestimator will multiply the samples by Σ − 1 / 2 and reduce the problem back to the isotropic setting.
After applying the estimator we propose for the isotropic case, the estimator will obtain its ﬁ nal2 When ∥ θ ∗ ∥≤ β ( n, d, δ ) , the estimator ˆ θ cov ( X n 1 ; k ) in Theorem 1 only achieves a rate β ( n, d, δ ) which is larger than the promised rate ∥ θ ∗ ∥ in (7).
However, since the estimator is assumed to know t (but not the direction of θ ∗ ; a common formulation in high-dimensional statistics), then it can output the zero vector.
It then incurs loss ∥ θ ∥ ∗ for any θ ∗ ∈ R d , matching the promised rate when ∥ θ ∗ ∥≤ β ( n, d, δ ) .
To summarize, for any value of t , the minimax rate is achieved by the minimum rate of ˆ θ cov ( X n 1 ; k ) and ˆ θ 0 ( X n 1 ) ≡ 0 . estimate by multiplying its isotropic estimate by Σ 1 / 2 .
The loss of the estimator will then be gaugedby the Mahalanobis distance, parameterized by Σ .
We refer the reader to Appendix G for a discussionon more challenging directions in which the isotropic Gaussian assumption can be relaxed.
Flip probability estimation for a given estimator of θ ∗
In this section, we consider the problem of estimating δ whenever θ ∗ is approximately known to be 253 θ ♯ .
We propose a simple estimator, and then discuss the importance of the accuracy of θ ∗ .
We thenderive an impossibility result for the matched case, θ ♯
= θ ∗ .
First note that an estimator for δ can be easily obtained from an estimator for ρ , with essentially thesame error rate, via ˆ δ = 12 (1 − ˆ ρ ) .
Thus we focus on estimating ρ .
Assume for simplicity that nis even.
Observing that E [ X ⊤ 2 i
X
i +1 ]
= ρ ∥ θ ∗ ∥ 2 , we propose the following natural estimator for ρ , 258 which replaces the population average with empirical average:
[Equation]
That is, the estimator is based on evaluating the correlation of each of two adjacent samples
X 2 i and
X 2 i − 1 .
We ﬁ rst state a general bound on the estimation error of this estimator.
We then consider thecase in which θ ∗ is known, and show how the estimation error is improved in this case.
Theorem 4.
Assume that d ≤ n .
Let θ ∗ ∈ R d and let θ ♯ be an estimate of θ ∗ .
Set ˆ ρ ≡ ˆ ρ corr ( X n 1 ; θ ♯ )and ˆ δ = 12 (1 − ˆ ρ ) .
Then, it holds with probability 1 − 8 n that
[Equation]
The proof of Theorem 4 appears in Appendix C.1.
Note that Theorem 4 states a high-probabilitybound, suitable to its usage later on in Section 4.
A bound on the expectation of the error can beobtained by standard methods (integrating tails).
The effect of knowledge of θ ∗
If θ ∗ is known up to a sign, i.e., θ ♯ = ± θ ∗ , then for the purpose of ρ 268 (or equivalently δ ) estimation, the model (2) can be reduced to a one-dimensional model by rotationalinvariance of isotropic Gaussian (See additional details in Appendix C.2).
It then immediately followsfrom Theorem 4 that:
Corollary 5.
Assume that d ≤ n , ∥ θ ∗ ∥≤ 1 and θ ♯ = ± θ ∗ .
Let U n 1 be de ﬁ ned in as U i :=
∥ θ ∗ ∥ · S i + W i where W i ∼ N (0 , 1) i.i.d., ˆ ρ ≡ ˆ ρ corr ( U n 1 ; θ ♯ ) and ˆ δ = 12 (1 − ˆ ρ ) .
Then it holds withprobability 1 − 8 n that
[Equation]
Numerical validation of the performance of the estimator ˆ δ [Equation] ( X
[Equation]
θ [Equation] =
[Equation]
) in themismatched (Theorem 4) and matched (Corollary 5) cases is provided in Appendix F.
We next consider an impossibility lower bound.
Proposition 6.
Suppose that θ ♯ = θ ∗ and ∥ θ ∗ ∥≤ 1 √ 2 .
Then inf ˆ δ ( U n 1 ) sup δ
[Equation]
32 √ n ,where the in ﬁ mum is over any estimator ˆ δ ( U n 1 ) based on the model U i =
∥ θ ∗ ∥ S i + W i where each 279 W i is i.i.d.
N (0 , 1) .
The proof of Proposition 6 is presented in Appendix C.2.
According to Corollary 5 and Proposition6, in estimating δ with a known θ ∗ , though the dependence Θ ( 1 √ n ) of the minimax error rate onthe sample size is shown to be nearly optimal, it is unclear what the optimal dependence on thesignal strength should be.
This is left as an interesting open question and we discuss the challengesassociated with this problem in Appendix C.2.
Mean estimation under an unknown ﬂ ip probability
As we have seen, if an estimator for θ ∗ knows the value of δ , and if both δ ≤ 12 and d ≤ δ n hold,then the estimator can achieve improved error rates over the GMM case ( δ = 12 ).
In this section, weassume that both θ ∗ and δ are unknown, and so the estimator is required to estimate δ in order to usethis knowledge for an estimator of θ ∗ .
We propose an estimation procedure of three steps based onsample splitting of 3 n samples.
We mention at the outset that the regime in which improvement ispossible will be for low signal strength ∥ θ ∗ ∥ ≲ 1 (low separation between the components), and up 292 to a dimension which depends on δ .
Of course the estimation procedure does not know ( θ ∗ , δ ) in 293 advance, and so it is required to identify if ( θ ∗ , δ ) are in this regime during its operation.
We now begin with an overview of the steps of the estimation algorithm.
At Step A, the algorithmestimates θ ∗ based on X n 1 assuming a Gaussian mixture model ( δ = 12 ) to obtain an estimate ˆ θ ( A ) .
Then, based on ∥ ˆ θ ( A ) ∥ , the algorithm decides whether improvement is potentially possible had δwas known.
There are two cases.
The ﬁ rst case is that ∥ ˆ θ ( A ) ∥ is too low, and then its estimate is notsuf ﬁ ciently accurate to be used in the next steps.
Essentially, this happens when the norm is belowthe global minimax rate ( dn ) 1 / 4 , and the estimation error of the norm on the same scale as the normof ∥ θ ∗ ∥ .
A trivial estimator of ˆ θ = 0 is then optimal in terms of error rates.
It can be already notedat this step that while the global minimax rate for the known δ case is ( δ dn ) 1 / 4 , here the algorithmalready stops and estimates ˆ θ = 0 even if just ∥ θ ∗ ∥ ≲ ( d n ) 1 / 4 , leading to larger global minimax rate.
The second case is that ∥ θ ∗ ∥ is larger than a constant.
In this case, the estimation based on a GMMalready achieves the optimal parametric O ( q dn ) error rate of the Gaussian location model, and so nofurther estimation steps are necessary.
Otherwise, an improvement in the estimation is possible.
Thealgorithm proceeds to Step B, and uses X 2 nn +1 to obtain an estimate ˆ δ ( B ) of δ based on the mismatchedθ ♯ ≡ ˆ θ ( A ) .
Then, based on the estimate ˆ δ ( B ) the algorithm decides whether the accuracy of ˆ δ ( B ) issuf ﬁ cient to be used in an re ﬁ ned estimation of θ ∗ .
If the accuracy of ˆ δ ( B ) is not good enough, thenthe algorithm outputs the estimate from Step A, that is ˆ θ ( A ) .
Otherwise, it proceeds to Step C, inwhich θ ∗ is re-estimated using X 3 n 2 n +1 , based on a mismatched choice of k , that is k ≍ 1ˆ δ ( B ) insteadof k = 18 δ .
Intuitively, the estimated value ˆ δ ( B ) should be larger than δ so the resulting block sizek ≍ 1ˆ δ ( B ) will be such that the gain in the block is still close to 1 with high probability.
On the otherhand, it is desired that ˆ δ ( B ) will be on the same scale as δ so that the estimation rate (295) (see also(7)) – which now essentially holds with ˆ δ ( B ) instead of δ – would be as small as possible.
Thus, if thealgorithm has assured in Step B that ˆ δ ( B ) ≍ δ , then at Step C it will achieve the error rate indicatedin (295).
The formal description of the estimation algorithm is provided in Algorithm 1.
We remark thatre ﬁ ning the estimation of δ can be easily incorporated as a fourth step of this algorithm, but we donot present this in order to keep the statement of the result simple.
The error of the estimator outputby Algorithm 1 is as follows:
Theorem 7.
There exist constants c 1 , c 2 ≥ 0 and λ θ , λ δ ≥ 1 such that if d ≤ n 4 λ 2 θ log 2 ( n ) ∧ 16 then theoutput ˆ θ of Algorithm 1 satis ﬁ es for any θ ∗ ∈ R d , with probability 1 − O ( 1 n ) : If d ≤ 1 64 λ 2 δ λ 2 θ log 2 ( n )
[Equation]
4 nthen
[Equation]
If d ≥ 1 64 λ 2 δ λ 2 θ log 2 ( n ) δ 4 n thenloss
[Equation]
[Equation]
[Equation] [Equation] [Equation]
[Equation]
Theorem 7 implies that up to logarithmic factors, the error rates of the known δ case are recoveredfor low enough dimension d ≲ δ 4 n and ∥ θ ∗ ∥ ≳
� d δ 2 n  1 / 4 .
The analysis of Algorithm 1 appears in
Appendix D. Numerical validation of the performance of Algorithm 1 can be found in Appendix F.
The impact of unknown δ on the estimation error of θ ∗
Comparing Theorems 2 (known δ ) and 7(unknown δ ) reveals the deterioration in the estimation of θ ∗ due to the lack of knowledge of δ fromthe three aspects mentioned in Section 1.3 (ignoring logarithmic factors):
First, the global minimaxrate is O ( dn ) 1 / 4 as for the GMM, instead of the rate O ( δ dn ) 1 / 4 for the Markov model case with knownδ .
Second, at the regime ( dn ) 1 / 4 ≲ ∥ θ ∗ ∥ ≲ ( d δ 2 n ) 1 / 4 the error rate is O ( 1 ∥ θ ∗ ∥
q dn ) instead of the lower
O ( 1 ∥ θ ∗ ∥
q δ dn ) .
Third, the algorithm is only effective when the dimension is as low as d ≲ δ 4 n .
Forhigher dimensions, the rates of the GMM are achieved, which can be achieved even without theknowledge of δ .
Conclusion and future work
In this paper, we have considered an elementary, yet fundamental, high-dimensional model withmemory.
We have obtained a sharp bound on the minimax rate of estimation in case the underlyingstatistical dependency ( ﬂ ip probability) is known, and proposed a three-step estimation algorithmwhen it is unknown.
This has revealed the gains possible in estimation rates due to the memorybetween the samples, and smoothly interpolated between the extreme cases of GLM and GMM.
Aninteresting open problem is to either characterize the optimality of the algorithm or improving inthe unknown δ case; this requires understanding optimal estimation of the ﬂ ip probability, and thechallenges associated with this problem are illuminated.
Naturally, as the model considered in this paper is basic, there is an ample of possibilities to generalizethis model.
These include, a larger number of components in the mixture, statistical dependency witha more complicated graphical structure between the data samples, existence of nuisance parameterssuch as the noise variance, sharp ﬁ nite-sample/
ﬁ nite-iteration analysis of speci ﬁ c practical algorithmssuch as Baum-Welch, location-scale model with anisotropic noise, heavy-tailed noise, and so on.