The price of ignorance: how much does it cost to forget noise structure in low-rank matrix estimation?
Anonymous Author(s)
AfﬁliationAddress email
Abstract
We consider the problem of estimating a rank- 1 signal corrupted by structured rotationally invariant noise, and address the following question: how well do inferencealgorithms perform when the noise statistics is unknown and hence Gaussian noiseis assumed?
While the matched Bayes-optimal setting with unstructured noise iswell understood, the analysis of this mismatched problem is only at its premises.
Inthis paper, we make a step towards understanding the effect of the strong sourceof mismatch which is the noise statistics.
Our main technical contribution is therigorous analysis of a Bayes estimator and of an approximate message passing(AMP) algorithm, both of which incorrectly assume a Gaussian setup.
The ﬁrstresult exploits the theory of spherical integrals and of low-rank matrix perturbations; the idea behind the second one is to design and analyze an artiﬁcial AMPwhich, by taking advantage of the ﬂexibility in the denoisers, is able to “correct” themismatch.
Armed with these sharp asymptotic characterizations, we unveil a richand often unexpected phenomenology.
For example, despite AMP is in principledesigned to efﬁciently compute the Bayes estimator, the former is outperformedby the latter in terms of mean-square error.
We show that this performance gap isdue to an incorrect estimation of the signal norm.
In fact, when the SNR is largeenough, the overlaps of the AMP and the Bayes estimator coincide, and they evenmatch those of optimal estimators taking into account the structure of the noise.
Introduction
The estimation of a low-rank matrix from noisy data is a central problem in machine learning, and itappears, e.g., in sparse principal component analysis (PCA)
[50, 88], community detection [1, 62]
and group synchronization [69].
In this paper, we consider the prototypical task of recovering asymmetric rank- 1 matrix XX ⊺ from noisy observations of the form
[Equation]
Here, λ ∗ > 0 is the signal-to-noise ratio (SNR) which quantiﬁes the signal strength, and Z ∈ R N × N isa random matrix that captures the noise.
A natural estimator of X is given by the principal eigenvectorof Y .
Its performance and, more generally, the behavior of the eigenvalues and eigenvectors of (1) hasbeen studied in exquisite detail in statistics [49, 68] and random matrix theory [7, 8, 21, 22, 26, 40,52, 6].
Going beyond the spectral estimator given by the principal eigenvector, approximate messagepassing (AMP) constitutes a popular family of iterative algorithms.
The reason for this popularity liesin two especially attractive features: (i) AMP algorithms can be tailored to exploit knowledge on thestructure of the signal; and (ii) under suitable assumptions, their performance in the high-dimensionallimit is accurately tracked by a deterministic recursion known as state evolution [18, 24, 48].
Usingthe state evolution machinery, it has been showed that AMP achieves Bayes-optimal performance in some Gaussian models [29, 30, 61, 13], and a bold conjecture from statistical physics is that
AMP is optimal in the class of polynomial-time algorithms for a large class of inference problemswith random design.
AMP algorithms have been recently considered for several practical scenarios,including genetics [86], inpainting [66] and MRI image recovery [76].
A general class of noise models that has received attention in the literature is given by the family ofrotationally invariant matrices .
This is much milder than requiring Z to be Gaussian: it only imposesthat the matrix of eigenvectors is uniformly random, and allows for an arbitrary spectrum.
Hence, Zcan capture the complex correlation structure which often occurs in applications (e.g., recommendersystems [20] and bioinformatics [72]).
However, estimating noise statistics from data is costly or, forproblems involving large-scale datasets (computational genomics is a paradigmatic example [36, 2]),may be impossible.
Thus, one natural idea is to simply assume Gaussian statistics for Z .
The case inwhich Z is actually Gaussian has been thoroughly studied [53, 28, 55, 54, 12, 61].
Beyond Gaussianity,a rapidly growing literature is focusing on rotationally invariant models assuming perfect knowledgeof the statistics of the structured matrix appearing in the problem (such as noise in inference, asensing, data, or coding matrix in regression tasks, weight matrices in neural networks, or a matrixof interactions in spin glass models)
However, despite this impressive progress when the noise statistics is known, low-rank estimationin a mismatched setting with partial to no knowledge of the statistics of the rotationally invariantnoise matrix remains poorly understood .
The goal of this work is thus to shed light into the followingfundamental question:
Suppose that the noise statistics is unknown or unreacheable, and hence Gaussian noise is naivelyassumed.
What is the impact of this mismatch on the overall performance of inference methods?
1.1 Summary of contributions
In this paper, we provide rigorous performance guarantees for two inference methods which incorrectly assume Gaussian noise statistics: a Bayes estimator, and an AMP algorithm.
Then, byexploiting these sharp analytical characterizations, we describe a number of surprising effects comingfrom numerical simulations.
Our main ﬁndings are detailed below.
Theoretical results.
(i)
We give a closed-form expression for the mean-square error (MSE) of the
Bayes estimator which samples from the mismatched posterior (cf.
Theorem 1).
Under a certainconcentration assumption, we also present an asymptotic result on the overlap of such estimator.
(ii)
We provide a state evolution analysis (cf.
Theorem 2) for the Gaussian AMP algorithm which isdesigned for Gaussian noise.
Numerical results.
The mismatched Bayes and AMP estimators display surprising behaviors –already for the case of a spherically distributed signal – and the two performance metrics (MSE andoverlap) exhibit a remarkably different phenomenology: (i) As for the MSE, the Gaussian AMP isoutperformed by the Bayes estimator.
Here, the surprise comes from the fact that AMP algorithmsare, in principle, designed to sample from the posterior distribution (and this is often what happens inthe matched case).
(ii)
In contrast, when the SNR is large enough, the overlaps of the two mismatchedestimators coincide , and they even match the overlap of estimators which exploit the noise statistics,namely the optimal spectral method which minimizes the MSE and the correct AMP designed in[37].
(iii) Under certain conditions, the MSE of the mismatched Bayes estimator matches that of a
Gaussian spectral method with no information on the noise structure.
(iv) The mismatched estimatorsare outperformed – in terms of MSE – by the optimal spectral method and the correct AMP, whoseperformance coincide.
(v) Finally, the MSE curves of the Bayes and Gaussian spectral methodsexhibit a striking non-monotone behavior.
1.2 Related work
The impact of mismatch.
Given the practical relevance of understanding the effect of mismatchin statistical inference, a line of work has approached the issue from various angles.
Regression beingprobably the most paradigmatic model of inference task, mismatch has been thoroughly studied in thiscontext.
In information theory, the trend was initiated in [83, 85].
Studies for M-estimation and robuststatistics followed based on statistical mechanics approaches [3], on the analysis of approximatemessage passing algorithms [25, 77, 32, 43], on Gordon’s convex min-max theorem [81, 80] and theleave-one-out method [35, 34].
Concerning the analysis of Bayesian approaches to mismatched linearregression, we refer to [27, 63, 75, 11] or [9] for a review.
In contrast, the problem studied in the present paper, namely low-rank matrix inference with mismatch,has received attention only recently.
It is clear that in some way, related questions were analyzedin the aforementioned random matrix theory literature, although from a rather different perspective.
Concerning the precise issue of the performance degradation due to mismatch in Bayesian inference,an exception (and inspiration for our work) is the recent paper [71].
However, the authors of [71]focus on mismatch in the signal-to-noise ratio for the Gaussian noise setting; their results can thus berecovered as a special case of the present ones.
To the best of our knowledge, this is the ﬁrst workthat considers mismatch in the noise statistics for low-rank matrix estimation.
Approximate message passing.
AMP algorithms have been applied to a wide range of inferenceproblems.
Examples include estimation in linear models [19, 18, 31], generalized linear models[73, 13, 58, 59], and low-rank matrix recovery with Gaussian noise [17, 29, 41, 51, 55, 61], see alsothe survey [39].
A general AMP iteration for rotationally invariant matrices has been recently analyzedin [37, 87], and by providing suitable instances of this abstract iteration, AMP algorithms have beendeveloped for low-rank [37, 87, 60] and generalized linear models [82].
Furthermore, an AMP-basedmethod which uses the classical idea of empirical Bayes to reduce the high-dimensional noise in
PCA is proposed in [86], which also provides applications to genetics.
However, the existing resultscannot be applied to the mismatched setting considered in this work, since the Gaussian AMP doesnot contain the right Onsager corrections.
In fact, the algorithm designer assumes Gaussian statisticsfor the noise and, therefore, constructs an AMP algorithm with the Onsager correction suitablefor Gaussian noise.
Finally, we remark that the performance of the Gaussian AMP is numericallycompared with that of the correct AMP (exploiting the knowledge of noise statistics) in [87].
Our
Theorem 2 provides rigorous foundations for such a comparison.
Setup of the problem
Random matrix
theory preliminaries
We start with some useful notions of random matrix theory.
Given a probability measure ρ of compactsupport K ⊆ R , we let H ρ ∶ R / K ↦ R be the Hilbert transform of ρ : H ρ ( z ) ∶= ∫ K ρ ( dγ )( z − γ ) − 1 .
We deﬁne γ ∶= max K , γ ∶= min K , h ∶= lim
z ↓ γ H ρ ( z ) and h ∶= lim z ↑ γ H ρ ( z ) , where the limitsexist due to the monotonicity of H ρ but may be inﬁnite.
As H ρ is a bijection between R / K and itsimage ( h,h )/{ 0 } , its inverse exists and it is denoted by K ρ ∶ ( h,h )/{ 0 } ↦ R / K .
The R -transformof ρ is R ρ ∶ ( h,h )/{ 0 } ↦ R / K given by R ρ ( x )
∶= K ρ ( x ) − 1 / x .
The coefﬁcients { ¯ κ k } k ≥ 1 of the
Taylor series of R ρ are the free cumulants associated to ρ , i.e., R ρ ( x ) =
∑ ∞ k =
¯ κ k + 1 x k , and they canbe computed from the moments of ρ , see e.g. Section 2.5 of [64].
Furthermore, R ′
ρ ( x ) and H ′
ρ ( z )denote the derivatives of the R -transform and the Hilbert transform of ρ , respectively.
2.2 Model of mismatched low-rank matrix estimation
We consider the problem of estimating a rank-one informative spike XX ⊺ corrupted by an additive symmetric noise matrix Z ∈ R N × N from data Y generated as in (1).
The scalar λ ∗ ∈ R ≥ 0is
the signal-to-noise ratio (SNR), and the noise Z can be decomposed as Z = O Σ O ⊺ , whereΣ ∶= diag ( γ 1 ,...,γ N ) ∈ R N × N is a diagonal matrix containing the eigenvalues of Z and Ois some orthogonal matrix.
We also denote by γ N and γ N respectively max { γ 1 ,...,γ N } andmin { γ 1 ,...,γ N } .
Similarly, ν N and ν N denote the largest and smallest eigenvalues of the datamatrix Y .
Throughout the paper, the main technical assumption will be the following.
Assumption 1.
The signal X has norm √
N .
The random noise matrix
Z is rotationally invariantand is independent of X .
Moreover, the empirical measure N − 1 ∑ i ≤ N δ γ i of eigenvalues of Zconverges weakly towards a limiting measure ρ of compact support K ⊆ R .
Finally, γ N and γ Nconverge a.s. to γ = max K and γ = min K , respectively.
The noise matrix Z being rotationally invariant means that O is a random orthogonal matrix (i.e.,sampled from the Haar measure) independent of Σ .
Under Assumption 1, we have that ν N and ν Nconverge a.s. to ﬁnite limits, which we denote by ν and ν , respectively.
These have an explicit form interms of ρ and λ ∗ given in [21, Theorem 2.1] and which we reproduce for convenience in Appendix A.
We remark that the assumption on the ground-truth signal is rather mild: X can have any distribution over the sphere of radius √
N , and it might even be deterministic.
For example, X ∈ {− 1 , 1 } N couldbe a uniform vector over the centred N -dimensional hyper-cube of sides√
N .
For the analysis of the Bayes estimator, we require a second assumption on the asymptotic eigenvaluedistribution ρ .
This ensures that the limit for the overlap of the spectral estimators with the signal iscontinuous in the SNR, whenever a small additive Wigner noise of variance (cid:15) ≥ 0 is added to Y .
Assumption 2.
Let ρ (cid:15) be the spectral density of the free convolution of ρ and a semicircle law ofradius 2 (cid:15) ≥ 0 .
Let H ρ (cid:15) be the associated Hilbert transform and γ (cid:15) the rightmost point of the supportof ρ (cid:15) .
Then, we assume that lim z ↓ γ (cid:15) H ′
ρ (cid:15) ( z ) =
−∞ for all (cid:15) ≥ 0 .
For the deﬁnition of the free convolution and its link to random matrix theory, we refer the reader to[4, 70].
We remark that Assumption 2 is satisﬁed by a wide class of random matrices.
In particular,by combining Theorem 2.2 of [10] with Proposition 2.4 of [21], it sufﬁces that the support of thelimiting spectral measure ρ is (i) compact, (ii) connected, and (iii) it has a proper decay rate at itsedges (i.e., ρ is of Jacobi type), see Assumption 2.1 of [10].
These conditions can be easily checkedfor many random matrix models, including the ones discussed in the examples of Section 4.
Note that, if ρ is the semicircle law of radius 2 , the noise is asymptotically equal in distributionto a standard Wigner matrix W with density ∼ exp (− N 4 Tr W 2 ) .
As the elements of this type ofsymmetric matrices are i.i.d.
Gaussian, in this case we say that there is Gaussian noise .
For otherlimiting ρ , the elements of Z ≠ W remain correlated and we say that there is structured noise .
Wedenote by W a sequence of standard Wigner matrices while we reserve Z for a generic sequence ofrotationally invariant matrices.
Sources of mismatch.
In this paper, we consider what happens when there is a mismatch betweenthe true noise statistics and the assumptions on the noise statistics made in the inference algorithm.
Inparticular, we study the case in which the noise is assumed to be Gaussian.
Gaussianity is in fact themost standard assumption made when precise knowledge of the noise structure is lacking.
We alsoconsider the case in which the SNR is estimated incorrectly, i.e., the statistician assumes that the datais generated according to ( √ λ / N ) XX ⊺ + W , where λ ≠ λ ∗ and λ ∗ is the correct SNR present in(1).
Our goal is to quantify how these sources of ignorance affect the algorithmic performance.
2.3 Three classes of inference procedures for PCA
Mismatched Bayes estimator.
The statistician assumes Gaussian noise and that the signal X hasno speciﬁc structure, i.e., it is uniformly distributed on the N -dimensional sphere S N − 1 ( √
N ) .
Themismatched posterior distribution reads, using the Gaussian log-likelihood − N
Tr ( Y − √ λN xx ⊺ ) 2 ,
[Equation]
where Z N ( Y ) is the normalization constant and µ N is the uniform measure over S N − 1 (√ N ) .
Incase the signal X lies on the sphere (by Assumption 1) but is not uniformly distributed on it, thenthis leads to a third source of mismatch.
However, the MSE formula of Theorem 1 does not dependon it; e.g. if X ∈ {− 1 , 1 } N the error remains the same.
The origin of this is the uniformly sphericalprior used in the posterior (2) which is the most uninformative one.
The Bayes estimators we analyze are obtained as the posterior means
[Equation]
which may not be practical to compute.
Notice that, if the noise is Wigner, then the likelihood exp (⋯)in (2) is correct.
This Bayes-optimal case has already been rigorously characterized (see, for example,[12, 14, 54]), and it will serve us as a base case for comparison with our mismatched setting.
The performance of a sequence of estimators M N ( Y ) ∈ R N × N of the spike XX ⊺ is quantiﬁed viathe asymptotic mean-square error (MSE):
[Equation]
where ∥ ⋅ ∥ 2F denotes the Frobenius norm and E the expectation over the spike XX ⊺ and the noise
Z .
We also consider another performance measure which is insensitive to the norm of an estimator ˆ x of X , namely, the rescaled overlap:
[Equation]
where ∥ X ∥ = √ N due to the spherical constraint
This alignment measure is sometimes alsoreferred to as cosine distance .
For both AMP and spectral estimators, this overlap will almostsurely be deterministic.
If the norm of the estimator vanishes, by convention we ﬁx the overlapto zero.
Note that the deﬁnition (5) is not meaningful for the Bayes case.
In fact, if one deﬁnesˆ x mis ∶= ∫ x P mis ( d x ∣ Y ) , then this quantity is zero by sign symmetry ( ± x have the same posteriorweight).
Thus, the correct deﬁnition of the overlap for the mismatched Bayes estimator is
[Equation]
while for the “one-shot” AMP and spectral estimators we use the former deﬁnition (5).
Approximate message passing.
We remark that, as opposed to our analysis of the Bayes estimator,the state evolution characterization of AMP (Theorem 2) does not require X to be uniformlydistributed on S N − 1 (√ N ) .
The AMP analysis can accomodate more generic (potentially mismatched)prior distributions and, in fact, AMP algorithms are well equipped to exploit structure in the signal.
For simplicity, we assume to have access to an initialization ˆ x 1 ∈ R N , which is independent of thenoise Z and has a strictly positive correlation with X , i.e.,
[Equation]
where ( X , ˆ x 1 ) W 2 —→ ( X, ˆ x 1 ) denotes convergence of the joint empirical distribution of ( X , ˆ x 1 ) tothe random variable ( X, ˆ x 1 ) in Wasserstein-2 ( W 2 ) distance.
Then, for t ≥ 1 the AMP iteration reads
[Equation]
where we assume that ˆ x 0 = 0 .
Here, the function h t + 1 ∶ R → R is applied component-wise, and it canbe chosen in order to exploit prior information about the signal; β 1 = 0 and, for t ≥ 2 , β t = ⟨ h ′ t ( x t − 1 )⟩ ,where h ′ t denotes the derivative of h t .
The AMP estimator of X is ˆ x t , and the one of the spike XX ⊺is M t AMP =
ˆ x t ( ˆ x t ) ⊺ .
We refer to this algorithm as Gaussian AMP , since this is the AMP that isimplemented for Gaussian noise (and known SNR).
For a discussion on how (8) can be derived, werefer the interested reader to the review [39].
We note that the initialization (7) is impractical andone can design AMP iterations which are initialized with the eigenvector v N of the data matrix Yassociated to the largest eigenvalue ν N , see [61, 60, 87].
We are also able to provide a state evolutionresult for the Gaussian AMP with spectral initialization, and this is discussed in Appendix C.4.
Spectral estimators.
Finally, we consider spectral estimators of the form C v N v ⊺ N , where v N isthe unit-norm eigenvector associated to the largest eigenvalue ν N ( Y ) and C > 0 is a scaling constanttaking into account the spectrum of the data matrix.
The asymptotic description of v N and ν N foradditive and multiplicative low-rank perturbations of rotationally invariant matrices has been obtainedin [21, 22, 6].
In particular, by Theorem 2.2 of [22], the squared overlap ⟨ X , v N ⟩ 2 / N converges a.s.to a constant C ( ρ,λ ∗ ) , whose explicit expression is recalled in Appendix A. We study two variants:
● The optimal spectral estimator (OptSpec) is given by M OS ∶= C OS v N v ⊺ N , where C OS ( ρ,λ ∗ ) ∶=
C ( ρ,λ ∗ ) is the optimal scaling that minimizes the MSE.
● The Gaussian mismatched spectral estimator (GauSpec) is given by M GS ∶= C GS v N v ⊺ N , whereC GS is the optimal scaling if the noise was Gaussian and the SNR was equal to λ > 0 .
By letting ρ SCbe the standard semi-circle law, we have C GS ( λ ) ∶= C ( ρ SC ,λ )
= 1 − 1 / λ .
For both OptSpec and GauSpec, the estimator of X is given by ˆ x GS =
ˆ x
OS = ˆ
x
Spec ∶= √
N v N .
Notice that, while OptSpec requires knowing exactly the statistics of the noise and the SNR, GauSpecrepresents the spectral estimator that would be used by a statistician who assumes the noise to be
Gaussian and the SNR λ (instead of λ ∗ ).
An application of Theorem 2.2 in [22] gives that
[Equation]
, and
[Equation]
Furthermore, the respective overlaps coincide and are given by √ C OS .
Note that MSE ( M GS ) =
MSE ( M OS ) + 12 ( C ( ρ,λ ∗ ) − ( 1 − 1 / λ )) 2 .
Thus, as expected, we have MSE ( M OS ) ≤ MSE ( M GS ) .
We also remark that, as the noise matrix is rotationally invariant, both the MSE and the overlap of thespectral estimators do not depend on the ground-truth signal distribution.
Main results: performance of inference algorithms
3.1 Mismatched Bayes estimator
Consider the following functions of λ,λ ∗ > 0 and of the asymptotic spectral noise density ρ :
[Equation]
where 1 ( E ) stands for the indicator function of the event E .
The idea is that Q ( λ,λ ∗ ) and M ( λ,λ ∗ )represent the squared norm of the mismatched Bayes estimator and its inner product with the signal,respectively.
This leads to our main result on the mismatched Bayes MSE, which is stated below.
Theorem 1 (Performance of mismatched Bayes estimator) .
Consider a spiked model (1) .
Let Assumptions 1 and 2 hold.
Then, the MSE of the mismatched Bayes estimator (3) is given by
[Equation]
As a consequence of the auxiliary lemmas used in the proof of Theorem 1 and presented in Appendix B, we have that N − 2 ∫ P mis ( d x ∣ Y )⟨ X , x ⟩ 2 converges almost surely to M ( λ,λ ∗ ) .
Furthermore, for noise matrices regularized by arbitrarily small Wigner matrices, by using similar techniquesas in [15] we can prove that ∥ M mis ∥ F / N converges almost surely to Q 1 / 2 ( λ,λ ∗ ) .
We conjecture thatthis convergence holds even without the Wigner regularization.
By assuming this conjecture, we havethe following almost sure convergence result for the overlap:
[Equation]
which holds whenever the denominator is non-zero.
When comparing the overlaps of differentmethods in Section 4, we will use (11) for the mismatched Bayes estimator.
Let us highlight the following property of the MSE (10) when there is no mismatch in the SNR:
If λ = λ ∗ and h ≥ 1 , then MSE ( M mis ) =
MSE
[Equation]
M
[Equation] [Equation]
[Equation]
However, in general MSE ( M mis ) and MSE ( M GS ) differ.
E.g., when the assumed SNR λ is differentfrom the real one λ ∗ , the norm of M mis incorporates information about λ ∗ while the one for M GSonly depends on λ .
Finally, we note that, from a statistical view-point, Assumption 1 can be replacedby the following: X is uniformly distributed on S N − 1 (√ N ) and is independent of Z , which may bea generic symmetric matrix with converging empirical spectral density.
Sketch of the proof of Theorem 1.
The goal is to evaluate the asymptotic values of E
∥ M mis ∥ 2Fand tr E [ M mis XX ⊺ ] , from which both the MSE and the mean overlap of the Bayes estimator can beobtained.
Our strategy is to ﬁrst compute the log-moment generating function f mis of the mismatched
Bayes model.
Then, the above quantities of interest can be accessed by taking derivatives of f miswith respect to appropriate parameters.
However, as the noise in the inference model is not Gaussian, 253 E ∥
M mis ∥ 2F cannot be computed in the standard way using an “I-MMSE” type of formula [46] (like itis done, e.g., in [71]).
To address this issue, we introduce a generalized model for which the noisematrix is given by the original one plus an independent Wigner matrix multiplied by a small parameter(cid:15) .
We then compute the log-moment generating function of this generalized model using the theory oflow-rank perturbations of rotationally invariant random matrices [21] and of the low-rank sphericalintegral [45].
Finally, the desired results are obtained by a limiting argument in (cid:15) → 0 .
3.2 Approximate message passing
Our main result is a characterization of the iterates (8) in the high-dimensional limit.
We show thatthe joint empirical distribution of ( x 1 ,..., x t , ˆ x 1 ,..., ˆ x t + 1 ) converges (in W 2 ) to the random vector 262 ( x 1 ,...,x t , ˆ x 1 ,..., ˆ x t + 1 ) .
The law of the random vector ( x 1 ,...,x t , ˆ x 1 ,..., ˆ x t + 1 ) is captured bya deterministic state evolution (SE) recursion, which can be expressed via a sequence of vectorsµ t = ( µ 1 ,...,µ t ) and matrices Σ t , ∆ t , B t ∈ R t × t deﬁned as follows.
We start with the initialization
[Equation]
where λ ∗ is the true SNR (see (1)), and { ¯ κ k } k ≥ 1 are the free cumulants associated to the asymptoticspectral measure of the noise ρ .
For t ≥ 1 , given µ t , Σ t , ∆ t , B t , let{( z 1 ,...,z t ) ∼ N( 0 , Σ t ) and independent of ( X, ˆ x 1 ) , [Equation]
[Equation]
where we have deﬁned ˆ x 0 = 0 , ¯ β 1 = 0 , and for t ≥ 2 , ¯ β t
= E [ h ′ t ( x t − 1 )] .
Then, the parameter µ t + 1 is
[Equation]
Next, we compute the matrices ∆ t + 1 and B t + 1 as
[Equation]
¯ κ
[Equation] ¯ Φ jt + 1 where ( ¯ Φ t + 1 ) i,j = 0 if i ≤ j, and ( ¯ Φ t + 1 )
i,j = [-
-]E
[ ∂
j ˆ x i ] if i > j, (16) where ∂ j ˆ x i denotes the partial derivative ∂ z j h
[Equation]
[Equation]
µ
[Equation]
+ ∑ [Equation] B
[Equation]
k ˆ x k )) .
Finally, we deﬁne the covariance matrix Σ t + 1 as
[Equation]
Note that the t × t top left sub-matrices of Σ t + 1 , ∆ t + 1 and B t + 1 are given by Σ t , ∆ t and B t .
Atthis point, we are ready to state our state evolution characterization of the AMP algorithm (8).
The result is presented in terms of pseudo-Lipschitz test functions.
A function ψ ∶ R m → R ispseudo-Lipschitz of order 2 , i.e., ψ ∈ PL ( 2 ) , if there is a constant C > 0 such that ∥ ψ ( x ) − ψ ( y )∥ ≤
C ( 1 + ∥ x ∥ + ∥
y ∥)∥ x − y ∥ .
We also make the following assumption on the functions { h t + 1 } t ≥ 1 .
Assumption 3.
The function h t + 1 ∶ R
→ R is Lipschitz, and the partial derivatives
[Equation]
are continuous on a set of probability 1 , under the laws of ( z 1 ...,z t ) and ( ˆ x 1 ,..., ˆ x t ) given in (14) .
This requirement covers most practically relevant choices of h t
+ 1 (e.g., soft-thresholding or ReLU),and
it is rather standard in the AMP literature, see e.g. [39, 60, 82, 87].
Theorem 2 (State evolution of Gaussian AMP) .
Consider a spiked model (1) and the AMP algorithm(8) initialized as in (7) .
Let Assumptions 1 and 3 hold, and let ψ ∶ R 2 t + 2 → R be any pseudo-Lipschitzfunctions of order 2 .
Then, for each t ≥ 1 , we almost surely have that, as N → +∞ ,
[Equation]
18) where the random variables on the right are deﬁned in (14) .
By using Deﬁnition 6.7 and Theorem 6.8 of [84], one readily obtains that (18) is equivalent tothe convergence of ( x 1 ,..., x t , ˆ x 1 ,..., ˆ x t + 1 , X ) to ( x 1 ,...,x t , ˆ x 1 ,..., ˆ x t + 1 ,X ) in W 2 distance.
Applying (18) to the pseudo-Lipschitz functions ψ ( ˆ x t + 1 ,X )
=
( ˆ x t + 1 − X ) 2 , ψ ( ˆ x t
+ 1 ,X )
=
ˆ x t + 1 ⋅ Xand ψ ( ˆ x t + 1 ,X ) =
( ˆ x t + 1 )
, we obtain a high-dimensional characterization of the AMP performance.
Corollary 1.
Consider the setting of Theorem 2.
Then, for each t ≥ 1 , we almost surely have that
[Equation]
Sketch of the proof of Theorem 2.
The key insight is to exploit the ﬂexibility given by thedenoisers of the abstract AMP iteration [87] to “correct” the mismatch.
These denoisers are denotedby { ˜ h t } t ≥ 2 (see (33) and (34) in Appendix C.1) and they serve solely as a proof technique, hencethey do not have an impact on the practicality of the algorithm; in contrast, the denoisers { h t } t ≥ 2 (see(8)) can be chosen by the algorithm designer to take advantage of the information available about thesignal.
Thus, we can design an auxiliary AMP such that (i) it admits a state evolution characterization,and (ii) its iterates are close to (8).
This requires the delicate induction argument in Appendix C: in
Appendix C.1, we deﬁne the auxiliary AMP; in Appendix C.2, we prove a state evolution result for it;in Appendix C.3, we show by induction that (i) the state evolution parameters of the auxiliary AMPcoincide with those deﬁned in this section, and (ii) the iterates of the auxiliary AMP are close (in (cid:96) 2norm) to the iterates of the AMP (8), thus concluding the proof of Theorem 2.
Numerical results and discussion
In all experiments, the density ρ of the eigenvalues of the noise has unit variance and the distributionof the ground-truth signal is uniform on the sphere of radius √
N .
For the sake of simplicity, wedecouple the effect of the mismatch in (i) the noise statistics and (ii) the SNR.
More speciﬁcally, inthe ﬁrst two examples, we set λ
= λ ∗ , so that only mismatch in the noise statistics is present; and inthe last example, the noise is Gaussian, so that there is only SNR mismatch.
Additional results when
Z is the free convolution of Rademacher and semicircle spectra are reported in Appendix D.3.
Models of mismatch.
(a) Marcenko-Pastur spectrum.
An example of non-symmetric density isthe Marcenko-Pastur law ρ ( x ) = √ x
( 4 − x )/( 2 πx ) 1 ( x ∈ [ 0 , 4 ]) .
The R -transform is R ρ ( x )
=1 /( 1 − x ) , and the results are displayed in Fig.
1a (the same formulas apply if the law is centered).(b)
Uniform spectrum.
We let ρ be the uniform distribution U[−√ 3 , √ 3 ] .
In this case, we have
R ρ ( x ) = √ 3 / tanh (√ 3 x ) − 1 / x , and the results are in Fig.
1b.(c)
Wigner matrix/semicircle spectrum with mismatched SNR.
We consider a Gaussian noise matrix
W (as assumed by the statistician), but with mismatched SNR by setting λ = 4 λ ∗ , see Fig. 1c.
We remark that Assumption 1 is veriﬁed for all models.
Furthermore, in all the three cases, the supportof ρ is a single non-empty interval.
Hence, Theorem 2.2 of [10] and Proposition 2.4 of [21] can beused to show that Assumption 2 holds.
Inference algorithms and set-up.
The estimators of the spike XX ⊺ are compared in terms ofthe MSE (4)
) is plotted
(GauSpec) is plotted in yellow in Fig.
1a and 1c (where it differs from ) or
in Fig.
1b (where it coincides with Bayes).
The performance measures of both variants of spectralestimators are given at the end of Section 2.3.
Each experiment is repeated for 10 independent runs.
We report the average and error bars at 1 standard deviation.
A rich and surprising phenomenology.
(i)
An intriguing phenomenon is that, in terms of MSE,the Gaussian AMP does not perform as well as the mismatched Bayes estimator .
This effect occursin all our settings, and it is surprising as the Gaussian AMP (8) is precisely designed to efﬁcientlycompute the Bayes estimator (3).
In the Bayes-optimal case, it does so (outside of its computationalgap) [12].
However, if there is a mismatch, AMP does not compute the (mismatched) posterior mean .
This ﬁnding adds to the already existing evidence [5] that AMP is poorly understood, and studyingthe fundamental reasons behind this behavior is an exciting avenue for future research.(ii)
By comparing MSE 1 and overlap curves, we understand that the discrepancy between Gaussian
AMP and Bayes comes from an incorrect estimation of the signal norm.
In fact, at large enough
SNR, the overlaps of Bayes and Gaussian AMP match, and they even coincide with the overlaps ofoptimal algorithms (correct AMP and OptSpec), which exploit knowledge about the noise structure.
[Figure] [Figure]
[Figure] [Figure] [Figure] Understanding the origin of the MSE discrepancy (namely, a wrong estimate of the signal length) canpotentially lead to procedures which correct for this effect and, hence, reduce the MSE.(iii) When there is no SNR mismatch and ¯ h ≥ 1 (cf.
(12)), the MSEs of the Bayes and Gaussianspectral estimators match (see Fig.
1b).
Both have no information about the noise structure and aboutthe signal distribution.
Yet, this equality is remarkable given that the spectral estimator is the solutionof an optimization problem (it is a “zero temperature” estimator in physics parlance), while the Bayesestimator aims at computing the mean of a certain posterior distribution (it is a “ﬁnite temperature”
estimator).
(iv) All mismatched estimators are outperformed by the optimal spectral method and the correct AMP .
Both these algorithms take advantage of the noise statistics and achieve the same MSE
This suggeststhat the two estimators are Bayes optimal
We remark that solving this conjecture would requireunderstanding the
information
-theoretic
limits of low- rank estimation with structured noise.
(v) Finally, a striking phenomenon – ﬁrst observed in [71] – is that the Bayes and Gaussian spectral
MSE curves may be non-decreasing with the true SNR λ ∗ , see Figs.
1a and 1c.
Going beyond the
analysis in [ , we remark that, for large enough SNR, all estimators yield the same overlap and
therefore all “point in the correct direction” given by the leading eigenvector, see also part (ii) of this
discussion.
This links the initial MSE increase to a wrong estimation of the signal’s
norm
, due to anover-
conﬁdence
in the
data
quality ( recall that the assumed SNR λ is
equal
to 4 λ ∗ in Fig.
1c).
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reﬂect
the paper’scontributions and scope
[Yes]
(b) Did you describe the limitations of your work?
[Yes] Our model assumptions are clearly
explained in Assumptions 1, 2 and 3.
To obtain the overlap of the mismatched Bayesestimator, we
make a conjecture on the convergence of ∣∣ M mis ∣∣ F / N (see the discussion
right after the statement of Theorem 1).
(c) Did you discuss any potential negative societal impacts of your work?
[N/A] We do
not see direct paths to applications with negative societal impacts.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to
them?
[Yes]
If you are including theoretical results...(
a) Did you state
the
full
set of assumptions of all theoretical results?
Yes ] Our modelassumptions are clearly explained in Assumptions 1, 2 and 3.
To obtain the overlapof the mismatched Bayes estimator, we make a conjecture on the convergence of∣∣
M mis ∣∣ F / N ( see
the discussion right after the statement of Theorem 1).
b) Did you include complete proofs of all theoretical results?
[Yes] Proof sketches are
included right after the corresponding statements
The full proofs of Theorem 1 and
are in Appendices B and C, respectively.
If you ran experiments...
a) Did you include the code, data, and instructions needed to reproduce the main experi
-mental results (either in the supplemental material
or
as
a URL)
[Yes]
In Section 4,
we describe the experimental setup containing the instructions to reproduce the resultsof Figure 1 (and of Figure 2 contained in Appendix D.3 as well).
Additional details aregiven in Appendices D.1 and D.2.
The code that implements the various algorithms isincluded in the supplementary material.
b) Did you specify all the training details (e
g., data splits, hyperparameters, how theywere chosen)?
[Yes] All details of the experimental setup are described in Section
and in Appendices D.1 and D.2.
c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[Yes] Each experiment is repeated for 10 independent runs, and we report the average together with the conﬁdence interval at 1 standard deviation.(d)
Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?
[N/A]
We did not perform large scale
experiments that required to specify this type of details.
If you are using existing assets (e
g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[N/A] We are not using
existing assets or curating/releasing new assets.
(b) Did you mention the license of the assets?
[N/A]
(c) Did you include any new assets either in the supplemental material or as a URL?
[N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you’reusing/curating?
[N/A]
(e) Did you discuss whether the data you are using/curating contains personally identiﬁableinformation or offensive content?
[N/A]
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if
applicable?
[N/A]
We did not use crowdsourcing or conducted research with human
subjects.
(b) Did you describe any potential participant risks, with links to Institutional Review
[Table]
Board (IRB) approvals, if applicable?
[N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount
spent on participant compensation?
[N/A]