The price of ignorance: how much does it cost to forget noise structure in low-rank matrix estimation?
Anonymous Author(s)
AfﬁliationAddress email
Abstract
We consider the problem of estimating a rank- 1 signal corrupted by structured rotationally invariant noise, and address the following question: how well do inferencealgorithms perform when the noise statistics is unknown and hence Gaussian noiseis assumed?
While the matched Bayes-optimal setting with unstructured noise iswell understood, the analysis of this mismatched problem is only at its premises.
Inthis paper, we make a step towards understanding the effect of the strong sourceof mismatch which is the noise statistics.
Our main technical contribution is therigorous analysis of a Bayes estimator and of an approximate message passing(AMP) algorithm, both of which incorrectly assume a Gaussian setup.
The ﬁrstresult exploits the theory of spherical integrals and of low-rank matrix perturbations; the idea behind the second one is to design and analyze an artiﬁcial AMPwhich, by taking advantage of the ﬂexibility in the denoisers, is able to “correct” themismatch.
Armed with these sharp asymptotic characterizations, we unveil a richand often unexpected phenomenology.
For example, despite AMP is in principledesigned to efﬁciently compute the Bayes estimator, the former is outperformedby the latter in terms of mean-square error.
We show that this performance gap isdue to an incorrect estimation of the signal norm.
In fact, when the SNR is largeenough, the overlaps of the AMP and the Bayes estimator coincide, and they evenmatch those of optimal estimators taking into account the structure of the noise.
Introduction
The estimation of a low-rank matrix from noisy data is a central problem in machine learning, and itappears, e.g., in sparse principal component analysis (PCA)
community detection [1, 62]
and group synchronization [69].
In this paper, we consider the prototypical task of recovering asymmetric rank- 1 matrix XX ⊺ from noisy observations of the form
[Equation]
Here, λ ∗ > 0 is the signal-to-noise ratio (SNR) which quantiﬁes the signal strength, and Z ∈ R N × N isa random matrix that captures the noise.
A natural estimator of X is given by the principal eigenvectorof Y .
Its performance and, more generally, the behavior of the eigenvalues and eigenvectors of (1) hasbeen studied in exquisite detail in statistics [49, 68] and random matrix theory [7, 8, 21, 22, 26, 40,52, 6].
Going beyond the spectral estimator given by the principal eigenvector, approximate messagepassing (AMP) constitutes a popular family of iterative algorithms.
The reason for this popularity liesin two especially attractive features: (i) AMP algorithms can be tailored to exploit knowledge on thestructure of the signal; and (ii) under suitable assumptions, their performance in the high-dimensionallimit is accurately tracked by a deterministic recursion known as state evolution [18, 24, 48].
Usingthe state evolution machinery, it has been showed that AMP achieves Bayes-optimal performance in some Gaussian models [29, 30, 61, 13], and a bold conjecture from statistical physics is that AMPis optimal in the class of polynomial-time algorithms for a large class of inference problems withrandom design.
A general class of noise models that has received attention in the literature is given by the family ofrotationally invariant matrices .
This is much milder than requiring Z to be Gaussian: it only imposesthat the matrix of eigenvectors is uniformly random, and allows for an arbitrary spectrum.
Hence, Zcan capture the complex correlation structure which often occurs in applications (e.g., recommendersystems [20] and bioinformatics [72]).
However, estimating noise statistics from data is costly or, forproblems involving large-scale datasets (computational genomics is a paradigmatic example [36, 2]),may be impossible.
Thus, one natural idea is to simply assume Gaussian statistics for Z .
The case inwhich Z is actually Gaussian has been thoroughly studied [53, 28, 55, 54, 12, 61].
Beyond Gaussianity,a rapidly growing literature is focusing on rotationally invariant models assuming perfect knowledgeof the statistics of the structured matrix appearing in the problem (such as noise in inference, asensing, data, or coding matrix in regression tasks, weight matrices in neural networks, or a matrixof interactions in spin glass models)
However, despite this impressive progress when the noise statistics is known, low-rank estimationin a mismatched setting with partial to no knowledge of the statistics of the rotationally invariantnoise matrix remains poorly understood .
The goal of this work is thus to shed light into the followingfundamental question:
Suppose that the noise statistics is unknown or unreacheable, and hence Gaussian noise is naivelyassumed.
What is the impact of this mismatch on the overall performance of inference methods?
1.1 Summary of contributions
In this paper, we provide rigorous performance guarantees for two inference methods which incorrectly assume Gaussian noise statistics: a Bayes estimator, and an AMP algorithm.
Then, byexploiting these sharp analytical characterizations, we describe a number of surprising effects comingfrom numerical simulations.
Our main ﬁndings are detailed below.
Theoretical results.
(i)
We give a closed-form expression for the mean-square error (MSE) of the
Bayes estimator which samples from the mismatched posterior (cf.
Theorem 1).
Under a certainconcentration assumption, we also present an asymptotic result on the overlap of such estimator.
(ii)
We provide a state evolution analysis (cf.
Theorem 2) for the Gaussian AMP algorithm which isdesigned for Gaussian noise.
Numerical results.
The mismatched Bayes and AMP estimators display surprising behaviors– already for the case of a spherical signal prior – and the two performance metrics (MSE andoverlap) exhibit a remarkably different phenomenology: (i) As for the MSE, the Gaussian AMP isoutperformed by the Bayes estimator.
Here, the surprise comes from the fact that AMP algorithmsare, in principle, designed to sample from the posterior distribution (and this is often what happens inthe matched case).
(ii)
In contrast, when the SNR is large enough, the overlaps of the two mismatchedestimators coincide , and they even match the overlap of estimators which exploit the noise statistics,namely the optimal spectral method which minimizes the MSE and the correct AMP designed in[37].
(iii) Under certain conditions, the MSE of the mismatched Bayes estimator matches that of a
Gaussian spectral method with no information on the noise structure.
(iv) The mismatched estimatorsare outperformed – in terms of MSE – by the optimal spectral method and the correct AMP, whoseperformance coincide.
(v) Finally, the MSE curves of the Bayes and Gaussian spectral methodsexhibit a striking non-monotone behavior.
1.2 Related work
The impact of mismatch.
Given the practical relevance of understanding the effect of mismatchin statistical inference, a line of work has approached the issue from various angles.
Regression beingprobably the most paradigmatic model of inference task, mismatch has been thoroughly studied in thiscontext.
In information theory, the trend was initiated in [82, 84].
Studies for M-estimation and robuststatistics followed based on statistical mechanics approaches [3], on the analysis of approximatemessage passing algorithms [25, 76, 32, 43], on Gordon’s convex min-max theorem [80, 79] and theleave-one-out method [35, 34].
Concerning the analysis of Bayesian approaches to mismatched linearregression, we refer to [27, 63, 75, 11] or [9] for a review.
In contrast, the problem studied in the present paper, namely low-rank matrix inference with mismatch,has received attention only recently.
It is clear that in some way, related questions were analyzedin the aforementioned random matrix theory literature, although from a rather different perspective.
Concerning the precise issue of the performance degradation due to mismatch in Bayesian inference,an exception (and inspiration for our work) is the recent paper [71].
However, the authors of [71]focus on mismatch in the signal-to-noise ratio for the Gaussian noise setting; their results can thus berecovered as a special case of the present ones.
To the best of our knowledge, this is the ﬁrst workthat considers mismatch in the noise statistics for low-rank matrix estimation.
Approximate message passing.
AMP algorithms have been applied to a wide range of inferenceproblems.
Examples include estimation in linear models [19, 18, 31], generalized linear models[73, 13, 58, 59], and low-rank matrix recovery with Gaussian noise [17, 29, 41, 51, 55, 61], see alsothe survey [39].
A general AMP iteration for rotationally invariant matrices has been recently analyzedin [37, 86], and by providing suitable instances of this abstract iteration, AMP algorithms have beendeveloped for low-rank [37, 86, 60] and generalized linear models [81].
Furthermore, an AMP-basedmethod which uses the classical idea of empirical Bayes to reduce the high-dimensional noise in
PCA is proposed in [85], which also provides applications to genetics.
However, the existing resultscannot be applied to the mismatched setting considered in this work, since the Gaussian AMP doesnot contain the right Onsager corrections.
In fact, the algorithm designer assumes Gaussian statisticsfor the noise and, therefore, constructs an AMP algorithm with the Onsager correction suitablefor Gaussian noise.
Finally, we remark that the performance of the Gaussian AMP is numericallycompared with that of the correct AMP (exploiting the knowledge of noise statistics) in [86].
Our
Theorem 2 provides rigorous foundations for such a comparison.
Setup of the problem
2.1 Random matrix
theory preliminaries
We start with some useful notions of random matrix theory.
Given a probability measure ρ of compactsupport K ⊆ R , we let H ρ ∶ R / K ↦ R be the Hilbert transform of ρ : [Equation]
We deﬁne γ ∶= max K , γ ∶= min K , h ∶= lim
z ↓ γ H ρ ( z ) and h ∶= lim z ↑ γ H ρ ( z ) , where the limits exist due to the monotonicity of H ρ but may be inﬁnite.
As H ρ is a bijection between R / K and its image ( h,h )/{ 0 } , its inverse exists and it is denoted by K ρ ∶ ( h,h )/{ 0 } ↦ R / K .
The R -transform of ρ is R ρ ∶ ( h,h )/{ 0 } ↦ R / K given by
[Equation]
The coefﬁcients { ¯ κ k } k ≥ 1 of the Taylor series of R ρ are the free cumulants associated to ρ , i.e., R ρ ( x ) =
∑ ∞ k =
¯ κ k + 1 x k , and they can be computed from the moments of ρ , see e.g. Section 2.5 of [64].
Furthermore, R ′
ρ ( x ) and H ′
ρ ( z ) denote the derivatives of the R -transform and the Hilbert transformof ρ , respectively.
2.2 Model of mismatched low-rank matrix estimation
We consider the problem of estimating a rank-one informative spike XX ⊺ corrupted by an additive symmetric noise matrix Z ∈ R N × N from data Y generated as in (1).
The scalar λ ∗ ∈ R ≥ 0is
the signal-to-noise ratio (SNR), and the noise Z can be decomposed as Z = O Σ O ⊺ , whereΣ ∶= diag ( γ 1 ,...,γ N ) ∈ R N × N is a diagonal matrix containing the eigenvalues of Z and Ois some orthogonal matrix.
We also denote by γ N and γ N respectively max { γ 1 ,...,γ N } andmin { γ 1 ,...,γ N } .
Similarly, ν N and ν N denote the largest and smallest eigenvalues of the datamatrix Y .
Throughout the paper, the main technical assumption will be the following.
Assumption 1.
The signal X has norm √
N .
The random noise matrix
Z is rotationally invariantand is independent of X .
Moreover, the empirical measure N − 1 ∑ i ≤ N δ γ i of eigenvalues of Zconverges weakly towards a limiting measure ρ of compact support K ⊆ R .
Finally, γ N and γ Nconverge a.s. to γ = max K and γ = min K , respectively.
The noise matrix Z being rotationally invariant means that O is a random orthogonal matrix (i.e.,sampled from the Haar measure) independent of Σ .
Under Assumption 1, we have that ν N andν N converge a.s. to ﬁnite limits, which we denote by ν and ν , respectively.
These have an explicitform in terms of ρ and λ ∗ given in [21, Theorem 2.1] and which we reproduce for convenience in
Appendix A. We remark that the assumption on the ground-truth signal is rather mild: X can haveany distribution over the sphere of radius √
N , and it might even be deterministic.
For the analysis of the mismatched Bayes estimator, we also require a second technical assumptionon the asymptotic eigenvalue distribution ρ .
This ensures that the limit for the overlap of the spectralestimators with the signal is a continuous function of the SNR, whenever a small additive Wignernoise of variance (cid:15) ≥ 0 is added to Y .
Assumption 2.
Let ρ (cid:15) be the spectral density of the free convolution of ρ and a semicircle law ofradius 2 (cid:15) ≥ 0 .
Let H ρ (cid:15) be the associated Hilbert transform and γ (cid:15) the rightmost point of the supportof ρ (cid:15) .
Then, we assume that lim z ↓ γ (cid:15) H ′
ρ (cid:15) ( z ) =
−∞ for all (cid:15) ≥ 0 .
For the deﬁnition of the free convolution and its link to random matrix theory, we refer the reader to[4, 70].
We remark that Assumption 2 is satisﬁed by a wide class of random matrices.
In particular,by combining Theorem 2.2 of [10] with Proposition 2.4 of [21], it sufﬁces that the support of thelimiting spectral measure ρ is (i) compact, (ii) connected, and (iii) it has a proper decay rate at itsedges (i.e., ρ is of Jacobi type), see Assumption 2.1 of [10].
Note that, if ρ is the semicircle law of radius 2 , then the noise matrix Z = W is asymptotically equalin distribution to a standard Wigner matrix with density ∼ exp (− N 4 Tr W 2 ) .
As the elements of thistype of symmetric matrices are i.i.d.
Gaussian, in this case we say that there is Gaussian noise .
Forother limiting ρ , the elements of the matrix Z ≠ W remain correlated and, thus, we say that there isstructured noise .
To distinguish between structured and Gaussian noise, we denote by W a sequenceof matrices asymptotically distributed as standard Wigner matrices, while we reserve Z for a genericsequence of rotationally invariant matrices.
Sources of mismatch.
In this paper, we consider what happens when there is a mismatch betweenthe true noise statistics and the assumptions on the noise statistics made in the inference algorithm.
Inparticular, we study the case in which the noise is assumed to be Gaussian.
Gaussianity is in fact themost standard assumption made when precise knowledge of the noise structure is lacking.
We alsoconsider the case in which the SNR is estimated incorrectly, i.e., the statistician assumes that the datais generated according to Y W = √ λN XX ⊺ + W , where λ ≠ λ ∗ and λ ∗ is the correct SNR present in(1).
Our goal is to quantify how these sources of ignorance affect the algorithmic performance.
2.3 Three classes of inference procedures for PCA
Mismatched Bayes estimator.
The statistician assumes Gaussian noise and that the signal X hasno speciﬁc structure, i.e., it is uniformly distributed on the N -dimensional sphere S N − 1 ( √
N ) .
Themismatched posterior distribution reads, using the Gaussian log-likelihood − N
Tr ( Y − √ λN xx ⊺ ) 2 ,
[Equation]
where Z N ( Y ) is the normalization constant and µ N is the uniform measure over S N − 1 (√ N ) .
The
Bayes estimators we analyze are obtained as the posterior means
[Equation]
which may not be practical to compute.
Notice that, if the noise is a Wigner matrix, then the posterior(2) is correct.
This Bayes-optimal case has already been rigorously characterized (see, for example,[12, 14, 54]), and it will serve us as a base case for comparison with our mismatched setting.
The performance of a sequence of estimators M N ( Y ) ∈ R N × N (simply denoted by M ( Y ) ) of thespike XX ⊺ is quantiﬁed via the asymptotic mean-square error (MSE):
[Equation]
where ∥ ⋅ ∥ 2F denotes the Frobenius norm and E the expectation over the spike XX ⊺ and the noise Z .
We also consider another performance measure which is insensitive to the norm of an estimator ˆ x of X , namely, the rescaled overlap:
[Equation]
where ∥ X ∥ = √ N due to the spherical constraint
For both AMP and spectral estimators, this overlapwill almost surely be deterministic.
If the norm of the estimator vanishes, by convention we ﬁx theoverlap to zero.
Note that the deﬁnition (5) is not meaningful for the Bayes case.
In fact, if one deﬁnesˆ x mis ∶= ∫ x P mis ( d x ∣ Y ) , then this quantity is zero by sign symmetry ( ± x have the same posteriorweight).
Thus, the correct deﬁnition of the overlap for the mismatched Bayes estimator is
[Equation]
while for the “one-shot” AMP and spectral estimators we use the former deﬁnition (5).
Approximate message passing.
We remark that, as opposed to our analysis of the Bayes estimator,the state evolution characterization of AMP (Theorem 2) does not require X to be uniformlydistributed on S N − 1 (√ N ) .
The AMP analysis can accomodate more generic (potentially mismatched)prior distributions and, in fact, AMP algorithms are well equipped to exploit structure in the signalprior.
For simplicity, we assume to have access to an initialization ˆ x 1 ∈ R N , which is independent ofthe noise Z and has a strictly positive correlation with X , i.e.,
[Equation]
where ( X , ˆ x 1 ) W 2 —→ ( X, ˆ x 1 ) denotes convergence of the joint empirical distribution of ( X , ˆ x 1 )to the random variables ( X, ˆ x 1 ) in Wasserstein-2 ( W 2 ) distance.
We note that this initialization isimpractical and one can design AMP iterations which are initialized with the eigenvector v N of thedata matrix Y associated to the largest eigenvalue ν N , see [61, 60, 86].
Then, for t ≥ 1 , the AMPiteration reads
[Equation]
where we assume that ˆ x 0 = 0 .
Here, the function h t + 1 ∶ R → R is applied component-wise; β 1 = 0and, for t ≥ 2 , β t = ⟨ h ′ t ( x t − 1 )⟩ , where h ′ t denotes the ﬁrst derivative of h t .
The AMP estimator of Xis ˆ x t , and the one of the spike XX ⊺ is M t AMP =
ˆ x t ( ˆ x t ) ⊺ .
We refer to this algorithm as Gaussian AMP , since this is the AMP that is implemented for Gaussian noise (and known SNR).
Spectral estimators.
Finally, we consider spectral estimators of the form C v N v ⊺ N , where v N isthe unit-norm eigenvector associated to the largest eigenvalue ν N ( Y ) and C > 0 is a scaling constanttaking into account the spectrum of the data matrix.
The asymptotic description of v N and ν N foradditive and multiplicative low-rank perturbations of rotationally invariant matrices has been obtainedin [21, 22, 6].
In particular, by Theorem 2.2 of [22], the squared overlap ⟨ X , v N ⟩ 2 / N converges a.s.to a constant C ( ρ,λ ∗ ) , whose explicit expression is recalled in Appendix A. We study two variantsof spectral estimators:
● The optimal spectral estimator (OptSpec) is given by M OS ∶= C OS v N v ⊺ N , where C OS ( ρ,λ ∗ ) ∶=C ( ρ,λ ∗ ) is the optimal scaling that minimizes the MSE. ●
The Gaussian mismatched spectral estimator (GauSpec) is given by M GS ∶= C GS v N v ⊺ N , where C GS is the optimal scaling if the noise was Gaussian and the SNR was equal to λ > 0 .
By letting ρ SCbe the standard semi-circle law, we have C GS ( λ ) ∶= C ( ρ SC ,λ )
= 1 − 1 / λ .
For both OptSpec and GauSpec, the corresponding estimator of the signal X is given by ˆ x GS =
ˆ x OS =
ˆ x Spec ∶= √
N v N .
Notice that, while OptSpec requires knowing exactly the statistics of thenoise and the SNR, GauSpec represents the spectral estimator that would be used by a statisticianwho assumes the noise to be Gaussian and the SNR λ (instead of λ ∗ ).
An application of Theorem 2.2in [22] gives that
[Equation]
Furthermore, the respective overlaps coincide and are given by √ C OS .
Note that MSE ( M GS ) =
MSE ( M OS ) + 12 ( C ( ρ,λ ∗ ) − ( 1 − 1 / λ )) 2 .
Thus, as expected, we have MSE ( M OS ) ≤ MSE ( M GS ) .
We also remark that, as the noise matrix is rotationally invariant, both the MSE and the overlap of thespectral estimators do not depend on the signal prior.
Main results: performance of inference algorithms
3.1 Mismatched Bayes estimator
Consider the following functions of λ,λ ∗ > 0 and of the asymptotic spectral noise density ρ :
[Equation]
where 1 ( E ) stands for the indicator function of the event E .
Then, our main result is a closed-formcharacterization of the MSE achieved by the mismatched Bayes estimator in terms of these functions.
Theorem 1 (Performance of mismatched Bayes estimator) .
Consider a spiked model (1) .
Let Assumptions 1 and 2 hold.
Then, the MSE of the mismatched Bayes estimator (3) is given by
[Equation]
As a consequence of the auxiliary lemmas used in the proof of Theorem 1 and presented in Appendix B, we have that N − 2 ∫ P mis ( d x ∣ Y )⟨ X , x ⟩ 2 converges almost surely to M ( λ,λ ∗ ) .
Furthermore,for noise matrices regularized by arbitrarily small Wigner matrices, by using similar techniques as in[15], we can prove that ∥ M mis ∥ F / N converges almost surely to Q 1 / 2 ( λ,λ ∗ ) .
We conjecture that thisconvergence holds even without the Wigner regularization.
By assuming this conjecture, we have thefollowing almost sure convergence result for the overlap:
[Equation]
which holds whenever the denominator is non-zero.
When comparing the overlaps of differentmethods in Section 4, we will use (11) for the mismatched Bayes estimator.
Let us highlight the following property of the MSE (10) when there is no mismatch in the SNR:
If λ = λ ∗ and h ≥ 1 , then MSE ( M mis ) =
MSE ( M GS ) .
However, in general MSE ( M mis ) and MSE ( M GS ) differ.
E.g., when the assumed SNR λ is differentfrom the real one λ ∗ , the norm of M mis incorporates some information about λ ∗ while the one for M GS only depends on λ .
Finally, it is worth noting that, from a statistical view-point, Assumption 1can be replaced by the following: X is uniformly distributed on S N − 1 (√ N ) and is independent of Z ,which may be a generic symmetric matrix with converging empirical spectral density. This includesmost symmetric random matrix distributions.
Sketch of the proof of Theorem 1.
The goal is to evaluate the asymptotic values of E
∥ M mis ∥ 2Fand tr E [ M mis XX ⊺ ] , from which both the MSE and the mean overlap of the Bayes estimator can beobtained.
Our strategy is to ﬁrst compute the log-moment generating function f mis of the mismatched
Bayes model.
Then, the above quantities of interest can be accessed by taking derivatives of f miswith respect to appropriate parameters.
However, as the noise in the inference model is not Gaussian, 244 E ∥ M mis ∥ 2F cannot be computed in the standard way using an “I-MMSE” type of formula [46] (like itis done, e.g., in [71]).
To address this issue, we introduce a generalized model for which the noisematrix is given by the original one plus an independent Wigner matrix multiplied by a small parameter(cid:15) .
We then compute the log-moment generating function of this generalized model using the theoryof low-rank perturbations of rotationally invariant random matrices [21] and of the spherical integral[45].
Finally, the desired results are obtained by a limiting argument in (cid:15) → 0 .
3.2 Approximate message passing
Our main result is a sharp characterization of the iterates of the AMP algorithm (8)in the high-dimensional limit.
In particular, we show that the joint empirical distribution of ( x 1 ,..., x t , ˆ x 1 ,..., ˆ x t + 1 ) converges (in W 2 distance) to the random vector 254 ( x 1 ,...,x t , ˆ x 1 ,..., ˆ x t + 1 ) .
The law of this random vector is captured by a deterministic state evolution (SE) recursion, which can be expressed via a sequence of vectors µ t = ( µ 1 ,...,µ t ) and matricesΣ t , ∆ t , B t ∈ R t × t deﬁned recursively as follows.
We start with the initialization
[Equation]
where λ ∗ is the true SNR (see (1)), and { ¯ κ k } k ≥ 1 are the free cumulants associated to the asymptoticspectral measure of the noise ρ .
For t ≥ 1 , given µ t , Σ t , ∆ t , B t , let( z 1 ,...,z t ) ∼ N( 0 , Σ t ) and independent of ( X, ˆ x 1 ) ,
[Equation]
where we have deﬁned ˆ x 0 = 0 , ¯ β 1 = 0 , and for t ≥ 2 , ¯ β t
= E [ h ′ t ( x t − 1 )] .
Then, the parameter µ t + 1 is
[Equation]
Next, we compute the entries of the matrix ∆ t + 1 as
[Equation]
Furthermore, the matrix B t + 1 is given by
[Equation]
+ 1 , where ( ¯ Φ t + 1 ) i,j = {+
+} if i ≤ j, and ( ¯ Φ t + 1 ) i,j = E [ ∂ j ˆ x i ] if i > j, (17) and
∂
j ˆ x i denotes the partial derivative ∂
z j h i ( z i − 1 + µ i − 1 X − ¯
β
{+i − 1 ˆ x i − 
 +∑ i − 1 k = 
 ( B t ) i − 1 ,k ˆ x k )) .+}
Finally,we deﬁne the covariance matrix Σ t + 1 as
[Equation]
¯
[Equation]
+
[Equation]
( j ) t + 1 , with Θ ( j ) t + 1 =
j ∑ i =
[Equation]
[Equation]
Note that the t × t top left sub-matrices of Σ t + 1 , ∆ t + 1 and B t + 1 are given by Σ t , ∆ t and B t .
At this point, we are ready to state our state evolution characterization of the AMP algorithm (8).
The result is presented in terms of pseudo-Lipschitz test functions.
A function ψ ∶ R m → R ispseudo-Lipschitz of order 2 , i.e., ψ ∈ PL ( 2 ) , if there is a constant C > 0 such that ∥ ψ ( x ) − ψ ( y )∥ ≤
C ( 1 + ∥ x ∥ + ∥
y ∥)∥ x − y ∥ .
We also make the following assumption on the functions { h t + 1 } t ≥ 1 .
Assumption 3.
The function h t + 1 ∶ R
→ R is Lipschitz, and the partial derivatives
[Equation]
are continuous on a set of probability 1 , under the laws of ( z 1 ...,z t ) and ( ˆ x 1 ,..., ˆ x t ) given in (14) .
We note that this is milder than requiring h t
+ 1 to be Lipschitz and continuously differentiable.
Theorem 2 (State evolution of Gaussian AMP) .
Consider a spiked model (1) and the AMP algorithm(8) initialized as in (7) .
Let Assumptions 1 and 3 hold, and let ψ ∶ R 2 t + 2 → R be any pseudo-Lipschitzfunctions of order 2 .
Then, for each t ≥ 1 , we almost surely have that, as N → +∞ ,
[Equation]
where the random variables on the right are deﬁned in (14) .
By using Deﬁnition 6.7 and Theorem 6.8 of [83], one readily obtains that (19) is equivalent to theconvergence of ( x 1 ,..., x t , ˆ x 1 ,..., ˆ x t + 1 , X ) to ( x 1 ,...,x t , ˆ x 1 ,..., ˆ x t + 1 ,X ) in W 2 distance.
Wealso remark that (19) allows for a sharp high-dimensional characterization of the AMP performance:by taking the pseudo-Lipschitz functions ψ ( ˆ x t + 1 ,X ) = ( ˆ x t + 1 − X ) 2 , ψ ( ˆ x t + 1 ,X )
=
ˆ x t + 1 ⋅ X andψ ( ˆ x t
+ 1 ,X )
=
( ˆ x t + 1 )
, we obtain MSE ( M t + 1 AMP ) and Overlap ( ˆ x t + 1 ) .
Sketch of the proof of Theorem 2.
The key insight is to exploit the ﬂexibility given by thedenoisers of the abstract AMP iteration developed in [86] so that the mismatched can be “corrected”.
More formally, by choosing such denoisers carefully, we are able to design an auxiliary AMPalgorithm such that (i) it is of a form which admits a state evolution characterization, and (ii) itsiterates are close to those of the AMP in (8).
This requires a delicate induction argument whosedetails are deferred to Appendix C. In particular, in Appendix C.1, we deﬁne the auxiliary AMP; in
Appendix C.2, we prove a state evolution result for the auxiliary AMP; in Appendix C.3, we show byinduction that (i) the state evolution parameters of the auxiliary AMP coincide with those deﬁnedin this section, and (ii) the iterates of the auxiliary AMP are close (in (cid:96) 2 norm) to the iterates of the AMP (8), thus concluding the proof of Theorem 2.
Numerical results and discussion
In all experiments, the density ρ of the eigenvalues of the noise has unit variance and the signal prioris uniform on the sphere of radius √
N .
For the sake of simplicity, we decouple the effect of themismatch in (i) the noise statistics and (ii) the SNR.
More speciﬁcally, in the ﬁrst two examples, weset λ = λ ∗ , so that only mismatch in the noise statistics is present; and in the last example, the noiseis Gaussian, so that there is only SNR mismatch.
Additional results when Z is the free convolutionof Rademacher and semicircle spectra are reported in Appendix D.3.
Models of mismatch.
(a) Marcenko-Pastur spectrum.
An example of non-symmetric density isthe Marcenko-Pastur law ρ ( x ) = √ x
( 4 − x )/( 2 πx ) 1 ( x ∈ [ 0 , 4 ]) .
The R -transform is R ρ ( x )
=1 /( 1 − x ) , and the results are displayed in Fig.
1a (the same formulas apply if the law is centered).(b)
Uniform spectrum.
We let ρ be the uniform distribution U[−√ 3 , √ 3 ] .
In this case, we have
R ρ ( x ) = √ 3 / tanh (√ 3 x ) − 1 / x , and the results are in Fig.
1b.(c)
Wigner matrix/semicircle spectrum with mismatched SNR.
We consider a Gaussian noise matrix
W (as assumed by the statistician), but with mismatched SNR by setting λ = 4 λ ∗ , see Fig. 1c.
We remark that Assumption 1 is veriﬁed for all models.
Furthermore, in all the three cases, the supportof ρ is a single non-empty interval.
Hence, Theorem 2.2 of [10] and Proposition 2.4 of [21] can beused to show that Assumption 2 holds.
Inference algorithms and set-up.
The estimators of the spike XX ⊺ are compared in terms ofthe MSE (4) (plots on the left).
The AMP and spectral estimators of X are compared in terms ofthe overlap (5), and the Bayes estimator of X in terms of the overlap (6) (plots on the right).
Weevaluate those formulas at N = 8000 for the various algorithms, and in the N → +∞ limit for thetheoretical predictions. The correct AMP (together with its own SE) is in red. This AMP is correct inthe sense that the statistician is aware of the noise statistics and, thus, incorporates the right Onsagercorrections. The form of the correct AMP and the corresponding state evolution are readily obtainedfrom the results in [37, 86] and, for the reader’s convenience, they are recalled in Appendix D.1.
Asnon-linearities, we use the posterior-mean denoising functions of Section 5.1 in [86], and we estimatethe SE parameters consistently from data (see Appendix D.2). The Gaussian AMP (together with itsown SE) is in blue. This is the AMP algorithm (8), which is chosen when the noise statistics or the SNR are unknown (as it would be optimal for Gaussian noise and known SNR).
Its state evolutionis given by Theorem 2. As non-linearities { h t + 1 } t ≥ 1 , we use again the posterior-mean denoisingfunctions, and estimate consistently the state evolution parameters from data (see Appendix D.2). Wenote that the denoisers of the Gaussian AMP depend only on a single iterate, while the denoisers ofthe correct AMP incorporate all the past iterates. In contrast, the state evolution parameters of the
Gaussian AMP at time t depend on all the past, in order to correct for the mismatch. The mismatched Bayes estimator is plotted in green. Its MSE and overlap are given by (10) and (11), respectively.
The optimal spectral estimator (OptSpec) is plotted in black, and the Gaussian mismatched spectralestimator (GauSpec) is plotted in yellow in Fig. 1a and 1c (where it differs from Bayes) or in greenin Fig.
1b (where it coincides with Bayes).
The performance measures of both variants of spectralestimators are given at the end of Section 2.3.
Each experiment is repeated for 10 independent runs.
We report the average and error bars at 1 standard deviation.
A rich and surprising phenomenology.
(i)
An intriguing phenomenon is that, in terms of MSE,the Gaussian AMP does not perform as well as the mismatched Bayes estimator .
This effect occursin all our settings, and it is surprising as the Gaussian AMP (8) is precisely designed to efﬁcientlycompute the Bayes estimator (3).
In the Bayes-optimal case, it does so (outside of its computationalgap) [12].
However, if there is a mismatch, AMP does not compute the (mismatched) posterior mean .
This ﬁnding adds to the already existing evidence [5] that AMP is poorly understood, and studyingthe fundamental reasons behind this behavior is an exciting avenue for future research.(ii)
By comparing MSE 1 and overlap curves, we understand that the discrepancy between Gaussian
AMP and Bayes comes from an incorrect estimation of the signal norm.
In fact, at large enough
SNR, the overlaps of Bayes and Gaussian AMP match, and they even coincide with the overlaps ofoptimal algorithms (correct AMP and OptSpec), which exploit knowledge about the noise structure.
[Figure]
[Figure] [Figure]
[Figure]
Understanding the origin of the MSE discrepancy (namely, a wrong estimate of the signal length) canpotentially lead to procedures which correct for this effect and, hence, reduce the MSE.(iii) When there is no SNR mismatch and ¯ h ≥ 1 (cf.
(12)), the MSEs of the Bayes and Gaussianspectral estimators match .
Both have no information about the noise structure and about the signalprior.
Yet, this equality is remarkable given that the spectral estimator is the solution of an optimizationproblem (it is a “zero temperature” estimator in physics parlance), while the Bayes estimator aims atcomputing the mean of a certain posterior distribution (it is a “ﬁnite temperature” estimator).
(iv) All mismatched estimators are outperformed by the optimal spectral method and the correct AMP .
Both these algorithms take advantage of the noise statistics and achieve the same MSE.
This suggeststhat the two estimators are Bayes-optimal .
We remark that solving this conjecture would requireunderstanding the information-theoretic limits of low-rank estimation with structured noise.(v
Finally, a striking phenomenon – ﬁrst observed in [71] – is that the Bayes and Gaussian spectral
MSE curves may be non-decreasing with the true SNR λ ∗ , see Figs.
1a and 1c.
Going beyond theanalysis in [71], we remark that, for large enough SNR, all estimators yield the same overlap andtherefore all “point in the correct direction” given by the leading eigenvector, see also part (ii) of thisdiscussion.
This links the initial MSE increase to a wrong estimation of the signal’s norm, due to anover-conﬁdence in the data quality (recall that the assumed SNR λ is equal to 4 λ ∗ in Fig. 1c).
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s
contributions and scope?
[Yes]
(b) Did you describe the limitations of your work?
[Yes] Our model assumptions are clearly
explained in Assumptions 1, 2 and 3.
To obtain the overlap of the mismatched Bayes
estimator, we make a conjecture on the convergence of ∣∣ M mis ∣∣ F / N (see the discussion
right after the statement of Theorem 1).
(c) Did you discuss any potential negative societal impacts of your work?
[N/A] We do
not see direct paths to applications with negative societal impacts.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to
them?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes] Our model assumptions are clearly explained in Assumptions 1, 2 and 3.
To obtain the overlap of the mismatched Bayes estimator, we make a conjecture on the convergence of
∣∣ M mis ∣∣ F / N (see the discussion right after the statement of Theorem 1).
(b) Did you include complete proofs of all theoretical results?
[Yes] Proof sketches are
included right after the corresponding statements.
The full proofs of Theorem 1 and 2are in Appendices B and C, respectively.
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi-
mental results (either in the supplemental material or as a URL)?
[Yes]
In Section 4,
we describe the experimental setup containing the instructions to reproduce the results of Figure 1 (and of Figure 2 contained in Appendix {+D.
+} as well).
Additional details are given in Appendices D.1 and D.2.
The code that implements the various algorithms is included in the supplementary material.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
[Yes] All details of the experimental setup are described in Section
and in Appendices D.1 and D.2.
(c) Did you report error bars (e g., with respect to the random seed after running experiments multiple times)?
[Yes] Each experiment is repeated for 10 independent runs, and
we report the average together with the conﬁdence interval at 1 standard deviation.
(d) Did you include the total amount of compute and the type of resources used (e g., typeof GPUs, internal cluster, or cloud provider)?
[N/A]
We did not perform large scale
experiments that required to specify this type of details.
If you are using existing assets (e
g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[N/A] We are not using
existing assets or curating/releasing new assets.
(b) Did you mention the license of the assets?
[N/A]
(c) Did you include any new assets either in the supplemental material or as a URL?
[N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you’re [Table] using/curating?
[N/A]
[Table] (e) Did you discuss whether the data you are using/curating contains personally identiﬁable [Table] information or offensive content?
[N/A]
[Table]
If you used crowdsourcing or conducted research with human subjects...
[Table]
(a) Did you include the full text of instructions given to participants and screenshots, ifapplicable?
[N/A]
We did not use crowdsourcing or conducted research with human
subjects.
[Table]
(b) Did you describe any potential participant risks, with links to Institutional Review
[Table]
Board (IRB) approvals, if applicable?
[N/A]
[Table]
(c) Did you include the estimated hourly wage paid to participants and the total amount
spent on participant compensation?
[N/A]
[Table]