The price of ignorance: how much does it cost to forget noise structure in low-rank matrix estimation?
Abstract
We consider the problem of estimating a rank- 1 signal corrupted by structured rotationally invariant noise, and address the following question: how well do inference algorithms perform when the noise statistics is unknown and hence Gaussian noise is assumed?
While the matched Bayes-optimal setting with unstructured noise is well understood, the analysis of this mismatched problem is only at its premises.
In this paper, we make a step towards understanding the effect of the strong source of mismatch which is the noise statistics.
Our main technical contribution is the rigorous analysis of a Bayes estimator and of an approximate message passing (AMP) algorithm, both of which incorrectly assume a Gaussian setup.
The ﬁrst result exploits the theory of spherical integrals and of low-rank matrix perturbations; the idea behind the second one is to design and analyze an artiﬁcial AMP which, by taking advantage of the ﬂexibility in the denoisers, is able to “correct” the mismatch.
Armed with these sharp asymptotic characterizations, we unveil a rich and often unexpected phenomenology.
For example, despite AMP is in principle designed to efﬁciently compute the Bayes estimator, the former is outperformed by the latter in terms of mean-square error.
We show that this performance gap is due to an incorrect estimation of the signal norm.
In fact, when the SNR is large enough, the overlaps of the AMP and the Bayes estimator coincide, and they even match those of optimal estimators taking into account the structure of the noise.
Introduction
The estimation of a low-rank matrix from noisy data is a central problem in machine learning, and it appears, e.g., in sparse principal component analysis (PCA)
[50, 88], community detection [1, 62], and group synchronization [69].
In this paper, we consider the prototypical task of recovering a symmetric rank- 1 matrix XX ⊺ from noisy observations of the form
[Equation]
Here, λ ∗ > 0 is the signal-to-noise ratio (SNR) which quantiﬁes the signal strength, and Z ∈ R N × N is a random matrix that captures the noise.
A natural estimator of X is given by the principal eigenvector of Y .
Its performance and, more generally, the behavior of the eigenvalues and eigenvectors of (1) has been studied in exquisite detail in statistics [49, 68] and random matrix theory [7, 8, 21, 22, 26, 40, 52, 6].
Going beyond the spectral estimator given by the principal eigenvector, approximate message 36th Conference on Neural Information Processing Systems (NeurIPS 2022).
[Table]
passing (AMP) constitutes a popular family of iterative algorithms.
The reason for this popularity lies in two especially attractive features: (i) AMP algorithms can be tailored to exploit knowledge on the structure of the signal; and (ii) under suitable assumptions, their performance in the high-dimensional limit is accurately tracked by a deterministic recursion known as state evolution [18, 24, 48].
Using the state evolution machinery, it has been showed that AMP achieves Bayes-optimal performance in some Gaussian models [29, 30, 61, 13], and a bold conjecture from statistical physics is that
AMP is optimal in the class of polynomial-time algorithms for a large class of inference problems with random design.
AMP algorithms have been recently considered for several practical scenarios, including genetics [86], inpainting [66] and MRI image recovery [76].
A general class of noise models that has received attention in the literature is given by the family of rotationally invariant matrices .
This is much milder than requiring Z to be Gaussian: it only imposes that the matrix of eigenvectors is uniformly random, and allows for an arbitrary spectrum.
Hence, Z can capture the complex correlation structure which often occurs in applications (e.g., recommender systems [20] and bioinformatics [72]).
However, estimating noise statistics from data is costly or, for problems involving large-scale datasets (computational genomics is a paradigmatic example [36, 2]), may be impossible.
Thus, one natural idea is to simply assume Gaussian statistics for Z .
The case in which Z is actually Gaussian has been thoroughly studied [53, 28, 55, 54, 12, 61].
Beyond Gaussianity, a rapidly growing literature is focusing on rotationally invariant models assuming perfect knowledge of the statistics of the structured matrix appearing in the problem (such as noise in inference, a sensing, data, or coding matrix in regression tasks, weight matrices in neural networks, or a matrix of interactions in spin glass models)
However, despite this impressive progress when the noise statistics is known, low-rank estimation in a mismatched setting with partial to no knowledge of the statistics of the rotationally invariant noise matrix remains poorly understood .
The goal of this work is thus to shed light into the following fundamental question:
Suppose that the noise statistics is unknown or unreacheable, and hence Gaussian noise is naively assumed.
What is the impact of this mismatch on the overall performance of inference methods?
1.1 Summary of contributions
In this paper, we provide rigorous performance guarantees for two inference methods which incorrectly assume Gaussian noise statistics: a Bayes estimator, and an AMP algorithm.
Then, by exploiting these sharp analytical characterizations, we describe a number of surprising effects coming from numerical simulations.
Our main ﬁndings are detailed below.
Theoretical results.
(i)
We give a closed-form expression for the mean-square error (MSE) of the
Bayes estimator which samples from the mismatched posterior (cf.
Theorem 1).
Under a certain concentration assumption, we also present an asymptotic result on the overlap of such estimator.
(ii)
We provide a state evolution analysis (cf.
Theorem 2) for the Gaussian AMP algorithm which is designed for Gaussian noise.
Numerical results.
The mismatched Bayes and AMP estimators display surprising behaviors – already for the case of a spherically distributed signal – and the two performance metrics (MSE and overlap) exhibit a remarkably different phenomenology: (i) As for the MSE, the Gaussian AMP is outperformed by the Bayes estimator.
Here, the surprise comes from the fact that AMP algorithms are, in principle, designed to sample from the posterior distribution (and this is often what happens in the matched case).
(ii)
In contrast, when the SNR is large enough, the overlaps of the two mismatched estimators coincide , and they even match the overlap of estimators which exploit the noise statistics, namely the optimal spectral method which minimizes the MSE and the correct AMP designed in [37].
(iii) Under certain conditions, the MSE of the mismatched Bayes estimator matches that of a
Gaussian spectral method with no information on the noise structure.
(iv) The mismatched estimators are outperformed – in terms of MSE – by the optimal spectral method and the correct AMP, whose performance coincide.
(v) Finally, the MSE curves of the Bayes and Gaussian spectral methods exhibit a striking non-monotone behavior.
1.2 Related work
The impact of mismatch.
Given the practical relevance of understanding the effect of mismatch in statistical inference, a line of work has approached the issue from various angles.
Regression being probably the most paradigmatic model of inference task, mismatch has been thoroughly studied in this context.
In information theory, the trend was initiated in [83, 85].
Studies for M-estimation and robust statistics followed based on statistical mechanics approaches [3], on the analysis of approximate message passing algorithms [25, 77, 32, 43], on Gordon’s convex min-max theorem [81, 80] and the leave-one-out method [35, 34].
Concerning the analysis of Bayesian approaches to mismatched linear regression, we refer to [27, 63, 75, 11] or [9] for a review.
In contrast, the problem studied in the present paper, namely low-rank matrix inference with mismatch, has received attention only recently.
It is clear that in some way, related questions were analyzed in the aforementioned random matrix theory literature, although from a rather different perspective.
Concerning the precise issue of the performance degradation due to mismatch in Bayesian inference, an exception (and inspiration for our work) is the recent paper [71].
However, the authors of [71] focus on mismatch in the signal-to-noise ratio for the Gaussian noise setting; their results can thus be recovered as a special case of the present ones.
To the best of our knowledge, this is the ﬁrst work that considers mismatch in the noise statistics for low-rank matrix estimation.
Approximate message passing.
AMP algorithms have been applied to a wide range of inference problems.
Examples include estimation in linear models [19, 18, 31], generalized linear models [73, 13, 58, 59], and low-rank matrix recovery with Gaussian noise [17, 29, 41, 51, 55, 61], see also the survey [39].
A general AMP iteration for rotationally invariant matrices has been recently analyzed in [37, 87], and by providing suitable instances of this abstract iteration, AMP algorithms have been developed for low-rank [37, 87, 60] and generalized linear models [82].
Furthermore, an AMP-based method which uses the classical idea of empirical Bayes to reduce the high-dimensional noise in
PCA is proposed in [86], which also provides applications to genetics.
However, the existing results cannot be applied to the mismatched setting considered in this work, since the Gaussian AMP does not contain the right Onsager corrections.
In fact, the algorithm designer assumes Gaussian statistics for the noise and, therefore, constructs an AMP algorithm with the Onsager correction suitable for Gaussian noise.
Finally, we remark that the performance of the Gaussian AMP is numerically compared with that of the correct AMP (exploiting the knowledge of noise statistics) in [87].
Our
Theorem 2 provides rigorous foundations for such a comparison.
Setup of the problem 2.1 Random matrix theory preliminaries
We start with some useful notions of random matrix theory.
Given a probability measure ρ of compact support K ⊆ R , we let H ρ ∶ R / K ↦ R be the Hilbert transform of ρ : H ρ ( z ) ∶= ∫ K ρ ( dγ )( z − γ ) − 1 .
We deﬁne γ ∶= max K , γ ∶= min K , h ∶= lim
z ↓ γ H ρ ( z ) and h ∶= lim z ↑ γ H ρ ( z ) , where the limits exist due to the monotonicity of H ρ
but may be inﬁnite.
As H ρ is a bijection between R / K and its image ( h,h )/{ 0 } , its inverse exists and it is denoted by K ρ ∶ ( h,h )/{ 0 } ↦ R / K .
The R -transform of ρ is R ρ ∶ ( h,h )/{ 0 } ↦ R / K given by R ρ ( x )
∶= K ρ ( x ) − 1 / x .
The coefﬁcients { ¯ κ k } k ≥ 1 of the
Taylor series of R ρ are the free cumulants associated to ρ , i.e., R ρ ( x ) =
∑ ∞ k =
¯ κ k + 1 x k , and they can be computed from the moments of ρ , see e.g. Section 2.5 of [64].
Furthermore, R ′
ρ ( x ) and H ′
ρ ( z ) denote the derivatives of the R -transform and the Hilbert transform of ρ , respectively.
2.2 Model of mismatched low-rank matrix estimation
We consider the problem of estimating a rank-one informative spike XX ⊺ corrupted by an additive symmetric noise matrix Z ∈ R N × N from data Y generated as in (1).
The scalar λ ∗ ∈ R ≥is the signal-to-noise ratio (SNR), and the noise Z can be decomposed as Z = O Σ O ⊺ , where Σ ∶= diag ( γ 1 ,...,γ N ) ∈ R N × N is a diagonal matrix containing the eigenvalues of Z and O is some orthogonal matrix.
We also denote by γ N and γ N respectively max { γ 1 ,...,γ N } and min { γ 1 ,...,γ N } .
Similarly, ν N and ν N denote the largest and smallest eigenvalues of the data matrix Y .
Throughout the paper, the main technical assumption will be the following.
Assumption 1.
The signal X has norm √
N .
The random noise matrix Z is rotationally invariant and is independent of X .
Moreover, the empirical measure N − 1 ∑ i ≤ N δ γ i of eigenvalues of Z converges weakly towards a limiting measure ρ of compact support K ⊆ R .
Finally, γ N and γ N converge a.s. to γ = max K and γ = min K , respectively.
The noise matrix Z being rotationally invariant means that O is a random orthogonal matrix (i.e., sampled from the Haar measure) independent of Σ .
Under Assumption 1, we have that ν N and ν
N converge a.s. to ﬁnite limits, which we denote by ν and ν , respectively.
These have an explicit form in terms of ρ and λ ∗ given in [21, Theorem 2.1] and which we reproduce for convenience in Appendix A.
We remark that the assumption on the ground-truth signal is rather mild: X can have any distribution over the sphere of radius √
N , and it might even be deterministic.
For example, X ∈ {− 1 , 1 } N could be a uniform vector over the centred N -dimensional hyper-cube of sides√
N .
For the analysis of the Bayes estimator, we require a second assumption on the asymptotic eigenvalue distribution ρ .
This ensures that the limit for the overlap of the spectral estimators with the signal is continuous in the SNR, whenever a small additive Wigner noise of variance (cid:15) ≥ 0 is added to Y .
Assumption 2.
Let ρ (cid:15) be the spectral density of the free convolution of ρ and a semicircle law of radius 2 (cid:15) ≥ 0 .
Let H ρ (cid:15) be the associated Hilbert transform and γ (cid:15) the rightmost point of the support of ρ (cid:15) .
Then, we assume that lim z ↓ γ (cid:15) H ′
ρ (cid:15) ( z ) =
−∞ for all (cid:15) ≥ 0 .
For the deﬁnition of the free convolution and its link to random matrix theory, we refer the reader to [4, 70].
We remark that Assumption 2 is satisﬁed by a wide class of random matrices.
In particular, by combining Theorem 2.2 of [10] with Proposition 2.4 of [21], it sufﬁces that the support of the limiting spectral measure ρ is (i) compact, (ii) connected, and (iii) it has a proper decay rate at its edges (i.e., ρ is of Jacobi type), see Assumption 2.1 of [10].
These conditions can be easily checked for many random matrix models, including the ones discussed in the examples of Section 4.
Note that, if ρ is the semicircle law of radius 2 , the noise is asymptotically equal in distribution to a standard Wigner matrix W with density ∼ exp (− N 4 Tr W 2 ) .
As the elements of this type of symmetric matrices are i.i.d.
Gaussian, in this case we say that there is Gaussian noise .
For other limiting ρ , the elements of Z ≠ W remain correlated and we say that there is structured noise .
We denote by W a sequence of standard Wigner matrices while we reserve Z for a generic sequence of rotationally invariant matrices.
Sources of mismatch.
In this paper, we consider what happens when there is a mismatch between the true noise statistics and the assumptions on the noise statistics made in the inference algorithm.
In particular, we study the case in which the noise is assumed to be Gaussian.
Gaussianity is in fact the most standard assumption made when precise knowledge of the noise structure is lacking.
We also consider the case in which the SNR is estimated incorrectly, i.e., the statistician assumes that the data is generated according to ( √ λ / N ) XX ⊺ + W , where λ ≠ λ ∗ and λ ∗ is the correct SNR present in (1).
Our goal is to quantify how these sources of ignorance affect the algorithmic performance.
2.3 Three classes of inference procedures for PCA
Mismatched Bayes estimator.
The statistician assumes Gaussian noise and that the signal X has no speciﬁc structure, i.e., it is uniformly distributed on the N -dimensional sphere S N − 1 ( √
N ) .
The mismatched posterior distribution reads, using the Gaussian log-likelihood − N
Tr ( Y − √ λN xx ⊺ ) 2 ,
[Equation]
where Z N ( Y ) is the normalization constant and µ N is the uniform measure over S N − 1 (√ N ) .
In case the signal X lies on the sphere (by Assumption 1) but is not uniformly distributed on it, then this leads to a third source of mismatch.
However, the MSE formula of Theorem 1 does not depend on it; e.g. if X ∈ {− 1 , 1 } N the error remains the same.
The origin of this is the uniformly spherical prior used in the posterior (2) which is the most uninformative one.
The Bayes estimators we analyze are obtained as the posterior means
[Equation]
which may not be practical to compute.
Notice that, if the noise is Wigner, then the likelihood exp (⋯) in (2) is correct.
This Bayes-optimal case has already been rigorously characterized (see, for example, [12, 14, 54]), and it will serve us as a base case for comparison with our mismatched setting.
The performance of a sequence of estimators M N ( Y ) ∈ R N × N of the spike XX ⊺ is quantiﬁed via the asymptotic mean-square error (MSE):
[Equation]
where ∥ ⋅ ∥ 2F denotes the Frobenius norm and E the expectation over the spike XX ⊺ and the noise
Z .
We also consider another performance measure which is insensitive to the norm of an estimator ˆ x of X , namely, the rescaled overlap:
[Equation]
where ∥ X ∥ = √ N due to the spherical constraint
This alignment measure is sometimes also referred to as cosine distance .
For both AMP and spectral estimators, this overlap will almost surely be deterministic.
If the norm of the estimator vanishes, by convention we ﬁx the overlap to zero.
Note that the deﬁnition (5) is not meaningful for the Bayes case.
In fact, if one deﬁnes ˆ x mis ∶= ∫ x P mis ( d x ∣ Y ) , then this quantity is zero by sign symmetry ( ± x have the same posterior weight).
Thus, the correct deﬁnition of the overlap for the mismatched Bayes estimator is
[Equation]
while for the “one-shot” AMP and spectral estimators we use the former deﬁnition (5).
Approximate message passing.
We remark that, as opposed to our analysis of the Bayes estimator, the state evolution characterization of AMP (Theorem 2) does not require X to be uniformly distributed on S N − 1 (√ N ) .
The AMP analysis can accomodate more generic (potentially mismatched) prior distributions and, in fact, AMP algorithms are well equipped to exploit structure in the signal.
For simplicity, we assume to have access to an initialization ˆ x 1 ∈ R N , which is independent of the noise Z and has a strictly positive correlation with X , i.e.,
[Equation]
where ( X , ˆ x 1 ) W 2 —→ ( X, ˆ x 1 ) denotes convergence of the joint empirical distribution of ( X , ˆ x 1 ) to the random variable ( X, ˆ x 1 ) in Wasserstein-2 ( W 2 ) distance.
Then, for t ≥ 1 the AMP iteration reads
[Equation]
where we assume that ˆ x 0 = 0 .
Here, the function h t + 1 ∶ R → R is applied component-wise, and it can be chosen in order to exploit prior information about the signal; β 1 = 0 and, for t ≥ 2 , β t = ⟨ h ′ t ( x t − 1 )⟩ , where h ′ t denotes the derivative of h t .
The AMP estimator of X is ˆ x t , and the one of the spike XX ⊺ is M t AMP =
ˆ x t ( ˆ x t ) ⊺ .
We refer to this algorithm as Gaussian AMP , since this is the AMP that is implemented for Gaussian noise (and known SNR).
For a discussion on how (8) can be derived, we refer the interested reader to the review [39].
We note that the initialization (7) is impractical and one can design AMP iterations which are initialized with the eigenvector v N of the data matrix Y associated to the largest eigenvalue ν N , see [61, 60, 87].
We are also able to provide a state evolution result for the Gaussian AMP with spectral initialization, and this is discussed in Appendix C.4.
Spectral estimators.
Finally, we consider spectral estimators of the form C v N v ⊺ N , where v N is the unit-norm eigenvector associated to the largest eigenvalue ν N ( Y ) and C > 0 is a scaling constant taking into account the spectrum of the data matrix.
The asymptotic description of v N and ν N for additive and multiplicative low-rank perturbations of rotationally invariant matrices has been obtained in [21, 22, 6].
In particular, by Theorem 2.2 of [22], the squared overlap ⟨ X , v N ⟩ 2 / N converges a.s. to a constant C ( ρ,λ ∗ ) , whose explicit expression is recalled in Appendix A. We study two variants:
● The optimal spectral estimator (OptSpec) is given by M OS ∶= C OS v N v ⊺ N , where C OS ( ρ,λ ∗ ) ∶= C ( ρ,λ ∗ ) is the optimal scaling that minimizes the MSE.
● The Gaussian mismatched spectral estimator (GauSpec) is given by M GS ∶= C GS v N v ⊺ N , where C GS is the optimal scaling if the noise was Gaussian and the SNR was equal to λ > 0 .
By letting ρ SC be the standard semi-circle law, we have C GS ( λ ) ∶= C ( ρ SC ,λ )
= 1 − 1 / λ .
For both OptSpec and GauSpec, the estimator of X is given by ˆ x GS =
ˆ x OS
=
ˆ x Spec ∶= √
N v N .
Notice that, while OptSpec requires knowing exactly the statistics of the noise and the SNR, GauSpec represents the spectral estimator that would be used by a statistician who assumes the noise to be
Gaussian and the SNR λ (instead of λ ∗ ).
An application of Theorem 2.2 in [22] gives that
MSE ( M OS ) = 1 2 ( 1 − C 2 ( ρ,λ ∗ )) , and MSE ( M GS )
= 1 2 ( 1 + ( 1 − 1 / λ ) 2 − 2 ( 1 − 1 / λ ) C ( ρ,λ ∗ )) .
Furthermore, the respective overlaps coincide and are given by √ C OS .
Note that MSE ( M GS ) =
MSE ( M OS ) + 12 ( C ( ρ,λ ∗ ) − ( 1 − 1 / λ )) 2 .
Thus, as expected, we have MSE ( M OS ) ≤ MSE ( M GS ) .
We also remark that, as the noise matrix is rotationally invariant, both the MSE and the overlap of the spectral estimators do not depend on the ground-truth signal distribution.
Main results: performance of inference algorithms 3.1 Mismatched Bayes estimator
Consider the following functions of λ,λ ∗ > 0 and of the asymptotic spectral noise density ρ :
[Equation]
where 1 ( E ) stands for the indicator function of the event E .
The idea is that Q ( λ,λ ∗ ) and M ( λ,λ ∗ ) represent the squared norm of the mismatched Bayes estimator and its inner product with the signal, respectively.
This leads to our main result on the mismatched Bayes MSE, which is stated below.
Theorem 1 (Performance of mismatched Bayes estimator) .
Consider a spiked model (1) .
Let Assumptions 1 and 2 hold.
Then, the MSE of the mismatched Bayes estimator (3) is given by
[Equation]
As a consequence of the auxiliary lemmas used in the proof of Theorem 1 and presented in Appendix B, we have that N − 2 ∫ P mis ( d x ∣ Y )⟨ X , x ⟩ 2 converges almost surely to M ( λ,λ ∗ ) .
Furthermore, for noise matrices regularized by arbitrarily small Wigner matrices, by using similar techniques as in [15] we can prove that ∥ M mis ∥ F / N converges almost surely to Q 1 / 2 ( λ,λ ∗ ) .
We conjecture that this convergence holds even without the Wigner regularization.
By assuming this conjecture, we have the following almost sure convergence result for the overlap:
[Equation]
which holds whenever the denominator is non-zero.
When comparing the overlaps of different methods in Section 4, we will use (11) for the mismatched Bayes estimator.
Let us highlight the following property of the MSE (10) when there is no mismatch in the SNR:
[Equation]
However, in general MSE ( M mis ) and MSE ( M GS ) differ.
E.g., when the assumed SNR λ is different from the real one λ ∗ , the norm of M mis incorporates information about λ ∗ while the one for M GS only depends on λ .
Finally, we note that, from a statistical view-point, Assumption 1 can be replaced by the following: X is uniformly distributed on S N − 1 (√ N ) and is independent of Z , which may be a generic symmetric matrix with converging empirical spectral density.
Sketch of the proof of Theorem 1.
The goal is to evaluate the asymptotic values of E
∥ M mis ∥ 2F and tr E [ M mis XX ⊺ ] , from which both the MSE and the mean overlap of the Bayes estimator can be obtained.
Our strategy is to ﬁrst compute the log-moment generating function f mis of the mismatched
Bayes model.
Then, the above quantities of interest can be accessed by taking derivatives of f mis with respect to appropriate parameters.
However, as the noise in the inference model is not Gaussian, E ∥ M mis ∥ 2 F cannot be computed in the standard way using an “I-MMSE” type of formula [46] (like it is done, e.g., in [71]).
To address this issue, we introduce a generalized model for which the noise matrix is given by the original one plus an independent Wigner matrix multiplied by a small parameter (cid:15) .
We then compute the log-moment generating function of this generalized model using the theory of low-rank perturbations of rotationally invariant random matrices [21] and of the low-rank spherical integral [45].
Finally, the desired results are obtained by a limiting argument in (cid:15) → 0 .
3.2 Approximate message passing
Our main result is a characterization of the iterates (8) in the high-dimensional limit.
We show that the joint empirical distribution of ( x 1 ,..., x t , ˆ x 1 ,..., ˆ x t + 1 ) converges (in W 2 ) to the random vector ( x 1 ,...,x t , ˆ x 1 ,..., ˆ x t + 1 ) .
The law of the random vector ( x 1 ,...,x t , ˆ x 1 ,..., ˆ x t + 1 ) is captured by a deterministic state evolution (SE) recursion, which can be expressed via a sequence of vectors µ t = ( µ 1 ,...,µ t ) and matrices Σ t , ∆ t , B t ∈ R t × t deﬁned as follows.
We start with the initialization
[Equation]
where λ ∗ is the true SNR (see (1)), and { ¯ κ k } k ≥ 1 are the free cumulants associated to the asymptotic spectral measure of the noise ρ .
For t ≥ 1 , given µ t , Σ t , ∆ t , B t , let {( z 1 ,...,z t ) ∼ N( 0 , Σ t ) and independent of ( X, ˆ x 1 ) ,
[Equation]
where we have deﬁned ˆ x 0 = 0 , ¯ β 1 = 0 , and for t ≥ 2 , ¯ β t
= E [ h ′ t ( x t − 1 )] .
Then, the parameter µ t + 1 is
[Equation]
Next, we compute the matrices ∆ t + 1 and B t + 1 as
[Equation]
where ( ¯ Φ t + 1 ) i,j = 0 if i ≤ j
[Equation]
( ¯
[Equation]
i
[Equation]
=
E [ ∂ j ˆ x i ] if
[Equation]
16) where ∂ j ˆ x i denotes the partial derivative ∂
z j h i ( z i − 1 + µ i − 1 X − ¯
β
i − 1 ˆ x
i −
+ ∑ i − 1 k =
( B t ) i − 1 ,k ˆ x k )) .
Finally, we deﬁne the covariance matrix Σ t + 1 as
[Equation]
Note that the t × t top left sub-matrices of Σ t + 1 , ∆ t + 1 and B t + 1 are given by Σ t , ∆ t and B t .
At this point, we are ready to state our state evolution characterization of the AMP algorithm (8).
The result is presented in terms of pseudo-Lipschitz test functions.
A function ψ ∶ R m → R is pseudo-Lipschitz of order 2 , i.e., ψ ∈ PL ( 2 ) , if there is a constant C > 0 such that ∥ ψ ( x ) − ψ ( y )∥ ≤
C ( 1 + ∥ x ∥ + ∥
y ∥)∥ x − y ∥ .
We also make the following assumption on the functions { h t + 1 } t ≥ 1 .
Assumption 3.
The function h t + 1 ∶ R
→ R is Lipschitz, and the partial derivatives
[Equation]
are continuous on a set of probability 1 , under the laws of ( z 1 ...,z t ) and ( ˆ x 1 ,..., ˆ x t ) given in (14) .
This requirement covers most practically relevant choices of h t
+ 1 (e.g., soft-thresholding or ReLU), and it is rather standard in the AMP literature, see e.g. [39, 60, 82, 87].
Theorem 2 (State evolution of Gaussian AMP) .
Consider a spiked model (1) and
the AMP algorithm
(8) initialized as in (7) .
Let Assumptions 1 and 3 hold, and let ψ ∶ R 2 t + 2 → R be any pseudo-Lipschitz functions of order 2 .
Then, for each t ≥ 1 , we almost surely have that, as N → +∞ ,
[Equation]
X
[Equation] [Equation]
18) where the random variables on the right are deﬁned in (14) .
By using Deﬁnition 6.7 and Theorem 6.8 of [84], one readily obtains that (18) is equivalent to the convergence of ( x 1 ,..., x t , ˆ x 1 ,..., ˆ x t
+ 1 , X ) to ( x 1 ,...,x t , ˆ x 1 ,..., ˆ x t
+ 1 ,X ) in W 2 distance.
Applying (18) to the pseudo-Lipschitz functions ψ ( ˆ x t + 1 ,X )
=
( ˆ x t + 1 − X ) 2 , ψ ( ˆ x t
+ 1 ,X )
=
ˆ x t + 1 ⋅ X and ψ ( ˆ x t + 1 ,X )
=
( ˆ x t + 1 )
, we obtain a high-dimensional characterization of the AMP performance.
Corollary 1.
Consider the setting of Theorem 2.
Then, for each t ≥ 1 , we almost surely have that
[Equation]
t ⋅ X ]) 2 + ( E [( ˆ x t ) 2 ]) 2 ) ,
Overlap ( ˆ x t ) ∶= lim N →+∞
[Equation]
E
[Equation]
Sketch of the proof of Theorem 2.
The key insight is to exploit the ﬂexibility given by the denoisers of the abstract AMP iteration [87] to “correct” the mismatch.
These denoisers are denoted by { ˜ h t } t ≥ 2 (see (33) and (34) in Appendix C.1) and they serve solely as a proof technique, hence they do not have an impact on the practicality of the algorithm; in contrast, the denoisers { h t } t ≥ 2 (see (8)) can be chosen by the algorithm designer to take advantage of the information available about the signal.
Thus, we can design an auxiliary AMP such that (i) it admits a state evolution characterization, and (ii) its iterates are close to (8).
This requires the delicate induction argument in Appendix C: in
Appendix C.1, we deﬁne the auxiliary AMP; in Appendix C.2, we prove a state evolution result for it; in Appendix C.3, we show by induction that (i) the state evolution parameters of the auxiliary AMP coincide with those deﬁned in this section, and (ii) the iterates of the auxiliary AMP are close (in (cid:96)norm) to the iterates of the AMP (8), thus concluding the proof of Theorem 2.
Numerical results and discussion
In all experiments, the density ρ of the eigenvalues of the noise has unit variance and the distribution of the ground-truth signal is uniform on the sphere of radius √
N .
For the sake of simplicity, we decouple the effect of the mismatch in (i) the noise statistics and (ii) the SNR.
More speciﬁcally, in the ﬁrst two examples, we set λ
= λ ∗ , so that only mismatch in the noise statistics is present; and in the last example, the noise is Gaussian, so that there is only SNR mismatch.
Additional results when
Z is the free convolution of Rademacher and semicircle spectra are reported in Appendix D.3.
Models of mismatch.
(a) Marcenko-Pastur spectrum.
An example of non-symmetric density is the Marcenko-Pastur law ρ ( x ) = √
[Equation]
x )/( 2 πx ) 1 ( x ∈ [ 0 , 4 ]) .
The R -transform is R ρ ( x )
= 1 /( 1 − x ) , and the results are displayed in Fig.
1a (the same formulas apply if the law is centered).
(b) Uniform spectrum.
We let ρ be the uniform distribution U[−√ 3 , √ 3 ] .
In this case, we have R ρ ( x ) = √ 3 / tanh (√ 3 x ) − 1 / x , and the results are in Fig.
1b.
(c) Wigner matrix/semicircle spectrum with mismatched SNR.
We consider a Gaussian noise matrix W (as assumed by the statistician), but with mismatched SNR by setting λ = 4 λ ∗ , see Fig. 1c.
We remark that Assumption 1 is veriﬁed for all models.
Furthermore, in all the three cases, the support of ρ is a single non-empty interval.
Hence, Theorem 2.2 of [10] and Proposition 2.4 of [21] can be used to show that Assumption 2 holds.
Inference algorithms and set-up.
The estimators of the spike XX ⊺ are compared in terms of the MSE (4) (plots on the left).
The AMP and spectral estimators of X are compared in terms of the overlap (5), and the Bayes estimator of X in terms of the overlap (6) (plots on the right).
We evaluate those formulas at N = 8000 for the various algorithms, and in the N → +∞ limit for the theoretical predictions.
The correct AMP (together with its own SE) is in red.
This AMP is correct in the sense that the statistician is aware of the noise statistics and, thus, incorporates the right Onsager corrections.
The form of the correct AMP and the corresponding state evolution are readily obtained from the results in [37, 87] and, for the reader’s convenience, they are recalled in Appendix D.1.
As non-linearities, we use the posterior-mean denoising functions of Section 5.1 in [87], and we estimate the SE parameters consistently from data (see Appendix D.2).
The Gaussian AMP (together with its own SE) is in blue.
This is the AMP algorithm (8), which is chosen when the noise statistics or the
SNR are unknown (as it would be optimal for Gaussian noise and known SNR).
Its state evolution is given by Theorem 2.
As non-linearities { h t + 1 } t ≥ 1 , we use again the posterior-mean denoising functions, and estimate consistently the state evolution parameters from data (see Appendix D.2).
We note that the denoisers of the Gaussian AMP depend only on a single iterate, while the denoisers of the correct AMP incorporate all the past iterates.
In contrast, the state evolution parameters of the
Gaussian AMP at time t depend on all the past, in order to correct for the mismatch.
The mismatched
Bayes estimator is plotted in green.
Its MSE and overlap are given by (10) and (11), respectively.
The optimal spectral estimator (OptSpec) is plotted in black, and the Gaussian mismatched spectral estimator (GauSpec) is plotted in yellow in Fig.
1a and 1c (where it differs from Bayes) or in green in Fig.
1b (where it coincides with Bayes).
The performance measures of both variants of spectral estimators are given at the end of Section 2.3.
Each experiment is repeated for 10 independent runs.
We report the average and error bars at 1 standard deviation.
A rich and surprising phenomenology.
(i)
An intriguing phenomenon is that, in terms of MSE, the Gaussian AMP does not perform as well as the mismatched Bayes estimator .
This effect occurs
[Figure] [Figure]
[Figure]
in all our settings, and it is surprising as the Gaussian AMP (8) is precisely designed to efﬁciently compute the Bayes estimator (3).
In the Bayes-optimal case, it does so (outside of its computational gap) [12].
However, if there is a mismatch, AMP does not compute the (mismatched) posterior mean .
This ﬁnding adds to the already existing evidence [5] that AMP is poorly understood
[Figure]
and studying the fundamental reasons behind this behavior is an exciting avenue for future research.
(ii)
By comparing MSE 2 and overlap curves, we understand that the discrepancy between Gaussian
AMP and Bayes comes from an incorrect estimation of the signal norm.
In fact, at large enough
SNR, the overlaps of Bayes and Gaussian AMP match, and they even coincide with the overlaps of optimal algorithms (correct AMP and OptSpec), which exploit knowledge about the noise structure.
Understanding the origin of the MSE discrepancy (namely, a wrong estimate of the signal length) can potentially lead to procedures which correct for this effect and, hence, reduce the MSE.
(iii) When there is no SNR mismatch and ¯ h ≥ 1 (cf.
(12)), the MSEs of the Bayes and Gaussian spectral estimators match (see Fig. 1b).
Both have no information about the noise structure and about the signal distribution.
Yet, this equality is remarkable given that the spectral estimator is the solution of an optimization problem (it is a “zero temperature” estimator in physics parlance), while the Bayes
estimator aims at computing the mean of a certain posterior distribution (it is a “ﬁnite temperature” estimator).
(iv) All mismatched estimators are outperformed by the optimal spectral method and the correct AMP .
Both these algorithms take advantage of the noise statistics and achieve the same MSE.
This suggests that the two estimators are Bayes-optimal .
We remark that solving this conjecture would require understanding the information-theoretic limits of low-rank estimation with structured noise.
(v) Finally, a striking phenomenon – ﬁrst observed in [71] – is that the Bayes and Gaussian spectral MSE curves may be non-decreasing with the true SNR λ ∗ , see Figs.
1a
and 1c.
Going beyond the analysis in [71], we remark that, for large enough SNR, all estimators yield the same overlap and therefore all
“point
in the correct direction” given by the leading eigenvector, see also part (ii) of this discussion.
This
links
the
initial
MSE
increase to a wrong
estimation of the signal’s norm, due to an over-conﬁdence in the data quality (recall that the assumed SNR λ is equal to 4 λ ∗ in Fig. 1c).
Acknowledgements
M. Mondelli was partially supported by the 2019 Lopez-Loreta Prize.
The authors acknowledge discussions with A. Krajenbrink, M. Robinson, A. Depope, N. Macris and F. Pourkamali.
[-For this proof we can set (cid:15) = 
-]
[-Choose λ large enough so that we simultaneously have λλ ∗ >and ( 1 − 1 /√ λλ ∗ ) 
-]
[-= 
-]
[-h 
-]
[-Here, h t + 1 is the denoiser of the true AMP (see (8)), and ( µ t , ¯ β t , B t ) come from the state evolution recursion of the true AMP: µ 1 is given by (13), and, for t ≥ 2 , µ t is given by (15); ¯ β 1 = 
-]
[-By comparing the deﬁnition of ˆ x s in (14) with the deﬁnition of ˜ x s in (37) and by using the choice of ˜ h s in (34) (for s ∈ { 2 ,...,t + 1 } ), we readily obtain that ∆ t + 
-]