Misspeciﬁed Phase Retrieval with Generative Priors
Abstract
In this paper, we study phase retrieval under model misspeciﬁcation and generative priors.
In particular, we aim to estimate an n -dimensional signal x from m i.i.d.
realizations of the single index model y =
f ( a T x ) , where f is an unknown and possibly random nonlinear link function and a ∈ R n is a standard Gaussian vector.
We make the assumption Cov[ y, ( a T x ) 2 ] (cid:54) = 0 , which corresponds to the misspeciﬁed phase retrieval problem.
In addition, the underlying signal x is assumed to lie in the range of an L -Lipschitz continuous generative model with bounded kdimensional inputs.
We propose a two-step approach, for which the ﬁrst step plays the role of spectral initialization and the second step reﬁnes the estimated vector produced by the ﬁrst step iteratively.
We show that both steps enjoy a statistical rate of order (cid:112) ( k log L ) · (log m ) /m under suitable conditions.
Experiments on image datasets are performed to demonstrate that our approach performs on par with or even signiﬁcantly outperforms several competing methods.
Introduction
Compressed sensing (CS) is perhaps the most popular instance of high-dimensional inverse problems, for which one has the linear measurement model
[Equation]
where a ∈ R n is the sensing vector, x ∈ R n is the sparse signal to estimate and η represents additive noise.
It has been well-known for CS that roughly m =
O ( s log( n/s ))
i.i d. Gaussian measurements are sufﬁcient to ensure the accurate recovery of a signal with s non-zero entries [88, 1, 23, 11, 77].
Phase retrieval (PR) arises in numerous scientiﬁc areas including X-ray crystallography, acoustics, astronomy, microscopy, optics, wireless communications, and quantum information [13], where one cannot measure a T x directly, and can only record its magnitude.
For example, the following noisy magnitude-only measurement model has been adopted in various prior works on sparse (real-valued)
PR [92, 39, 25, 36, 9, 7]:
[Equation]
where the signal x is assumed to be sparse, and the sensing vector a is assumed to be a standard
Gaussian vector.
However, both the linear and magnitude-only measure models in (1) and (2) are idealized views of the data generating process.
To make the setup more general, one can utilize the following semi-parametric single index model (SIM) for general nonlinear models:
[Equation]
36th Systems (NeurIPS 2022). where f : R → R is an unknown (possibly random) nonlinear link function, and a is typically assumed to be Gaussian.
In addition, since the norm of x ∈ R n is absorbed into the SIM, for brevity, it is common to assume that x is a unit vector, i.e., (cid:107) x (cid:107) 2 = 1 .
SIMs have long been studied in the conventional setting where the number of measurements m > n [30, 81, 45].
In recent years, they have also been analyzed in the high-dimensional setting where m (cid:28) n , mainly under the assumption that the underlying signal is sparse.
Relevant works include but are not limited to [72, 63, 26, 73, 68].
For all these works, it is crucial to impose the following assumption on the SIM:
[Equation]
The pivotal condition in (4) is fairly generic and encompasses notable special examples such as noisy 1 -bit measurements and general (non-binary) quantization schemes.
However, it is not satisﬁed by
PR models including (2) and the related models
[Equation]
where η refers to zero-mean random noise that is independent of a .
In order to formalize a class of
SIMs that encompass the above-mentioned PR models (and more general models, see the discussion in Section 2.2) as special cases, misspeciﬁed phase retrieval (MPR) has been studied in [64, 98], with the condition (4) being replaced by the assumption
[Equation]
It is worth mentioning that another motivation behind studying MPR is that the theoretical analysis for PR typically relies on the correct model speciﬁcation that the data points are indeed generated by the correct model, and the MPR model enables theoretical analysis under statistical model misspeciﬁcation.
In both works [64, 98], the signal x is assumed to be sparse.
Recently, motivated by tremendous successful applications of deep generative models and following the seminal work [6] on generative model based linear CS, it has been popular to study high-dimensional inverse problems under generative priors [76, 31, 32, 34, 96, 41, 2, 67, 95, 60, 40, 65].
In particular, instead of being sparse, the underlying signal is assumed to be contained in (or lie near) the range of a generative model.
It has been empirically demonstrated in [6] and its various follow-up works (e.g., see [19, 85, 48] and a literature review in [78]) that compared to sparsity based methods, corresponding generative model based algorithms require signiﬁcantly fewer samples to recover the signal up to a given accuracy.
In this paper, we study the MPR problem under generative modeling assumptions.
1.1 Related Work
In this subsection, we summarize some relevant works on PR and SIM, for both cases with or without generative priors.
PR and SIM without generative priors : There is a large amount of literature providing practical algorithms for PR, including convex methods [66, 14, 12, 90, 4, 29] and empirically more competitive non-convex methods [27, 21, 62, 70].
In particular, in the seminal work [62] and a variety of its follow-up works [13, 10, 15, 92, 91, 99, 39, 82, 8], whether it is for general PR (with no priors on the signal) or sparse PR, two-step approaches have been proposed with provable guarantees.
The ﬁrst step consists of a spectral initialization method, and the second step is typically an iterative (such as alternating minimization and gradient descent) algorithm that further reﬁnes the initial guess of the ﬁrst step.
For general PR, the spectral initialization step turns out to be unnecessary, and optimal sample complexity guarantees can be established even when using random initialization [83, 16].
However, to the best of our knowledge, it is not the case for sparse PR.
More speciﬁcally, all theoretically-guaranteed non-convex algorithms for sparse PR require spectral initialization, and this typically results in a sub-optimal sample complexity of O (cid:0) s 2 log n (cid:1) (instead of O ( s log n ) ), where s refers to the number of non-zero entries of the signal to estimate.
MPR is also closely related to SIMs, which have been extensively studied in the conventional setting where m > n , see, e.g., [30, 35, 59].
High-dimensional SIMs have received a lot of attention in recent years, with theoretical guarantees for signal estimation and support recovery [22, 24, 74, 84, 55, 72, 73, 17, 97, 28, 93, 69, 20].
In particular, motivated by the idea that under the assumption (4), a SIM can be converted into a scaled linear measurement model with unconventional noise, the authors of [72] consider minimizing the linear least-squares objective function over a convex set.
They show that a reliable estimation of the signal can be obtained by such a simple method despite the unknown nonlinear link function.
As mentioned earlier, MPR with sparse priors has been studied in [64, 98].
The work [64] implements a two-step procedure based on convex optimization, with the ﬁrst step being a spectral initialization method.
More speciﬁcally, a semideﬁnite program is solved to produce an initial vector, and then an (cid:96) 1 regularized least square is solved to obtain a reﬁned estimator.
Such a procedure suffers from high computational costs.
A more efﬁcient two-step approach, which is a simple variant of the thresholded
Wirtinger ﬂow method [10], is proposed in [98].
In the ﬁrst step, identical to that in [10], the initial vector is calculated by a thresholded spectral method that ﬁrst estimates the support of the sparse signal by thresholding and then performs the classic power method over the submatrix corresponding to the estimated support.
In the second step, a thresholded gradient descent algorithm is employed.
Both approaches in [64, 98] can attain the optimal statistical rate of order (cid:112) ( s log n ) /m , provided that the sensing vector is standard Gaussian and the number of samples m = Ω (cid:0) s 2 log n (cid:1) .
In addition, the second step of the approach proposed in [98] is shown to achieve a linear convergence rate.
PR and SIM with generative priors : PR with generative priors has been studied in [31, 37, 38, 80, 3, 47].
More speciﬁcally, an approximate message passing algorithm is proposed in [3].
The authors of [31, 80] minimize the objective over the latent space in R k using gradient descent, where k is the latent dimension of the generative prior.
Corresponding algorithms may easily get stuck in local minima since the explorable solution space is limited.
Recovery guarantees for projected gradient descent algorithms over the ambient space in R n for noiseless PR with pre-trained or untrained neural network priors have been proposed in [37, 38].
No initialization methods have been proposed in these works, making the assumption on the initial vector therein a bit stringent.
On the other hand, the authors of [47] propose a spectral initialization method for PR with generative priors and provide recovery guarantees with respect to globally optimal solutions of a corresponding optimization problem.
The optimization problem is non-convex, and a projected power method is proposed in [50] to approximately ﬁnd an optimal solution.
Generative model based SIMs have been studied in [51, 49, 46, 94].
The authors of [51, 49, 46] provide optimal sample complexity upper bounds under the assumption of Gaussian sensing vectors.
But their results rely on the assumption (4), which is not satisﬁed by widely adopted PR models.
The SIM studied in [94] encompasses certain PR models as special cases, and the sensing vector can be non-Gaussian.
However, the nonlinear link function f is assumed to be differentiable, making it not directly applicable to the PR model in (2).
Moreover, it is worth mentioning that in both works [94, 51], the recovery guarantees are with respect to globally optimal solutions of typically highly non-convex optimization problems.
Attaining these optimal solutions is practically difﬁcult.
1.2 Contributions
Throughout this paper, we assume that the signal x lies in the range of an L -Lipschitz continuous generative model with bounded k -dimensional inputs.
Our main contributions are as follows:
• We propose a two-step approach for MPR with generative priors.
In particular, in the ﬁrst step, we make use of the projected power method proposed in [50] to obtain a good initial vector for the iterative algorithm used in the second step.
• We show that under appropriate initialization, both steps attain a statistical rate of order (cid:112) ( k log L ) · (log m ) /m , and the second step achieves a linear convergence rate.
The initialization condition for the ﬁrst step is mild in the sense that it allows the inner product between the starting point and the signal x to be a sufﬁciently small positive constant.
In contrary, the initialization condition for the second step is more restrictive, making the ﬁrst step necessary.
Notably, unlike for the works on MPR with sparse priors [64, 98] that require the sub-optimal O (cid:0) s 2 log n (cid:1) sample complexity, the sample complexity requirement for our recovery guarantees is O (( k log L ) · (log m )) , which is naturally conjectured to be near-optimal [52, 42].
• We perform numerical experiments on image datasets to corroborate our theoretical results.
In particular, for the noisy magnitude-only measurement model (2), we observe that our approach gives reconstructions that are competitive with those of the alternating phase projected gradient descent (APPGD) algorithm proposed in [37], which is the corresponding state-of-the-art method, though we do not utilize the knowledge of the link function and allow for model misspeciﬁcation.
Moreover, for several closely related measurement models that satisfy the condition (6), our approach leads to superior reconstruction performance compared to all other competing methods, including APPGD.
Problem Formulation
In this section, we overview some important assumptions that we adopt.
Before proceeding, we present the notation used in this paper.
2.1 Notation
We use upper and lower case boldface letters to denote matrices and vectors respectively.
For any positive integer N , we use the shorthand notation [ N ] = { 1 , 2 , · · · , N } , and I N represents the identity matrix in R N × N .
The support (set) of a vector is the index set of its non-zero entries.
We use (cid:107) X (cid:107) 2 → 2 to represent the spectral norm of X .
We use B k ( r ) to denote the radius- r in R k , i.e., B k ( r ) :
= { z ∈ R
k : (cid:107) z (cid:107) 2 ≤ r } , and S n − 1 represents the unit sphere in R n , i.e.,
S n − 1 := { s ∈ R n : (cid:107) s (cid:107) 2 = 1 } .
We use G to denote a pre-trained L -Lipschitz continuous generative model from B k ( r ) to R n .
We focus on the setting where k (cid:28) n .
For any set S ⊆ B k
( r ) , we write
G ( S ) =
{ G ( z ) : z ∈ S } .
Our goal is to estimate the signal x ∈ Range( G ) =
G ( B k ( r )) from realizations of the sensing vector a and the observation y (generated according to the SIM in (3)).
For any two sequences of real values { a j } and { b j } , we write a j = O ( b j ) if there exist an absolute constant C 1 and a positive integer j 1 such that for any j > j 1 , | a j | ≤ C
b
j , a j = Ω( b j ) if there exist an absolute constant C 2 and a positive integer j 2 such that for any j > j 2 , | a j | ≥ C 2 b j .
We use the generic notations C and C (cid:48) to denote large positive constants, and we use c to denote a small positive constant; their values may differ from line to line.
2.2 Setup
First, the following are the standard deﬁnitions of a sub-exponential random variable and the associated sub-exponential norm.
Deﬁnition 1.
A random variable X is said to be sub-exponential if there exists a positive constant
C such that ( E [ | X | p ]) 1 /p ≤
Cp for all p ≥ 1 , or equivalently, if there exists a positive constant C (cid:48) such that P ( | X | > u ) ≤ exp(1 − u/C (cid:48) ) for all u ≥ 0 .
The sub-exponential norm of X is deﬁned as
[Equation]
We will focus on the following settings except where stated otherwise:
• The observations are independently generated according to the SIM (3), where f is the link function that is unknown and possibly random.
• We have an L -Lipschitz continuous generative model G : B k ( r ) → R n .
For convenience, similarly to [48, 50], we assume that the generative model is normalized
such that Range( G ) ⊆ S n − 1 .
For a general (unnormalized) generative model, we may essentially consider its normalized version.
See, e.g., [50, Remark 1].
The signal x is contained in the range of G , i e., x ∈ Range( G ) ⊆
S n − 1 .
• The sensing vector a ∈ R n is standard Gaussian, i.e., a ∼ N ( 0 , I n ) .
• The random variable y =
f ( a T x ) is sub-exponential with the sub-exponential norm being denoted by K y , i.e., K y :
= (cid:107) y (cid:107) ψ 1 .
In addition, we use M y to denote the expectation of y , i.e., M y : =
E [ y ] .
Remark 1.
y will be sub-exponential when f ( x ) comprises of x c plus lower order terms with c ≤ 2 (since the product of two sub-Gaussian random variables is sub-exponential), and therefore we will see that the y corresponding to all the measurement models presented in our paper is sub-exponential.
We remark that the assumption of sub-exponential y is not essential and it can be easily relaxed.
For example, when y = x c with c being an even integer that is larger than 2 , there will be only a minor change in the order of the log m term in the sample complexity and statistical rate.
However, for brevity, we follow [64, 98] and make the assumption of sub-exponential y to avoid non-essential complications.
• We consider MPR and assume that
[Equation]
or equivalently,
[Equation]
where g ∼ N (0 , 1) is a standard normal random variable.
Note that the assumption (8) is only with respect to the nonlinear link function f .
The condition in (7) is satisﬁed by PR models described in (2) and (5).
It is also satisﬁed by relevant models such as [98]
[Equation]
See [64, Proposition 3 and Remark 4] for more general examples.
Algorithm
In this section, we describe our two-step algorithm devised for MPR with generative priors.
Suppose that we have m i.i.d.
realizations of a and y , namely a 1 , . .
. , a m and y 1 , . .
. , y m .
To estimate the signal x , we consider the following two-step approach:
We perform T 1 iterations in the ﬁrst step.
In particular, let
[Equation]
We perform the projected power method proposed in [50]: For t = 0 , 1 , . . .
, T 1 − 1 , let
[Equation]
where P G ( · ) denotes the projection function onto Range( G ) , 2 and we obtain x (0) :
= w ( T 1 ) .
Similarly to [43, 47], we set the starting point w (0) as the column of 1 m (cid:80) mi =
y i a i a Ti (i.e., a shifted version of V ) that corresponds to the largest diagonal entry.
Note that it is easy to calculate that E
[ V ] =
ν xx T (see, e.g., [47, Lemma 8]), for which each column is a scalar product of x .
This motivates the use of a shifted version of V to get the initialization vector.
We perform T 2 iterations in the second step.
In particular, let
[Equation]
be the empirical mean of the observations.
We perform the following iterative procedure:
For t = 0 , 1 , 2 , . . .
, T 2 − 1 , let
[Equation]
where ζ > 0 is a tuning parameter, and ˆ ν ( t ) can be thought of as an approximation of ν deﬁned in (7).
The idea behind calculating ˜ y ( t ) i is that by comparing (4) and (6), we observe that to transform the MPR model into a conventional SIM, we may use ( y − E
[ y ])( a T x ) to [Equation] [Equation]
Similarly to [79, 37, 38, 71, 50], we implicitly assume the exact projection in our analysis. In practice, approximate methods such as gradientand GAN-based projections [79, 75] have been shown to work well. Input : { ( a i , y i ) } mi =1 , step size ζ > 0 , number of iterations T 1 for the ﬁrst step, number of iterations T 2 for the second step, pre-trained generative model G , initial vector w (0)
First step : 1: for t = 0 , 1 , . .
, T 1 − 1 do 2: w ( t +1) =
P G (cid:0) Vw ( t ) (cid:1) 3: end for
Second step :
Let x (0) :
= w ( T 1 ) 1: for t = 0 , 1 , . . .
, T 2 − 1 do 2:
Calculate ˆ ν ( t ) , ˜ y ( t ) i , ˜ x
( t +1) and x ( t +1) according to (13), (14), (15), and (16), respectively 3: end for
Output : ˆ x := x ( T 2 ) replace y , see also [64].
Moreover, (15) can be considered as a gradient descent step with respect to a linear measurement model with the scale factor ˆ ν ( t ) and observations ˜ y
( t ) i .
Then, in (16), we project the calculated vector ˜ x
( t +1) onto Range( G ) .
Finally, we use ˆ x := x ( T 2 ) to represent the estimated vector obtained after T 2 iterations.
For convenience, we summarize the details in Algorithm 1.
Theoretical Results
The following theorem establishes recovery guarantees for the ﬁrst step of Algorithm 1.
The proof is given in the supplementary material.
Note that K y :
= (cid:107) y (cid:107) ψ 1 ( cf.
Section 2.2) is considered a ﬁxed constant and is omitted in the O ( · ) notation.
Theorem 1.
Assume that there exists a positive integer t 0 such that x T w ( t 0 ) ≥ c 0 , where c 0 is a sufﬁciently small positive constant.
Suppose that m =
Ω
(( k log( nLr )) · (log m )) with a large enough implied constant.
Then, we have that with probability 1 − O (1 /m ) , it holds for all t > t 0 that
[Equation]
Since a d -layer feedforward neural network generative model from B k ( r ) to R n is typically LLipschitz continuous with L = n Θ( d ) [6] and we may set r = n Θ( d ) , the upper bound in (17) is of order (cid:112) ( k log L ) · (log m ) /m .
Such a statistical rate is naturally conjectured to be near-optimal according to information-theoretic lower bounds established for MPR with sparse priors [64] and generative model based principal component analysis [50].
Therefore, Theorem 1 reveals that the ﬁrst step of Algorithm 1 attains the near-optimal statistical rate under appropriate initialization and exact projections.
The accurate projection assumption is perhaps the major caveat to Theorem 1.
However, it is a standard assumption in relevant works including [79, 37, 38, 71, 50].
In practice, both gradient- and GAN-based projection methods [79, 75] have been shown to be highly effective in approximating the projection step.
Remark 2.
Spectral initialization steps in relevant works on sparsity based PR [92, 39, 25, 36, 9, 7] or MPR [64, 98] require the sub-optimal sample complexity O
( s 2 log n ) , where s refers to the number of non-zero entries.
In contrary, according to Theorem 1, our spectral initialization step only requires the near-optimal O (( k log L ) · (log m )) sample complexity (with a linear rather than a quadratic dependence on k ).
However, we note that such an advantage of our spectral initialization step comes at a price.
In particular, we require the initialization condition x T w ( t 0 ) ≥ c 0 , which is not required by spectral initialization steps in the above-mentioned works on sparse PR/MPR.
Remark 3.
For some applications, we may assume that the dataset contains only vectors whose elements are all non-negative. For example, this is a natural assumption for image datasets.
During pre-training, we can easily set the activation function of the last layer of the neural network generative model to be a certain non-negative function such as ReLU or sigmoid, and the range of such a generative model is contained in the non-negative orthant.
Therefore, the assumption that x T w ( t 0 ) ≥ c 0 for a sufﬁciently small positive constant c 0 is also mild.
Similar assumptions have been made in relevant works including [18, 50] where it is not appropriate to assume that − x is also contained in the structured set (such as a closed convex cone or the range of a deep generative model).
As a result, we provide an upper bound on (cid:107) w ( t ) − x (cid:107) 2 , instead of min {(cid:107) w ( t ) − x (cid:107) 2 , (cid:107) w ( t ) + x (cid:107) 2 } , which is a commonly adopted distance measure in relevant literature on real-valued PR.
Moreover, although the projected power iterations in the ﬁrst step of Algorithm 1 can attain the near-optimal statistical rate under appropriate conditions, it is evident in a large body of literature on
PR (see, e.g., [62, 13, 10, 99, 98, 64]) that such a spectral method better serves as the initialization step of a subsequent iterative approach.
This motivates us to propose the second step of Algorithm 1, and in our numerical experiments, we clearly observe the beneﬁt of the second step.
More speciﬁcally, compared to simply performing the projected power method, performing both steps of Algorithmleads to signiﬁcantly better reconstructed images when the total number of iterations is ﬁxed to be the same, namely T 1 + T 2 .
Next, we present the following theorem, which is proved in the supplementary material.
This theorem shows that under appropriate initialization and the assumption of exact projections, the iterative algorithm in the second step of Algorithm 1 converges linearly to a point achieving the near-optimal statistical rate of order (cid:112) ( k log L ) · (log m ) /m .
Theorem 2.
Assume that the step size ζ and x (0) , which is the initial vector for the second step of
Algorithm 1, satisfy
[Equation]
where both β 1 and β 2 are positive constants.
3 Suppose that m = Ω(( k log( nLr )) · (log m )) with a large enough implied constant.
Then, we have with probability 1 − O (1 /m ) , the following event occurs: There exists a positive integer T
= O (cid:0) log (cid:0)
[Equation]
, such that the sequence (cid:8) (cid:107) x ( t ) − x (cid:107) 2 (cid:9) t ∈ [0 ,T 0 ] is monotonically decreasing, with the following inequality holds for all t ≤ T 0 :
[Equation]
[Equation]
In addition, we have for all t ≥ T 0 that
[Equation]
Remark 4.
In our analysis, we need to impose the assumption (18) on the step size ζ and initial vector x (0) .
This makes the ﬁrst step of Algorithm 1 necessary since when (cid:107) x (0) − x (cid:107) 2 is not small, say (cid:107) x (0) − x (cid:107) 2 = 1 , the condition (18) cannot be satisﬁed.
In comparison, to attain the near-optimal statistical rate O ( (cid:112) ( k log L ) · (log m ) /m ) , the initialization condition of the ﬁrst step of Algorithmis milder, and x T w ( t 0 ) (see Theorem 1) only needs to be lower bounded by a sufﬁciently small positive constant (thus (cid:107) w ( t 0 ) − x (cid:107) 2 can be close to √ 2 ).
However, although the second step of Algorithmrequires a more restrictive initialization condition, we observe from our experimental results that it clearly reﬁnes the estimate of the ﬁrst step.
Such a phenomenon is also observed in various works related to PR, including [62, 13, 10, 99, 98, 64].
Remark 5.
In Remark 4, we have brieﬂy discussed the comparison of the initialization condition x T w ( t 0 ) ≥ c 0 in Theorem 1 and the typical initialization condition (cid:107) x − w ( t 0 )
(cid:107) 2 < δ (cid:107) x (cid:107) 2 .
In the following, we provide a more detailed discussion: When both x and w ( t 0 ) are unit vectors (this is the setting of our Theorem 1), the typical initialization requirement (cid:107) x − w ( t 0 ) (cid:107) 2 < δ (cid:107) x (cid:107) 2 can be reduced to 2 (cid:0)
1 − x T w ( t 0 ) (cid:1) < δ 2 , or equivalently, x T w ( t 0 ) 1 − δ 2 2 . Note that δ is typically a small positive constant (e.g., δ = 16 in [10] and δ = 18 in [13]), and thus the typical initialization condition requires x T w ( t 0 ) to be larger than some positive constant that is close to 1 . This is stronger than the assumption x T w ( t 0 ) ≥ c 0 , 4 where c 0 is a sufﬁciently small positive constant.
Remark 6.
The condition in (18) requires | 1 − ζν | < 12 . This reveals that we should choose ζ such that ζ ∈ (cid:0) 12 ν , 32 ν (cid:1) , and a good choice of ζ is ζ = 1 ν (for this case, the condition (18) reduces to (cid:107) x (0) − x (cid:107) 2 < 15 ).
Since the knowledge of the link function f (and thus the knowledge of ν , which is dependent on f ; See (8) ) cannot be assumed, in our experiments, we use ˆ ν ( t ) to approximate ν .
That is, ζ is set to be 1ˆ ν ( t ) in the t -th iteration of the second step of Algorithm 1 (though it is slightly varying instead of being ﬁxed).
Numerical Results
In this section, we demonstrate the empirical performance of our Algorithm 1 (denoted by MPRG ).
We remark that these numerical experiments are proof-of-concept rather than seeking to be comprehensive since our contributions are primarily theoretical.
We present some numerical results for the MNIST [44] dataset in the main document. Additional results for the MNIST dataset and some experimental results for the CelebA [53] dataset are presented in the supplementary material.
The MNIST dataset contains 60 , 000 images of handwritten digits.
The size of each image is 28 × 28 , and thus the dimension of the image vector is n = 784 .
For the MNIST dataset, the generative model
G is set to be (the normalized version of) a pre-trained variational autoencoder (VAE) model with the latent dimension being k = 20 .
We make use of the VAE model pre-trained by the authors of [6] directly.
For this VAE model, both the encoder and decoder are set to be fully connected neural networks with two hidden layers, and the architecture is 20 − 500 − 500 − 784 .
The VAE model is trained by the Adam optimizer with a mini-batch size 100 and a learning rate of 0 . 001 , and is trained from the images in the training set.
The projection step P G ( · ) ( cf.
(11)) is approximated by the Adam optimizer with a learning rate of 0 . 1 and 120 steps.
We report the results on 10 testing images that are selected from the test set.
Note that these images are unseen by the pre-trained generative model.
We perform 10 random restarts.
For reconstructed images, we choose the best among these 10 random restarts to reduce the impact of local minima.
We also provide quantitative comparisons with respect to the reconstruction error (cid:107) ˆ x − x (cid:107) 2 , where x is the underlying signal and ˆ x refers to the estimated (normalized) vector produced by each of the methods described below.
The reconstruction error is averaged over both the 10 testing images and 10 random restarts.
All experiments are run using Python 3.6 and TensorFlow 1.5.0, with a NVIDIA
GeForce GTX
Ti 11GB GPU.
For Algorithm 1, we set T 1 = 20 and T 2 = 30 .
As mentioned in Section 3, the starting point w (0) is set to be the column of 1 m (cid:80) mi =
y i a i a Ti (i.e., a shifted version of V deﬁned in (10)) that corresponds to the largest diagonal entry.
In addition, as mentioned in Remark 6, we set the step size ζ as ζ = 1ˆ ν ( t ) ( cf.
(13)) in the t -th iteration of the second step of Algorithm 1.
We compare our Algorithm 1 (denoted by MPRG ) with the following methods: 1) The method proposed in [98], which is for misspeciﬁed phase retrieval with sparse priors and is denoted by MPRS .
All the involved parameters are set to be the same as those in [98].
2) Simply performing the ﬁrst step of Algorithm 1 for T 1 + T 2 =iterations.
The corresponding method is denoted by PPower .
The purpose of comparing to PPower is to verify the beneﬁt of the second step of Algorithm 1. 3)
Simply performing the second step of Algorithm 1 for T 1 + T 2 = 50 iterations.
The corresponding method is denoted by Step2 .
The purpose of comparing to Step2 is to check whether the ﬁrst step of Algorithm 1 is practically useful.
The Alternating Phase Projected Gradient Descent (denoted by APPGD ) algorithm proposed in [37].
This algorithm is speciﬁcally designed for phase retrieval with magnitude-only measurements ( cf.
and generative priors, and the corresponding iterative procedure is
[Equation]
where τ > 0 is the step size.
We follow [37] to set τ = 0 .
For a fair comparison, we use the vector produced after T 1 = 20 iterations of the ﬁrst step of Algorithm 1 as the initialization vector of APPGD , and then we run APPGD for T 2 = 30 iterations.
We ﬁrst consider the noisy magnitude-only measurement models for i ∈ [ m ] ,
[Equation]
where η i are i.i
[Figure] d. realizations of an N (0 , σ 2 ) random variable.
For such a measurement model, the corresponding random nonlinear link function f is f ( x ) = | x | + η or f ( x ) =
| x + η | , where η ∼ N (0 , σ 2 ) .
The numerical results for (22) and (23) are demonstrated in Figures 1 and 2.
From these ﬁgures, we observe that the sparsity based method MPRS attains poor reconstructions, and the results of PPower are not desirable.
The three methods Step2 , APPGD , MPRG lead to similar reconstruction error
[Figure]
but the reconstructed images of APPGD and MPRG are better than those of Step2 .
In particular, our method MPRG leads to mostly accurate reconstructions that are competitive compared to those of APPGD , even if we do not make use of the knowledge of the link function f and MPRG is not speciﬁcally designed for the magnitude-only measurement models
[Figure] [Figure]
[Figure]
[Figure]
Next, we consider the following two measurement models:
[Equation]
where again η i are i.i.d.
realizations of an N (0 , σ 2 ) random variable.
For both models in (24) and (25), the corresponding link functions satisfy the condition (8) for MPR [98].
The numerical results are presented in Figures 3 and 4.
We observe from these ﬁgures that for the measurement models (24) and (25), our method MPRG achieves the best reconstructions.
In particular, it outperforms APPGD in terms of recovery quality and/or reconstruction error.
Conclusion and Future Work
We have proposed a two-step approach for phase retrieval under model misspeciﬁcation and generative priors.
We show that under suitable conditions, both steps of our approach obtain estimated vectors that achieve the near-optimal statistical rate of order (cid:112) ( k log L ) · (log m ) /m , where k is the latent dimension and L is the Lipschitz constant of the generative model respectively, and m refers to the number of samples.
O r i g i n a l
M P R S
PP o w e r
S t e p
A PP G D
M P R G
O r i g i n a l
M P R S
PP o w e r
S t e p
A PP G D
M P R G
[Figure] [Figure]
We assume accurate projections in our analysis and use a gradient-based method to approximate the projection step P G ( · ) in our experiments. Although the exact projection assumption is commonly made in relevant works [79, 37, 38, 71, 50], it is a very interesting future research direction to design provably-guaranteed efﬁcient methods for the projection step. In addition
[Figure]
we focus on real Gaussian measurements.
While we believe that based on the technical results in [62, 13] (which study complex Gaussian measurements), it is straightforward to extend our work to the complex case, the extension to more practical non-Gaussian measurement models such as sub-sampled Fourier measurements is a very interesting future direction. Another direction is to use different preprocessing functions to enhance the performance of our spectral initialization method [56, 54, 57].
Moreover, if one has the access to the nonlinear link function f , the Bayesoptimal performances can characterized using message-passing algorithms [5, 58, 3]. It would be interesting to connect or compare our results with the corresponding Bayes-optimal rate. Acknowledgment. J. Liu was partially supported by NSFC Grant #12288201 and the Fund of the
Youth Innovation Promotion Association, CAS (2022002).
We sincerely thank the ﬁve anonymous reviewers and area chair for their careful reading and helpful comments.
We are extremely grateful to
Dr. Jonathan Scarlett for proofreading the manuscript and giving valuable suggestions
Checklist
For all authors... (a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s contributions and scope?
[Yes] (b) Did you describe the limitations of your work?
[Yes] The limitations are relevant to our future work, described in Section 6.
(c) Did you discuss any potential negative societal impacts of your work?
[N/A] (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes] 2.
If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
[Yes] (b) Did you include complete proofs of all theoretical results?
[Yes] 3.
If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[Yes] The code is included in the supplementary material.
(b) Did you specify all the training details (e
g., data splits, hyperparameters, how they were chosen)?
[Yes] (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[Yes] (d) Did you include the total amount of compute and the type of resources used (e
g., type of GPUs, internal cluster, or cloud provider)?
[Yes] 4.
If you are using existing assets (e g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
[N/A] (b) Did you mention the license of the assets?
[N/A] (c) Did you include any new assets either in the supplemental material or as a URL?
[N/A] (d) Did you discuss whether
and how consent was obtained from people whose data you’re using/curating?
[N/A] (e) Did you discuss whether the data you are using/curating contains personally identiﬁable information or offensive content?
[N/A] 5.
If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?
[N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
[N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
[N/A]