Signal Recovery with Non-Expansive Generative Network Priors
Anonymous Author(s)
AfﬁliationAddress email
Abstract
We study compressive sensing with a deep generative network prior.
Initial theoretical guarantees for efﬁcient recovery from compressed linear measurements havebeen developed for signals in the range of a ReLU network with Gaussian weightsand logarithmic expansivity: that is when each layer is larger than the previous oneby a logarithmic factor.
It was later shown that constant expansivity is sufﬁcient forrecovery.
It has remained open whether the expansivity can be relaxed, allowingfor networks with contractive layers (as often the case of real generators).
In thiswork we answer this question, proving that a signal in the range of a Gaussiangenerative network can be recovered from few linear measurements provided thatthe width of the layers is proportional to the input layer size (up to log factors).
Thiscondition allows the generative network to have contractive layers.
Our result isbased on showing that Gaussian matrices satisfy a matrix concentration inequalitywhich we term Range Restricted Weight Distribution Condition (R2WDC) andwhich weakens the Weight Distribution Condition (WDC) upon which previoustheoretical guarantees were based.
The WDC has also been used to analyze othersignal recovery problems with generative network priors.
By replacing the WDCwith the R2WDC, we are able to extend previous results for signal recovery withexpansive generative network priors to non-expansive ones.
We discuss theseextensions for phase retrieval, denoising, and spiked matrix recovery.
Introduction
The compressed sensing problem consists in estimating a signal y ?
2 R n from (possibly) noisy linearmeasurements
[Equation]
where A 2 R m ⇥ n is the measurements matrix, m < n and ⌘ 2 R m is the noise.
To overcome the ill-posedness of the problem, structural priors on the unknown signal y ?
need tobe enforced.
One now classical approach assumes that the target signal y ? is sparse with respectto a given basis.
In the last 20 years, efﬁcient reconstruction algorithms have been developed thatprovably estimate s -sparse signals in R n from m =
O ( s log n ) random measurements [5, 12].
Another approach recently put forward, leverages trained generative networks.
These networksare trained, in an unsupervised manner, to generate samples from a target distribution of signals.
Assuming y ?
belongs to the same distribution used to train a generative network G : R k !
R n withk ⌧ n , an estimate of y ?
can be found by searching the input
ˆ x (“latent code”) of G that minimizesthe reconstruction error
[Equation]
As empirically demonstrated in [3], the minimization problem (1) can be solved efﬁciently by gradientdescent methods.
Moreover, solving (1) can effectively regularize the solution of the compressedsensing problem, signiﬁcantly outperforming sparsity-based algorithms in the low measurementsregime[3].
Generative network based inversion algorithms have been subsequently developed for avariety of signal recovery problems, demonstrating their potential to outperform inversion algorithmsbased on non-learned (hand-crafted) priors [16, 31, 30, 20, 33, 28].
For a recent overview see [32].
The optimization problem (1) is in general non-convex and gradient-based methods could get stuckin local minima.
To better understand the empirical success of (1), in [18] the authors establishedtheoretical guarantees for the noiseless compressed sensing problem ( ⌘ = 0 ) where G : R k !
R n isa d -layer ReLU network of the form:
[Equation]
with W i 2 R n i ⇥ n i  1 , n 0 = k , n d = n , and ReLU ( z ) = max( z, 0) is applied entrywise.
The authorsof [18] used a probabilistic model for the generative network G and measurement matrix A .
Theyassumed that each layer W i has independent Gaussian entries and is strictly expansive .
Speciﬁcally itholds that
[Equation]
Moreover, they considered A to be a Gaussian matrix and m  k · log n · poly ( d ) .
Under this probabilistic model it was shown in [18] that, despite its non-convexity, f cs has a favorable optimizationgeometry and no spurious critical points exist apart from x ? and a negative multiple of it  ⇢ d x ? ,where ⇢ d is a function of the depth d of the network.
The landscape analysis was later extended to recovery guarantees using a gradient based method in[21], under the same probabilistic assumptions of [18].
In particular, [21] has shown that there isan efﬁcient gradient descent method (see Algorithm 1 in Section 3) that given as input A, G and boutputs a latent vector ˜ x such that k y ?
G (˜ x ) k 2 =
O ( k ⌘ k 2 ) .
This result demonstrated that efﬁcient 54 recovery is possible with a number of measurements which is information-theoretic optimal up tolog -factors in n and polynomials in d ( m = ˜ ⌦ ( k ) ).
Generative networks used in practice though, have often contractive layers.
For example, the outputof the layers near the end of the StyleGAN generators have larger dimension than the generatedimages [25, 24].
Thus, one major drawback of the theory developed in [18] is constituted by theexpansivity condition on the weight matrices (3).
Relaxing the condition (3) and accommodating forgenerative networks with contractive layers was formulated as an open problem in the survey paper 1[32].
An initial positive result on this problem came from [10].
Using a reﬁned analysis of the concentrationof Lipschitz functions, the authors proved that the results of [18, 21] hold true also for weight matricessatisfying n i  n i  1 · poly ( d ) .
While not allowing for contractive layers, this condition removed the 65 logarithmic expansivity requirement of (3).
More recently, [22, 23] have studied the denoising and compressive sensing problem with randomgenerative network prior as in [18, 21, 20], and have shown that the expansivity condition can indeedbe relaxed.
In [23] they have provided an efﬁcient iterative method that given as input A, b and G ,assuming that up to log -factors each layer width satisﬁes
[Equation]
and the number of measurement satisﬁes
[Equation]
outputs a latent vector ˜ x such that for y ?
=
G ( x ? )
it holds that
[Equation]
with high probability.
Notice that the condition (4) while requiring the width to grow with the depth,can allow for contractive layers n i < n i  1 .
1.1 Our contributions
It is natural to wonder whether the price to pay to remove the expansivity assumption is indeed theexponential factors in the depth d of the network and the use of less-standard non-gradient based iterative methods, as happens in [22, 23].
In this paper, we answer these questions.
Our main result issummarized below and provides guarantees for solving compressed sensing with random generativenetwork priors via a gradient descent method (Algorithm 1 in Section 3).
Theorem 1.1 (Informal version of Theorem 5.4) .
Assume that A has i.i.d.
N (0 , 1 /m )
entries andeach W i has i.i.d.
N (0 , 1 /n i ) entries.
Suppose that y ?
=
G ( x ? ) .
Furthermore assume that, up tolog -factors,
[Equation]
Suppose that the noise error and the step size ↵ > 0 are small enough.
Then with high probability,
Algorithm 1 with input loss function f cs , step size ↵ and number of iterations T = poly ( d ) , outputs anestimate G ( x T ) satisfying k G ( x T )  y ?
k
= O (
[Equation]
Compared to [21] and [10], our result do not require strictly expanding generative networks andallows for contractive layers.
Furthermore, we show that the same algorithm proposed in [21] has adenoising effect, leading to a reconstruction of the target signal y ?
of the order O ( q k m k ⌘ k 2 ) ratherthan only O ( k ⌘ k 2 ) .
We show that this holds true even in case of deterministic noise, while [19]discuss only the case of Gaussian noise.
Furthermore, the decrease in the reconstruction error withthe number of measurements has also been observed for trained generative networks (see for example[3]), and here we give a partial theoretical explanation for this phenomenon.
Compared to the results of [23] we show that it is sufﬁcient for the width of the layers as well as thenumber of measurements to grow polynomially with the depth rather than exponentially.
Similarly,compared to [23], we remove the exponential factor in the depth from the reconstruction error.
The analysis of [18] was based on a deterministic condition on the weight matrices termed Weight
Distribution Condition (WDC).
This condition, together with a deterministic condition on A (see Sec4 for details), was shown to be sufﬁcient for the absence of spurious local minima in (1) and to besatisﬁed by expansive Gaussian random generative networks as (2).
The WDC was also used in thesubsequent [21] to prove convergence of Algorithm 1.
Our main technical contribution is to showthat the WDC can be replaced by a weaker form of deterministic condition, termed Range Restricted
Weight Distribution Condition (R2WDC), and still, obtain the absence of spurious local minima andrecovery guarantees via Algorithm 1.
We will then show that random Gaussian networks satisfyingthe Assumption 1. of Theorem 1.1 satisfy the R2WDC.
The framework introduced in [18] was used in a number of recent works to analyze other signalrecovery problems with generative network priors, from one-bit recovery to blind demodulation[34, 27, 16, 15, 35, 8].
These works considered expansive generative network priors, using the WDCand the results of [18] in their analysis.
Replacing the WDC with our R2WDC we can extend theprevious results in the literature to more realistic (non-expansive) generative networks.
This paperdetails these extensions for three representative signal recovery problems.
Theorem 1.2.
Suppose G is random generative network as in (2) , satisfying Assumption 1. of
Theorem 1.1.
Then Algorithm 1 with appropriate loss functions, step sizes, and number of steps,succeed with high probability for Phase Retrieval, Denoising, and Spiked Matrix Recovery.
Our result on the denoising problem, implies a similar result on the inversion of a generative network.
The problem of inverting a generative neural network has important applications [39, 1, 33], andhas been recently analyzed theoretically [26, 22, 2].
Our result shows that a random generativenetwork can be efﬁciently inverted by gradient descent, even when containing contractive layers.
This motivates the empirical use of gradient-based methods for inverting generative networks.1.
2 Organization of the paper
This paper is organized as follows.
In Section 2 we introduce some notation used in the rest of thepaper.
In Section 3 we formalize the compressed sensing problem with a generative network priorand describe an algorithm for the recovery.
In Section 4 we describe our novel deterministic conditionon the weights of the network (R2WDC) and provide theoretical guarantees for solving compressed sensing with a generative network prior satisfying this condition via the algorithm described in
Section 3.
Then in Section 5 we demonstrate that random non-expansive generative networks satisfythe R2WDC with high probability.
The appendix contains the full proof of the results described in themain text.
Appendix F contains the extension of the theoretical guarantees for compressed sensingwith a generative network prior to other signal recovery problems.
Preliminaries
We use I n to denote the n ⇥ n identity matrix.
For j  0 , we deﬁne the j -th sub-network G j : R k !
R n j as G j ( x ) = ReLU ( W j · · · ReLU ( W 2 ReLU ( W 1 x ))) , with the convention that G 0 ( x ) = I
k x = x .
For a matrix W 2 R n ⇥ k , let diag ( Wx > 0) be the diagonal matrix with i -th diagonal elementequal to 1 if ( Wx ) i > 0 and 0 otherwise, and W + ,x = diag
( Wx > 0) W .
We then deﬁne
W 1 , + ,x = ( W 1 ) + ,x =
diag ( W 1 x > 0) W 1 and W j, + ,x = diag ( W j W j  1 , + ,x · · · W 2 , + ,x
[Equation]
1 , + ,x ) W j .
Finally, we let ⇤ 0 ,x
= I k and for j  1 ⇤ j,x =
Q j ` =1 W ` , + ,x with ⇤ x = ⇤ d,x =
Q d ` =1 W ` , + ,x . 138 Notice in particular that G j ( x ) =
⇤ j,x x and G ( x ) =
⇤ x x .
For r, s nonzero vectors in R ` , we deﬁne the matrix
[Equation]
(6) where ✓ r,s = \ ( r, s ) , ˆ r = r/ k
r k 2 , ˆ s
= s/ k s k 2 , I ` is the ` ⇥ ` identity matrix and M ˆ r $ ˆ s is thematrix that sends ˆ r 7!
ˆ s , ˆ s 7!
ˆ r , and with kernel span ( { r, s } ) ?
If r or s are zero, then we let 142 Q r,s = 0 .
The operator Q r,s is used to deﬁne the WDC in the next sections, and allows tocontrol how a random ReLU layer distorts its inputs.
Speciﬁcally, for very r, s 2 R ` we have
E ⇥
ReLU ( Wr ) T ReLU ( Ws ) ⇤ =
r T Q r,s s when W 2 R n ⇥ ` has i.i.d.
N (0 , 1 /n ) .  Problem statement and recovery algorithm
Consider a generative network G : R k !
R n as in (2).
The compressive sensing problem with agenerative network prior can be formulated as follows.
COMPRESSED SENSING WITH A DEEP GENERATIVE PRIOR
Let :
G : R k !
R n generative network, A 2 R m ⇥ n measurement matrix.
Let : y ?
=
G ( x ? )
for some unknown x ? 2 R k .
Given :
G and A .
Given :
Measurements b
=
Ay ?
+ ⌘ 2 R m with m ⌧ n and ⌘ 2 R m noise.
Estimate : y ? .
To solve the compressed sensing problem with deep generative prior G , in [21], the authors proposethe gradient descent method described in Algorithm 1 with objective function f = f cs .
This algorithmattempts to minimize the objective function f cs in (1).
Because of the ReLU activation function, theloss function f cs is nonsmooth.
Algorithm 1 therefore resorts to the notion of Clarke subdifferential .
Indeed, being continuous and piecewise smooth, f cs is differentiable almost everywhere (by
Rademacher’s theorem) and admits a Clarke subdifferential given by 2 :
[Equation]
where with conv ( · ) we denote the convex hull and with dom ( r f ) the subset of R k where f isdifferentiable.
The vectors v x 2 @
f cs ( x ) are called the subgradients of f cs at x , and at a point xwhere f cs is differentiable it holds that @
f cs ( x ) = { r f cs ( x ) } .
Input: Objective function f , initial point x 0 2 R k \ { 0 } and step size ↵
Output latent do
else x t  v 6 end Notice that, as described in line 5, Algorithm 1 corresponds to a subgradient descent method withconstant step size ↵ .
Before taking a step in the direction of the subgradient though, the algorithmchecks whether the objective function at the current state x t has a larger value than the value at itsnegative  x t , and if so it updates the current state with its negative (line 3-4).
This negation step allowsthe algorithm to escape the spurious critical point in a neighborhood of  ⇢ d x ? where ⇢ d 2 (0 , 1) , andit is motivated by the landscape analysis of f cs under the deterministic and probabilistic assumptionsthat we describe in the coming sections.
Recovery guarantees under deterministic conditions
The strategy taken in [18] and [21] to analyze the landscape of the minimization problem (1) andthe convergence of Algorithm 1, consists in identifying a set of deterministic conditions on themeasurements matrix A and the generative network G , that ensure that the objective function f cs iswell behaved and Algorithm 1 converges efﬁciently to an estimate of x ? and y ? .
These conditions arethen shown to hold with high probability under probabilistic models for A and G .
This is akin to theresults on compressed sensing with sparsity where, for example, recovery guarantees were developedunder the Restricted Isometry Property [4].
The ﬁrst condition, introduced in [18], is on the measurement matrix A and ensures that A T A behaveslike an isometry over differences of points in the range of a generative network G .
Deﬁnition 4.1 (RRIC [18]) .
A matrix A 2 R m ⇥ n satisﬁes the Restricted Isometry Condition withrespect to G with constant ✏ if for all x 1 , x 2 , x 3 , x 4 2 R k , it holds that
[Equation]
The second deterministic condition introduced in [18] is on the weight matrices of G , ensures thatthey are approximately distributed like a Gaussian, and allows the control of how the layers of thenetwork distort angles.
Deﬁnition 4.2 (WDC [18]) .
We say that a generative network G as in (2), satisﬁes the Weight
Distribution Condition (WDC) with constant ✏ > 0 if for all i = 1 , . . .
, d , for all r, s 2 R n i  1 :
[Equation]
Strictly speaking, in [18] the authors deﬁne the WDC as a property of a single weight matrix W , andthen assume that the WDC is satisﬁed at each layer W i of G .
This is equivalent to the deﬁnition aboveand simpliﬁes the introduction of a novel, weaker, condition on the weight matrices, the R2WDCbelow.
Deﬁnition 4.3 (R2WDC) .
We say that a generative network G as in (2), satisﬁes the Range Restricted
Weight Distribution Condition (R2WDC) with constant ✏ > 0 if for all i = 1 , . . .
, d , and for allx, y, x 1 , x 2 , x 3 , x 4 2 R k , it holds that
[Equation]
+ ,s  Q r,s  u, v i   ✏ k u kk v k , where r = G i  1 ( x ) , s =
G i  1 ( y ) , u = G i  1 ( x 1 )  G i  1 ( x 2 ) , and v =
G i  1 ( x 3 )  G i  1 ( x 4 ) .
Notice that the R2WDC is weaker than the WDC.
Indeed, (8) and (9) are equivalent for i = 1 , but fori  2 equation (8) requires ( W i ) t + ,r ( W i )
+ ,s to be close to the matrix Q r,s for any vector r, s 2 R n i  1and when acting on any vector u, v 2 R n i  1 , while equation (9) requires ( W i ) t + ,r ( W i )
+ ,s to be closeto the matrix Q r,s only for vectors r, s on the range of G i  1 and when acting on vectors u, v 2 R n i  1given by the difference of points on the range of G i  1 .
Notice that contrary to (8), deﬁning the
R2WDC (9) for layer i requires considering the input/ouput pairs of the layers up to i  1 .
Our ﬁrst technical result provides theoretical guarantees for efﬁciently estimating a target signal y ?
onthe range of a generative network from few linear measurements under the RRIC and the R2WDC .
Theorem 4.4.
Suppose d  2 , and A and G satisfy the RRIC and the R2WDC with constant✏ < K 1 /d 90 .
Assume that k ⌘ k 2  K 2 k x ?
k 2 d 42 2 d/ 2 .
Let { x t } be the iterates generated by Algorithm 1 withloss function f cs , initial point x 0 2 R k \ { 0 } and step size ↵ = K
3 2 d d 2 .
Then there exists a number ofsteps T satisfying T  K 4 f ( x 0 )
d d 4 ✏ k x ?
k
such that
[Equation]
In addition, for all t  T , we have
[Equation]
where C = 1  78 ↵ 2 d 2 (0 , 1)
Here, K 1 , . . .
, K 7 are universal positive constants.
Remark 1.
The exponential factors 2 d appearing in the conditions and theses of the theorem areartifacts of the scaling of the weights of the generative network.
For example, the output G ( x ) of thenetwork scales like k x k 2 / 2 d/ 2 and the loss function f cs ( x ) as k x k 22 / 2 d (see for example Proposition
C.1).
Hence, for new constants K 0 2 , K 0 4 the bounds for ⌘ and T could be equivalently written ask ⌘ k 2  K 0 2 k y ?
k 2 /d 42 and T  K 0 4 f ( x 0 ) /
( d 4 ✏ k y ?
k 22 ) .
Choosing the weights of the network to be{ p 2 W i } i 2 [ d ] would remove the 2 d factors in the above theorem (and scale the deﬁnition of R2WDC).
This theorem shows that, despite the nonconvexity of the minimization problem (1), if the RRICand the R2WDC hold with constant ✏ , after T
= O ( ✏  1 ) number of iterations the iterates of thesubgradient descent method described in Algorithm 1 enter in a region of local convergence aroundx ? .
Moreover, after a large enough number of steps, G ( x t ) gives an estimate of the target signal y ?
up to the noise level O ( k ⌘ k ) .
Theorem 3.1 in [21] shows that Theorem 4.4 holds assuming that the RRIC and the WDC hold.
Ourﬁrst technical contribution is to show that the WDC in Theorem 3.1 of [21], can be relaxed into the
R2WDC.
Relaxing the WDC into the R2WDC, will enable the relaxing of the expansivity assumptionneeded to show that the WDC holds for Gaussian generative networks as we demonstrate in Section5.
We next describe the role of these deterministic conditions in the analysis of the problem (1).
Thefull proof of Theorem 4.4 is given in Appendix C.
4.1 Global landscape analysis via the R2WDC
The analysis of [18] and [21] follows the approach recent line of works that analyze the globallandscape geometry of non-convex optimization problems arising in statistical and signal recoveryproblems (see for example [36, 37, 14, 13] and [6] for an overview).
The analysis roughly consists oftwo steps:
i)
Showing that f cs ( x ) ⇡ f E ( x ) and @
f cs ( x ) ⇡ h x uniformly over x .
ii)
Analyzing the global properties of f E ( x ) and h x , and transfer them to f cs ( x ) and h x using theﬁrst step.
Here f E ( x ) and h x are continuous functions of x , corresponding to the expected value of f cs ( x ) and@ f cs ( x ) under Gaussian weights and measurement matrix A (see next section for details) and zero noise.
The RRIC and the WDC are used in [18] and [21] to obtain the uniform concentration in theﬁrst step, as well as directly proving convexity-like properties of @
f cs ( x ) in the vicinity of x ? .
To illustrate how the WDC and the R2WDC come into play, consider for simplicity the noiselesscase ⌘ = 0 .
Then at a point x 2 R k where G is differentiable, the gradient of f cs is given by
[Equation]
where ⇤ d,x and ⇤ d,x ? ar deﬁned in Section 2 and the approximation uses the fact that A satisﬁes
the
RRIC with respect to G .
Then if G satisﬁes the WDC we have that
[Equation]
where the last line used the WDC to control the concentration of ( W d ) T + ,G d  1 ( x ) ( W d  + ,G d  1 ( x ) and( W d ) T + ,G d  1 ( x ) ( W d  + ,G d  1 ( x ? ) .
The resulting terms are then controlled again applying the WDC tothe the other d  1 weights of G , so that proceeding by induction over d one obtains [Equation]
[Equation]
where ˜ h is a deterministic vector ﬁeld deﬁned in Appendix C.
In Appendix C we show that the R2WDC can be used to control directly the concentration of theterms
[Equation]
and
[Equation]
around their expectation (with respect to W d ) obtaining in this way
[Equation]
Then again applying the R2WDC to the other layers of G , we can show that (10) still holds.
We canthen borrow the analysis of h x from [21] and obtain the same convergence guarantees.
The advantage of using the R2WDC over the original WDC, is that it is satisﬁed by random generativenetworks with contractive layers as we demonstrate in the next section.
Recovery guarantees under probabilistic assumptions
In this section we give probabilistic models for the measurement matrix A , generative network G ,and noise vector ⌘ that will ensure that the RRIC and the R2WDC are satisﬁed with high probabilityand Algorithm 1 efﬁciently estimate the target signal y ?
up to an error of the order ˜ O
( p k/m k ⌘ k ) .
We make the following assumption on the sensing matrix A 2 R m ⇥ n .
Assumptions A.
A.1 A is independent from { W i } di =1 .A.
A has i.i.d.
N (0 , 1 /m ) entries.
A.3 There are sufﬁcient number of linear measurements:
[Equation]
(11) where
b C ✏ depends polynomially on ✏  1 .
Under Assumptions A, the measurement matrix satisﬁes the RRIC with respect to G with highprobability.
Lemma 5.1 (Consequence of Proposition 6 in [18]) .
Let Assumptions A be satisﬁed.
Then A satisﬁesthe RRIC with constat ✏ > 0 with respect to G , with probability at least
[Equation]
where ˆ  and ˆ c are positive universal constants.
Proof.
This result is proved in Proposition 6 in [18] for a number of measurements m satisfying [Equation]
Q dj =1 n j where C 0 ✏ depends polynomially on ✏ .
To imporove the lower bound onm to (11) it is enough to follow the proof of Proposition 6 in [18] and use the sharper upper bound onthe number of afﬁne subspaces in the range of a gnerative network given in Lemma D.1.
We then provide a probabilistic model for a generative network G : R k !
R n as in (2).
Assumptions B.
B.1 Each weight matrix
W i 2 R n i ⇥ n i  1 have i.i.d.
N (0 , 1 /n i ) entries.
B.2 The ﬁrst layer satisﬁes n 1  e C ✏ · k , and for any i = 2 , . . .
, d :
n
i
[Equation]
✏ k log
[Equation]
(12) where ˜ C ✏ depends polynomially on ✏  1 .
B.3 The { W j } dj =1 are independent.
Under Assumptions B, the generative network G satisﬁes the R2WDC .
Lemma 5.2.
Fix 0 <
✏ < 1 .
Consider a d -layer ReLU network G with weight matrices { W i } di =1 .
Assume that the { W i } di =1 satisfy Assumptions
B. Then G satisﬁes the R2WDC with constant ✏ with 277 probability at least
[Equation]
where c ✏ depends polynomially on ✏  1 and  is a positive absolute constant.
We ﬁnally conclude with some assumptions on the noise vector ⌘ 2 R m .
Assumption C.
The noise vector ⌘ is independent from A and the weights { W i } di =
The next lemma is used to bound the perturbation of the objective function f cs and its gradient due tothe presence of the noise term ⌘ .
These bounds are then used to show that Algorithm 1 leads to areconstruction of y ?
of the order O ( p k/m k ⌘ k ) .
Lemma 5.3.
Suppose G : R k !
R n satisﬁes the R2WDC with ✏ < 1 / (16 ⇡ d 2 ) 2 and d  2 .
Let
A 2 R m ⇥ n be a matrix with i.i.d.
entries N (0 , 1 /m ) and ⌘ 2 R m satisﬁes Assumption C. Let
[Equation]
Then with probability at least
[Equation]
for every x 2 R k we have that
[Equation]
14) if in addition G is differentiable at x we also have that [Equation]
[Equation]
Given the previous assumptions, we are now ready to state the main result of this section.
Theorem 5.4.
Suppose
d  2 , ✏ < K 1 /d 90 and !
k ⌘ k 2  K 2 k x ?
k 2 d 42 2 d/ 2 where ! is deﬁned in (13) .
Assume that A , G and ⌘ satisfy Assumptions A, B and C. Then with probability at least
[Equation]
where  , ˆ  and ˆ c are positive universal constants, the following holds.
Let { x t } be the iteratesgenerated by Algorithm 1 with loss function f cs , initial point x 0 2 R k \ { 0 } and step size ↵ = K
2 d d 2 .
There exists a number of steps T satisfying T  K 4 f
( x
[Equation]
d 4 ✏ k x ?
k
such that
[Equation]
In addition, for all t  T , we have
[Equation]
where C = 1  7 8 ↵ 2 d 2 (0 , 1)
Here, K 1 , . . .
, K 7 are universal positive constants.
Proof.
Combining Lemma 5.1, Lemma 5.2 and Theorem 4.4 we obtain Theorem 5.4 with !
= 1 andprobability at least
[Equation]
Inspecting the proof of Theorem 3.1 in [21], it is easy to see that if Lemma 5.3 holds, then theconclusions of Theorem 5.4 hold with !
given by (13) and probability at least (16).
Remark 2.
As for Theorem 4.4, the exponential factors 2 d are artifacts of the scaling of the weightsof the network.
Had the entries of W i been drawn from N (0 , 2 /n i ) the 2 d factors would not bepresent.
Remark 3.
Notice that 4 k log( en/ ( k + 1)   4 k log( n ) / log(2) for every n  2 .
Thus if for everyi = 1 , . . .
, d , it holds that
[Equation]
the conclusions of the theorem hold with nontrivial probability bounds.
In Appendix G we provide anexample of a generative network G with contractive layers satisfying both (12) and (17) .
Theorem 5.4 provides guarantees for the efﬁcient recovery of a signal y ?
in the range of a generativenetwork G from few noisy linear measurements, using a nonconvex (sub)gradient descent method.
Notice that the intrinsic dimension of the signal y ?
is k (the dimension of the latent space) and thenumber of measurements required m is proportional to k and information-theoretically optimal upto log factors in the widths of the network and polynomials in the depth.
Notice moreover, that upto these factors, the width n i of each layer of the network is also required to be linear in k .
Thisis necessary to ensure that each subnetwork G i : R k !
R n i is invertible, and it is weaker thanthe assumptions in the previous works that required n i to be linear in n i  1 in order to ensure theinvertibility of every single layer.
In Appendix H we empirically verify the predictions of Theorem 5.4, demonstrating how (a practicalvariant of) Algorithm 1 recover signals y ? in the range of non-expansive generative networks fromundersampled noisy measurements. We show that the recovery is linear in k/m and that in practicethe dependence on the depth d of the networks is milder than that predicted by our theory. We leavefor future works the establishing of sharper bounds in the depth d .
Limitations of the current and previous works on theoretical guarantees for signal recovery withgenerative networks are the Gaussian assumption on the weights and the absence of biases.
Importantdirections of future research are the inclusion of biases in the generative network and the departurefrom the Gaussian weights assumptions for more realistic probabilistic models.