PLUGIn-CS: A simple algorithm for compressive sensing with generative prior
Abstract
We consider the problem of recovering an unknown latent code vector under a known generative model from compressive measurements.
For a d -layer deep generative network G : R n 0 → R n d with ReLU activation functions and compressive measurement matrix Φ ∈ R m × n d , let the observation be Φ G ( x ) + (cid:15) where (cid:15) is noise.
We introduce a simple novel algorithm, Partially Linearized Update for
Generative Inversion in Compressive Sensing (PLUGIn-CS), to estimate x (and thus G ( x ) ).
We prove that, when sensing matrix and weights are Gaussian, if layer widths n i (cid:38) 5 i n 0 and number of measurements m (cid:38) 2 d n 0 (both up to log factors), then the algorithm converges geometrically to a (small) neighbourhood of x with high probability.
Note the inequality on layer widths allows n i > n i +1 when i ≥and thus allows the network to have some contractive layers.
After a sufﬁcient number of iterations, the estimation errors for both x and G ( x ) are at most in the order of (cid:112) 4 d n 0 /m (cid:107) (cid:15) (cid:107) .
Numerical experiments on synthetic data and real data are provided to validate our theoretical results and to illustrate that the algorithm can effectively recover images from compressive measurements.
Introduction
We consider the inverse problem of recovering an unknown structured vector z ∗ ∈ R N from a noisy compressive observation y ∈ R m of the form
[Equation] where (cid:15) ∈
R m is noise, Φ ∈ R m × N is the compressive measurement matrix.
Traditional approaches for solving (1) often use priors on the signal z ∗ , for example, a sparsity prior with respect to a ﬁxed basis or dictionary [1, 2, 3].
An emerging viewpoint is to use a generative prior that assumes the signal z ∗ is in the range of a known deep generative model G : R n 0 → R N , i.e., z ∗ =
G ( x ∗ ) for some x ∗ ∈ R n 0 .
We assume G has the form
[Equation]
where σ ( · ) = max( · , 0) is the ReLU activation function and A i ∈ R n i × n i − 1 is the weight matrix in the i -th layer (and n d = N ).
One can then develop algorithms that can estimate the latent code vector x ∗ from Φ G ( x ∗ )
+ (cid:15) , thus recovering G ( x ∗ ) .
Recent advancements in training deep neural networks have shown that generative priors can effectively map low dimensional vectors to the space of natural image classes [4, 5, 6].
Learned generative models can then be used as priors to solve various inverse problems including denoising [7, 8], compressive sensing [9, 10, 11, 12, 13, 14], phase retrieval [15], blind deconvolution [16, 17], low-rank matrix recovery [18] and have been shown to perform on par or outperform classical sparsity based approaches for these inverse problems.
For example, in [7] the authors empirically showed that an end-to-end approach for denoising using a neural network that maps noisy patches in an image to noise-free ones achieves state-of-the-art performance and is on par with BM3D. Similarly, in [9] the authors empirically showed that for compressive sensing using generative prior, optimization of the empirical risk objective over the latent code space (of the generative prior) can recover a vector that effectively estimates the uncompressed signal with 5-10 times less measurements compared to Lasso in some cases.
Given that y equals Φ G ( x ∗ ) , with possibly some additive noise, a standard way to estimate x ∗ would be to look for a minimizer of the program
[Equation]
Unfortunately, this program is non-convex and to our knowledge there is no known efﬁcient method that can achieve its global minimum in general.
On the other hand, in the case with random weight matrices and sensing matrix, a line of papers showed that gradient-based algorithms can provably avoid local minima with high probability [10, 12, 15].
In particular, [12] considers a model with small noise, Gaussian measurement matrix Φ , and Gaussian weight matrices
A i which are highly expansive at each layer.
Under these conditions, the authors show that the latent code vector x ∗ can be accurately estimated if m (cid:38) dn 0 (up to log factors) using a gradient-based method that uses the (sub-)gradient updates given by
[Equation]
(4) where x k is the k -th estimate, η ∈ R is step size, and D j is a diagonal matrix with entries that are either zero or one.
Each D j zeros out the inactive rows of A j with respect to the estimate x k and so it is a function of x k (and A p for p < j ).
Thus, at each iteration all D j need to be updated.
In this paper, we show that one can drop all D j and still recover an accurate estimate of x ∗ .
This result follows from our previous work in the denoising case ( Φ = I n d ) [19], where we introduced a novel algorithm called Partially Linearized Update for Generative Inversion (PLUGIn).
For the compressive sensing case, we propose the following iterative algorithm to estimate x ∗ :
[Equation]
This algorithm was inspired by previous work showing that latent vectors for non-linear single-index function can be approximately estimated by treating the function as linear [20, 21].
Similar to [8, 9, 10, 12, 15, 17, 22], for theoretical analysis, we assume the weight matrices are Gaussian.
To show that the algorithm works more broadly, we conduct real data simulations.
Applying the ideas in [20, 21] one can show that for any ﬁxed x 0 , the ﬁrst iteration of PLUGIn-CS provides an unbiased estimate of x ∗ with η = 2 d , which is generally not the case for the gradient descent estimates given by (4).
Additionally, each iteration of PLUGIn-CS maps the difference Φ G ( x k ) − y to the low dimensional latent code space using a static matrix A (cid:124) 1 A (cid:124) 2 . . .
A (cid:124) d
Φ (cid:124) , which can be pre-multiplied and reused in subsequent iterations.
Building upon the theory for PLUGIn [19], we show that the estimates provided by PLUGIn-CS converge geometrically to a neighbourhood of x ∗ (and also G ( x k ) to a neighbourhood of G ( x ∗ ) ) with high probability.
This result holds with the following assumptions:
A1.
Each A i ∈ R n i × n i − 1 has i.i.d.
N (0 , 1 /n i ) entries and { A i } i ≤ d are independent.
A2.
Layer widths (number of nodes in each layer) satisfy
[Equation]
for some (sufﬁciently large) absolute constant C 0 .
A3.
The measurement matrix Φ ∈
R m × n d has i i
d N (0 , 1 /m )
entries (independent from weight matrices) with
[Equation]
for some absolute constant c 0 .
A4.
The noise (cid:15) does not depend on { A i } i ≤ d or Φ .
(The noise may be deterministic or random.)
Note that A2 allows n i > n i +1 for i ≥ 1 and thus can provide theoretical guarantees even when
G has some contractive layers.
Under these assumptions, PLUGIn-CS algorithm converges to a neighbourhood of x ∗ for a range of step sizes near 2 d .
Precisely, we have the following theorem.
Theorem 1.
Let θ ∈ (0 , 43 ) and let α = | 1 − θ | + 12 θ ∈ (0 , 1) .
Let R be a positive number such that (cid:107) x 0 − x ∗ (cid:107) ≤ R .
Under assumptions A1-A4, the k -th estimate x k given by PLUGIn-CS with constant step size η = θ 2 d satisﬁes
[Equation]
with probability at least 1 − 2( k + 4) e − 10 n 0 .
When θ = 1 , Theorem 1 reduces to the following corollary.
Corollary 1.
Let R be a positive number such that (cid:107) x 0 − x ∗ (cid:107) ≤ R .
Under assumptions A1-A4, the k -th estimate x k given by PLUGIn-CS with constant step size η = 2 d satisﬁes
[Equation]
, and
[Equation]
(cid:15) (cid:107) with probability at least 1 − 2( k + 4) e − 10 n 0 .
Remark 1 (Contractive layers) .
In A2, (5) states a lower bound on n i with respect to the latent code dimension n 0 (up to log factors).
While this bound strictly increases with layer depth i , it is not necessary for n i to always increase with i (except in the ﬁrst layer).
For example, consider n i = βC {+
+} 5 d n 0 d (2 d − i ) where β is any ﬁxed number such that βC 0 ∈ N and β ≥ 4 + log C 0 .
It is easy to see n 1 > n 2 > · · · > n d , and we can also verify (see Appendix E) that such n i satisfy (5).
In this case, the network is contractive in each layer after the ﬁrst, and Theorem 1 still applies.
Remark 2 (Initialization may depend on random weight matrices and sensing matrix) .
The results of the theorem can still hold when x 0 is chosen randomly, dependent on the weight matrices
A i .
In this case, suppose that (cid:107) x 0 − x ∗ (cid:107) ≤ R with probability at least 1 − δ .
Then, the error bounds hold with probability at least 1 − 2( k + 4) e − 10 n 0 − δ .
This does not follow directly from the theorem as stated (which ﬁxes x 0 , then takes random weight matrices), but follows from the proof.
Remark 3 (Comparison to guarantees for gradient-based method) .
Here we compare our results to the ones in [12], which uses (4) for iterations and considers a model with small noise, i.e., (cid:107) (cid:15) (cid:107) (cid:46) (cid:107) x ∗ (cid:107) 2 d 42 2 d/ 2 .
They show that when the weight matrices and sensing matrix are Gaussian with weight matrices sufﬁciently expansive at each layer, the iterates of the gradient-based method converge to a neighborhood of the target signal x ∗ .
After sufﬁciently many iterations N , the iterates converge geometrically to a neighborhood of x ∗ of radius at most on the order of 2 d/ 2 (cid:107) (cid:15) (cid:107) .
This rate of convergence takes the form (1 − C/ 2 d ) , thus giving slower convergence for deeper nets.
On the other hand, we note that dependence on d is of relatively minor concern.
Generative models usually have small depth in practice, our MNIST experiments (below) work well with depth 3, and typical applications use depth less than 8.
In comparison, Theorem 1 holds for any noise (cid:15) that does not depend on { A i } i ≤ d or Φ .
Under similar randomness assumptions, the iterates of PLUGIn-CS converge to a neighborhood of the latent code x ∗ of radius at most on the order of 2 d (cid:112) n 0 /m (cid:107) (cid:15) (cid:107) .
This result can hold for networks with contractive layers and the rate of convergence is geometric starting at the initial iterate of PLUGIn-CS.
Numerical Experiments
In this section, we provide numerical experiments on synthetic data and MNIST images where the oberservations follow the model in (1).
All experiments were conducted using Google Colaboratory.
In the synthetic experiments, we let the generative prior be a 2-layer neural network G ( z )
=
σ ( A 2 σ ( A 1 x ))) , where the entries of weight matrix A i ∈ R n i × n i − 1 are sampled from N (0 , 1 /n i ) .
[Figure]
We sample the target latent code x ∗ uniformly from S n 0 − 1 , set the noise level as α ∈ R , and set the noise to be αν where ν is sampled uniformly from S n 3 − 1 .
Then we set y =
Φ G ( x ∗ ) + αν , where the entries of the compressive measurement matrix Φ ∈ R m × n 2 are sampled from N (0 , 1 /m ) .
We run PLUGIn-CS and gradient descent each for 10,000 iterations or until the relative successive error is less than 10 − 13 , and set ˆ x to be the output.
We use a ﬁxed step size of 3 and 10 for PLUGIn-CS and gradient descent, respectively, with the gradient computed using PyTorch [23].
For the ﬁrst experiment, we ﬁx n 0 = 10 , n 1 = 400 , n 2 = 300 , m = 150 , and sample the noise level α uniformly in the interval [0 , 1] .
In ﬁgures 1a and 1b, the solid line corresponds to the performance of PLUGIn-CS and the dotted line represents the performance of gradient descent.
Figure 1a shows the empirical dependence of the the relative recovery error (cid:107) ˆ x − x ∗ (cid:107) / (cid:107) x ∗ (cid:107) on the noise-to-signal ratio given, given by α , from 20 independent trials.
Similarly, ﬁgure 1b shows the empirical dependence of the the relative reconstruction error (cid:107)G (ˆ x ) −G ( x ∗ ) (cid:107) / (cid:107)G ( x ∗ ) (cid:107) from 20 independent trials.
The ﬁgures show that PLUGIn-CS can stably solve the compressive sensing problem (1) with a generative prior.
For the second experiment, we ﬁx α = 0 , n 1 = 250 , n 2 = 700 , m = 150 , and sample the latent code dimension n 0 in the interval [1 , 100] .
In ﬁgures 1c, the solid line corresponds to the performance of
PLUGIn-CS and the dotted line represents the performance of gradient descent.
Figure 1c shows the empirical success probability from 20 independent trials.
We now empirically show that PLUGIn-CS can effectively recover MNIST images from compressive measurements and compare its performance to gradient descent.
We trained a VAE [24] using Adam optimizer [25] with a learning rate of 0.001 and mini-batch size 100 on the MNIST dataset [26].
The decoder network in the VAE is a fully connected network with parameters 20 − 500 − 500 − 784 .
The compressive sensing matrix Φ ∈ R m × 784 follows i.i.d N (0 , 1 /m )
entries with m = 150 and the observation y satisﬁes y =
Φ z ∗ , where z ∗ is an image from the MNIST database.
In all MNIST experiments, we use a ﬁxed step size of η = 1 /γ for PLUGIn-CS, where γ is the product of the operator norms of the weight matrices; for gradient descent, we use a ﬁxed step size of 1000.
Similar to the synthetic experiment, we run PLUGIn-CS and gradient descent each for 10,000 iterations or until the relative successive error is less than 10 − 13 .
In ﬁgure 1d, the images in the top row are the observations, the images in the second row and third row are the recovered images corresponding to
PLUGIn-CS and gradient descent, respectively.
Acknowledgments and Disclosure of Funding
Y. Plan is partially supported by an NSERC Discovery Grant (GR009284), an NSERC Discovery Accelerator Supplement (GR007657), and a Tier II Canada Research Chair in Data Science (GR009243).
O. Yılmaz is partially supported by an NSERC Discovery Grant (22R82411) and Paciﬁc Institute for the Mathematical Sciences (PIMS)
CRG 33: High-Dimensional Data Analysis. B. Joshi is partially supported by the Paciﬁc Institute for the Mathematical Sciences (PIMS). (The research and ﬁndings may not reﬂect those of the Institute.) Notations in proofs
For a positive integer n , let [ n ] = { 1 , 2 , . .
. , n } .
For a vector x , let (cid:107) x (cid:107) be its Euclidean norm; for a matrix A , let (cid:107) A (cid:107) be its operator norm; for a matrix A and a set T , let (cid:107) A (cid:107) T :=
sup x ∈T \{ 0 } (cid:107) Ax (cid:107) (cid:107) x (cid:107) .
Let B ( x, r ) be the Euclidean ball of radius r centered at x and let B n (0 , r ) be the Euclidean ball in R n with radius r , centered at origin.
We use C and c to denote absolute constants (often c for small ones and C for large ones) which may vary from line to line.
We also use c 0 , C 0 , C 1 , etc., to denote particular absolute constants, which do not change throughout the paper.
We use P A i to denote that the probability is taken only with respect to A i .
In neural network
G : R n 0 → R n d , let G i : R n 0 → R n i be the mapping that corresponds to the ﬁrst i layers, i.e.
G i ( x ) =
σ ( A i . . .
σ ( A 1 x ) . . . ) .
For its weight matrices, let ˜ A
= I n 0 and ˜
A
i =
A i
A
i − 1 · · · A 1 for i ∈ [ d ] .
For x ∈ R n 0 , denote x 0 = x and x i =
G i ( x ) for i ∈ [ d ]
A Proof Outline
Our proofs for PLUGIn-CS builds upon the proofs for PLUGIn [19].
Here we include all the proofs for completeness, and note that many parts of these proofs are the same as in [19].
The main differences are the parts dealing with sensing matrix Φ .
In particular, we added Lemma 9 and modiﬁed Lemma 5, Lemma 6 as well as Proof of Theorem 1 to incorporate Φ in the new proofs.
Below we give a sketch for the proof of Theorem 1.
For simplicity, we will only focus on analyzing one iteration of PLUGIn-CS with step size η = 2 d .
The complete proof can be found in Appendix D.
A Special Case
Let us ﬁrst look at the special case where d = 1 , (cid:15) = 0 and Φ
= I .
The analysis here highlights some of the key ideas in our proofs, while its result Lemma 1 serves as a building block for proof in the general case.
In this special case, PLUGIn-CS with η = 2 d reduces to
[Equation]
where σ = ReLU and A ∈ R m × n is random with i.i.d.
N (cid:0) 0 , 1 m (cid:1) entries.
In fact, the ﬁrst iterate provides an unbiased estimate of x ∗ when x 0 does not depend on A .
Indeed, the rotation invariance property of the Gaussian distribution may be leveraged to show [20, 21], for any ﬁxed x ,
[Equation]
For completeness, we also include a proof for (7) in Appendix B, Lemma 2.
Applying (7) to the ﬁrst iteration gives
[Equation]
Thus, even the ﬁrst iterate can be shown to be a good estimate by showing that x 1 concentrates around its mean.
Further iterates are generally no longer unbiased estimators because they pick up complex dependence on the random matrix A .
We overcome this by developing a series of uniform deviation inequalities, as below.
Let us suppose we have shown that, with high probability, (cid:107) x k − x ∗ (cid:107) ≤ r for some (small) constant r > 0 .
Then we wish to show that (cid:107) x k +1 − x ∗ (cid:107) ≤ r/ 2 with high probability.
Notice that − ( x k +1 − x ∗ ) =
[Equation]
where
[Equation]
We wish to bound the supremum of random process Z ( u, x k ; x ∗ ) over u ∈ S n − 1 .
However, this process is challenging to analyze since x k depends on A when k ≥ 1 .
To alleviate this dependency, we bound by the supremum of Z ( u, v ; x ∗ ) over ( u, v ) ∈ T 0 :
= B n (0 , 1) × B ( x ∗ , r ) instead.
It is worth noting that Z ( u, v ; x ∗ ) is centred, namely E Z ( u, v ; x ∗ ) = 0 for any ﬁxed ( u, v ) .
We now arrive at the estimate
[Equation]
The following Lemma 1 provides a bound on sup T
Z ( u, v ; x ∗ ) .
In fact, it is slightly more general because we replaced T 0 with T 1 × T 2 (this replacement is helpful when studying the general case d > 1 ).
The complete proof of this lemma can be found in Appendix C.
The proof idea is to ﬁrst establish that Z ( u, v ; x ∗ ) has mixed (sub-Gaussian and sub-exponential) tail increments through
Bernstein’s inequality, and then apply the result from [27], which provides a general bound for the supremum of random processes with mixed tail increments.
Lemma 1.
Let σ = ReLU .
Fix w ∈ R n and let A ∈ R m × n have i.i.d.
N (cid:0) 0 , 1 m (cid:1) entries.
Deﬁne
[Equation]
Suppose T 1 , T 2 are sets (not depending on A ) such that
[Equation]
and
[Equation]
for some q -dimensional (afﬁne) subspaces S 1 , S 2 ⊆ R n and real numbers α, r > 0 .
Then for any t ≥ 1 ,
[Equation]
with probability at least 1 − e − t .
Here C 1 > 0 is an absolute constant.
We can apply Lemma 1 to estimate (8) (with S 1 = S 2 = R n ) and get, for example,
[Equation]
with probability at least 1 − e − n , provided that m ≥ (16 C 1 ) 2 n .
The General Case
Let us illustrate the proof idea with d = 2 (the extension to d > 2 is straightforward).
Denote x ki = G i ( x k ) and x ∗ i =
G i ( x ∗ ) for i = 1 , 2 .
By adding and subtracting 2 A (cid:124)
( x k 1 − x ∗ 1 ) and [Equation] A[-(cid:124) 
-]
[Equation] 1 A (cid:124) 2 ( x k 2 − x ∗ 2 ) , we can write PLUGIn-CS with η = 2 d as
[Equation]
Similar to the special case above, we can get
[Equation]
where (denote x ∗
= x ∗ )
Z
[Equation]
Also assume that (cid:107) x k − x ∗ (cid:107) ≤ r , it remains to bound each term on the right hand side of (9).
The ﬁrst term can be bounded directly through Lemma 1 (with t = 10 n 0 ).
The last term is also easy to bound by the randomness of A i (Appendix D, Lemma 6), in which case we have (cid:107) A (cid:124)
A (cid:124) 2 (cid:15) (cid:107) ≤ 15 (cid:112) n 0 /m (cid:107) (cid:15) (cid:107) with high probability.
For the second term, ﬁrst notice that range( A 1 ) is a n 0 -dimensional subspace in R n 1 .
Using the ideas from [9, 28], we can also show that range( G 1 ) is contained in a union of N many n 0 -dimensional (afﬁne) subspaces, where N ≤ ( en 1 /n 0 ) n 0 .
Furthermore, let E be the event such that mappings
A 1 , A 2 A 1 , G 1 , G all have Lipschitz constants being at most 3 , then we can show (Appendix D,
Lemma 8) that P ( E ) ≥ 1 − 3 e − 10 n 0 .
Also on event E (note that (cid:107) A 1 (cid:107) ≤ 3 and (cid:107) x k 1 − x ∗ 1 (cid:107) ≤ 3 r ), we have
[Table] where S 1 and S 2 ,j are n 0 -dimensional (afﬁne) subspaces. Applying Lemma 1 on each T 1 × T 2 ,j , followed by a union bound over j ∈ [ N ] , we get (denote T 2 = ∪ j ∈ [ N ] T 2 ,j )
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
with probability (over A 2 and conditioning on A 1 ) at least 1 − Ne − t .
By choosing t = 2 n 0 log( en 1 /n 0 ) , we obtain a high probability bound for sup u ∈ S n 0 − 1 Z 2 ( A 1 u, x k 1 ) .
For the third term, use the fact that (cid:107) A 2 A 1 (cid:107) ≤ 3 and (cid:107) x k 2 − x ∗ 2 (cid:107) ≤ 3 r on E , together with Lemmawe can obtain a high probability bound for (cid:107) A 2 A 1 (cid:107)(cid:107) ( I − Φ (cid:124) Φ)
( x k 2 − x ∗ 2 ) (cid:107) .
Finally, if C 0 and c 0 are sufﬁciently large, we can thus show from (9) that, with high probability, (cid:107) x k +1 − x ∗ (cid:107) ≤ 1 2 (cid:16) r + 30 · 2 2 (cid:112) n 0 /m (cid:107) (cid:15) (cid:107) (cid:17) .
B Some Results on Gaussian Matrices
Here we state some results on Gaussian Matrices, which will be used in the proofs later.
Lemma 2 ([20, 21]) .
Let σ : R → R be a positively homogeneous activation function.
Let
A ∈ R m × n have i.i.d.
N (cid:0) 0 , 1 m (cid:1) entries.
Then for any x ∈ R n ,
[Equation]
where λ :=
E g · σ ( g ) with g ∼ N (0 , 1) .
In particular, λ = 12 when σ is ReLU.
Proof.
Since σ is positively homogeneous, we can assume (without loss of generality) x ∈ S n − 1 .
Denote by a (cid:124)
j the j -th row of A .
Then
[Equation]
x
[Equation]
a where a := √ ma 1 ∼ N (0 , I n ) .
Take an orthogonal matrix U such that Ux = (cid:107) x (cid:107) e 1 = e 1 where e 1 = (1 , 0 , . .
. , 0) (cid:124) .
Note that by rotation invariance for standard Gaussian, Ua and a have the same distribution N (0 , I n ) , thus
[Equation]
The following theorem is the concentration of (Gaussian) measure inequality for Lipschitz functions.
Here we only state a one-sided version, though it is more commonly stated with a two-sided one, i.e.,
P
[Equation]
Theorem 2.
Let f : R n → R be a Lipschitz function with Lipschitz constant L f .
Let g ∈ R n be a random vector with independent N (0 , 1) entries.
Then, for all t > 0 ,
[Equation]
A proof of Theorem 2 can be found in [29, Chap. 8].
Based on this theorem, it is easy to prove the following results.
Lemma 3.
Let A ∈ R m × n have i.i.d.
N (0 , 1) entries.
(a) For any ﬁxed point s ∈ R n , we have
[Equation]
(b) For any ﬁxed k -dimensional subspace S ⊆ R n , we have
[Equation]
and
[Equation]
Proof.
(a) Without loss of generality, assume (cid:107) s (cid:107)
= 1 .
Then As ∼ N (0 , I m ) and by Jensen’s inequality, E (cid:107) As (cid:107) ≤ (cid:112) E (cid:107) As (cid:107) 2 = √ m .
The result follows immediately from Theorem 2 (with f ( g )
= (cid:107) g (cid:107) and g = As ).
(b) Let U be an orthogonal matrix
such that U (cid:124) S
= span { e 1 , . . .
e k } =: S 0 , then (cid:107) A (cid:107) S
= (cid:107) AU (cid:107) S 0 .
Also, since AU has the same distribution as A (by rotation invariance), we get
[Equation]
Notice that (cid:107) A (cid:107) S 0 is the operator norm for a particular sub-matrix (obtained by taking ﬁrst k -columns) of A , so without loss of generality, we can assume k = n .
Let f ( A )
= (cid:107) A (cid:107) .
Since | f ( A ) − f ( A (cid:48) )
| ≤ (cid:107) A − A (cid:48) (cid:107) F , f is 1 -Lipschitz when viewed as a mapping from R mn to R .
By Theorem 2,
[Equation]
The one-sided result follows since √ m − √ n ≤ E (cid:107) A (cid:107) ≤ √ m + √ n (see, e.g., [30, Section 7.3]).
The two-sided result follows by also considering f ( A )
= −(cid:107)
A (cid:107) .
C Preliminaries and Proof for LemmaPreliminaries
For α ≥ 1 , the ψ α -norm of a random variable X is deﬁned as
[Equation]
We say X is sub-Gaussian if (cid:107) X (cid:107) ψ 2 < ∞ and sub-exponential if (cid:107) X (cid:107) ψ 1 < ∞ .
The ψ 2 and ψnorms are also called sub-Gaussian and sub-exponential norms respectively.
Loosely speaking, a sub-Gaussian (or a sub-exponential) random variable has tail dominated by the tail of a Gaussian (or an exponential) random variable.
For independent, mean zero, sub-exponential random variables X 1 , . . .
, X m , their sum concentrates around zero.
In particular, the following Bernstein’s Inequality [30, Section 2.8] holds:
[Equation]
The above inequality also suggests that (cid:80) mi =
1 X
i has a mixed tail, i.e., a tail consisting of both a sub-Gaussian part and a sub-exponential part.
In our proof, we will use the following result from generic chaining for mixed tail processes.
Theorem 3 (Theorem 3.5 [27]) .
If ( X t ) t ∈ T has a mixed tail with respect to metric pair ( d 1 , d 2 ) , i.e.
[Equation]
Then there are constants c, C > 0 such that for any u ≥ 1 ,
[Equation]
Here t 0 is any ﬁxed point in T , γ α ( T, d ) is the γ
α -functional and ∆ d i is the diameter given by ∆ d i ( T ) =
sup s,t ∈ T d i ( s, t ) .
The γ α -functional of ( T, d ) is deﬁned as
[Equation]
where the inﬁmum is taken with respect to all admissible sequences.
A sequence ( T n ) n ≥ 0 of subsets of T is called admissible if | T 0 | = 1 and | T n | ≤ 2 2 n for all n ≥ 1 .
For our proof, we will use the following estimate on γ α ( T, d ) , which involves the generalized
Dudley’s integral [31, 27].
[Equation]
where C ( α ) is a constant depending only on α and N ( T, d, ε ) is the covering number , i.e., the smallest number of balls (in metric d and with radius ε ) needed to cover set T .
Proof for Lemma[-
-]
We recall the statement of Lemma 1 below.
Lemma 1.
Let σ = ReLU .
Fix w ∈ R n and let A ∈ R m × n have i.i.d.
N (cid:0) 0 , 1 m (cid:1) entries.
Deﬁne
[Equation]
Suppose T 1 , T 2 are sets (not depending on A ) such that
T
[Equation]
, α ) and
T
[Equation]
S
[Equation]
∩ B
[Equation]
w, αr ) for some q -dimensional (afﬁne) subspaces S 1 , S 2 ⊆ R n and real numbers α, r > 0 .
Then for any t ≥ 1 ,
[Equation]
with probability at least 1 − e − t .
Here C 1 > 0 is an absolute constant.
Proof.
First, we establish that Z ( u, v ; w ) has a mixed tail.
Let a (cid:124) i be the i -th row of A , then a i ∼ N (0 , I n /m ) .
For u ∈ B n (0 , α ) and v ∈ B ( w, αr ) , deﬁne random variables [Equation] i
[Equation]
We have E Z iu,v = 0 by Lemma 2, and
[Equation]
12 (cid:104) u, v − w (cid:105) =
Z ( u, v ; w ) .
For the increments of Z iu,v , we have
Z
[Equation]
−
[Equation]
a
[Equation]
u −
[Equation]
[Equation]
(cid:124) i w
[Equation]
[Equation]
(cid:104)
u − u (cid:48) , w (cid:105)
[Equation]
We can estimate its sub-exponential norm from Lemma
[Equation]
which gives
[Equation]
By Bernstein’s inequality,
[Equation]
where the metrics d i are given by
[Equation]
Therefore ( Z u,v ) ( u,v ) ∈T has a mixed tail with respect to the metric pair ( Cd 1 , Cd 2 ) for some absolute constant C .
Next, we bound the supremum of Z ( u, v ; w ) .
Without loss of generality, we will assume that q ≥ 1 . (In fact, if q = 0 , then T 1 , T 2 are either empty set or singleton, in which case the result is trivial or follows directly from Bernstein’s inequality).
Denote T :=
T 1 × T 2 and deﬁne a metric d on T as
[Equation]
It is easy to see that d 2 = α √ m d and d 1 = αm d .
Also note that γ i ( T , td )
=
tγ i ( T , d ) from deﬁnition (10).
We can assume that S 1 is a subspace 1 , then Z 0 ,v = 0 for v ∈ T 2 .
Thus by Theorem 3, we have
[Equation]
with probability at least 1 − e − t .
It remains to estimate γ i ( T , d ) .
From (11) we have
[Equation]
Let d (cid:96) 2 be the Euclidean metric.
Note that one can always obtain a ε -covering on T (with metric d ) from the product set of a ε/ 2 -covering on T 1 (with metric rd (cid:96) 2 ) and a ε/ 2 -covering on T 2 (with metric d (cid:96) 2 ).
Moreover, note that T 1 is contained in a q -dimensional ball of radius α and T 2 is contained in a q -dimensional ball of radius αr .
Hence
[Equation]
Here the last line uses estimate N ( B q , d (cid:96) 2 , ε ) ≤ (cid:0)
+ 2 ε (cid:1) q for the covering number of unit balls (see e.g., [30, Section 4.2]).
Note the estimate 2 (cid:82) a 0 log (cid:0) 2 ax (cid:1) dx
= a (log 2 + 1) <
2 a , we get γ 1 ( T , d ) ≤
[Equation]
3 (cid:90) 4 αr
[Equation]
dε ≤ 16 C 3 αrq.
Also note the inequality (cid:112) log(1 + x ) < √ 2 log(1 + x ) for x ≥ 1 , we have
[Equation]
Therefore with probability at least 1 − e − t ,
[Equation]
(cid:32)(cid:114) q
[Equation]
q
[Equation]
Lemma 4.
Let σ = ReLU .
For u, x, y ∈ R n and g ∼ N (0 , I n ) , the (mean zero) random variable
Z
[Equation]
has sub-exponential norm (cid:107) Z g (cid:107) ψ 1 ≤ C 2 (cid:107) u (cid:107)(cid:107) x − y (cid:107) , where C 2 is an absolute constant.
Proof.
It is easy to see that Z g is mean zero from Lemma 2.
Also from the following two properties of ψ 1 , ψ 2 -norms (see [30, Section 2.7]):
[Equation]
and
[Equation]
we have (note that σ is 1 -Lipschitz)
[Equation]
The result follows by noting that (cid:107) (cid:104) g, u (cid:105) (cid:107) ψ 2 = (cid:107) g 1 (cid:107) ψ 2 (cid:107) u (cid:107) where g 1 ∼ N (0 , 1) .
D Proof for Theorem
Proof of Theorem 1.
First we write
[Equation]
∗ ) . For any ﬁxed r > 0 , using triangle inequality and Lemma 5 (with events E i deﬁned as in Lemma 5) we can conclude that if (cid:107) x k − x ∗ (cid:107) ≤ r , then with probability at least 1 − P ( E 1 ) − P ( E [Equation]
− P ( E
[Equation]
− 2 e − 10 n 0 ,
[Equation]
where
[Equation]
Now deﬁne a sequence { r k } k ∈ N such that r k +
=
α ( r k + βε ) and r 0 = R .
We can ﬁnd its general formula as follow:
[Equation]
Next, by induction on k (i.e., apply (12) with r = r k for k = 0 , 1 , 2 , . . . )
we get
[Equation]
Notice that the events E 1 , E 2 , E 3 remain unchanged throughout iterations, so (13) holds with probability at least 1 − P ( E 1 ) − P ( E 2 ) − P ( E 3 ) − 2 ke − 10 n 0 .
Lastly, from Lemma 6, Lemma 8 and Lemma 9 we know P ( E i ) ≤ 3 e − 10 n 0 for i = 1 ,
and
P ( E 3 ) ≤ 2 e − 10 n 0 .
Also, (cid:107)G ( x k ) − G ( x ∗ ) (cid:107) ≤ 3 (cid:107) x k − x ∗ (cid:107) on E c 2 .
This completes the proof.
Lemma 5.
Fix r > 0 and assume assumptions A1-A4 hold.
If (cid:107) x k − x ∗ (cid:107) ≤ r , then after one iteration according to PLUGIn-CS with step size η = 2 d , we have
[Equation]
with probability at least 1 − P ( E 1 ) − P ( E 2 ) − P ( [Equation] − [Equation]
Here E 1 , E 2 , E 3 are the events
[Equation]
L
[Equation]
L G
[Equation]
[Equation] 3 for all i ∈ [ d ] (cid:9) and E
[Equation]
where L G i and L ˜
A i denote the Lipschitz constants of G i ,
˜ A
i : R n 0 → R n i respectively [Equation] and
[Equation]
with R :=
range( G ) − range( G ) being the Minkowski sum of range( G ) and − range( G ) .
Proof.
For x ∈ R n
[Equation]
denote x 0 = x and x i =
G i ( x ) for i ∈ [ d ] .
Then
[Equation]
thus we can write
[Equation]
≤ I + II + III , where
I :=
[Equation]
[Equation]
with
[Equation]
We will estimate I , II and III as below. bound for I
On event E c 2 , ∀ i ∈
[ d − 1] we have
[Equation]
By Lemma 7, there are N G i many n 0 -dimensional afﬁne subspaces {S i,j } such that
T i 2 ⊆ ∪ j ∈ [ N G i ] T i 2 ,j where T i 2 ,j = S i,j ∩ B ( x ∗ i , 3 r ) ⊆
R n i and N G i ≤ ψ i := i (cid:89)
j = (cid:18) en j n(cid:19)
n.
For i ∈ [ d − 1] , apply Lemma 1 on T i
× T i 2 ,j followed by a union bound over j ∈ [ N G i ] , we get sup
T i 1 ×T i
Z i +1 ( u, v ) ≤ C [Equation] (9 r ) (cid:18)(cid:114) nn i +
+ nn i [-+
-]+
[Equation]
(cid:114) t i + n i +1 +
[Equation]
+
[Equation]
n i + (cid:19) with probability (over A i +1 and conditioning on { A j } j ∈
[ i ] ) at least 1 − ψ i e − t
i +1 .
Choose t
i +
= 2 log ψ i = 2 n 0 (cid:80) ij =1 log( en j n 0 ) , then we get
P A i + (cid:32) sup
T i 1 ×T i
Z i +1 ( u, v ) ≤ 9 C 1 r ·(cid:115) 2 log ψ
i n i + (cid:33) ≥ 1 − e − log ψ i , ∀ i ∈
[ d − 1] .
Also for i = 0 , applying Lemma 1 on B n 0 (0 , 1) × B ( x ∗ , r ) , we get sup u ∈ B n 0 (0 , 1) v ∈ B ( x ∗ ,r )
Z 1 ( u, v ) ≤ C 1 r ·(cid:114) 10 nnwith probability (over A 1 ) at least 1 − e − 10 n 0 .
Therefore under assumption A2 (with C 0 ≥ 160 · 144 2 C 21 ), we have d − 1 (cid:88) i
= 2 i + sup u ∈ S n 0 − 1 Z i + (cid:16) ˜
A i u, x ki (cid:17) ≤ r 144 + d − 1 (cid:88) i =
2 i +1 · 36 r (cid:114) 160 · 5 i + =
r 144 + r 4 ·  d − 1 (cid:88)
i = (cid:18) 2 √ (cid:19)
i < r 4 ·  ∞ (cid:88) i = (cid:18) 2 √ (cid:19)
i < r with probability at least 1 − P ( E 2 ) − e − 10 n 0 − (cid:80) d − 1 i =1 e − log ψ i .
Also note that (assume C 0 ≥ 160 · 144 2 ) log ψ i = ni (cid:88)
j = log (cid:18) en j n(cid:19)
[Equation]
bound for II
On event E c
[Equation]
E c 3 , we have (cid:107) ˜ A d (cid:107) ≤ 3 and
[Equation]
Thus II ≤ r/ 4 . bound for III
Note that on E c 1 ,
[Equation]
Lemma 6.
Under assumptions A1-A4, we have
[Equation]
Proof.
Denote A d +1 =
Φ and s i
=
A (cid:124) i +1 · · · A (cid:124)
d
+1 (cid:15) for i ∈
[ d ] .
Also let s d +1 = (cid:15) and n d +1 = m .
For i ∈ [ d + 1] , by Lemma 3(a) we have
[Equation]
Choose t 1 = 20 n 0 and t j = n j − 1 / 4 j − 1 for j > 1 , we get
P A
[Equation]
s
[Equation]
(cid:107) ≤ (1 + √ 20) (cid:114) n 0 n 1 (cid:107) s 1 (cid:107) (cid:19) ≥
1 − e − 10 n 0 , P A i (cid:18) (cid:107) A (cid:124) i s i (cid:107) ≤ (1 + 2 − i +1 ) (cid:114) n i − 1 n i (cid:107) s i (cid:107) (cid:19) ≥ 1 − e − n i − 1 / 4 i , i > 1 .
Thus with probability at least 1 − e − 10 n 0 − (cid:80) d
+1 i =
e − n i − 1 / 4 i , (cid:107) A (cid:124)
{+1 A (cid:124) 2 · · · A (cid:124) d Φ (cid:124) (cid:15) (cid:107) ≤ (cid:16) 1 + √ 20 (cid:17) (cid:114) n 0 n 1 · d +1 (cid:89) i = (cid:18) 1 + 2 i −(cid:19) (cid:114) n i −n i ≤ (cid:16) 1 + √ 20 (cid:17) (cid:114) nm · ∞ (cid:89) i = (cid:18) 1 + 1 
+}
i (cid:19) < 15 (cid:112) n 0 /m where the last inequality uses estimate 3 (cid:81) ∞
i =1 (cid:0)
+ 12 i (cid:1) ≤ e and (1 + √ 20) e < 15 .
It remains to show (
cid:80) d +1 i =
e − n i − 1 / 4 i ≤ 2 e − 10 n 0 for the desired probability bound.
Note that by assumption A2 (assume C 0 ≥ 40 ), n i 4 i +1 ≥ 4 C 0 ni − 1 (cid:88)
j = log (cid:18) en j n(cid:19) ≥ 10 n 0 i.
Hence
[Equation]
With ReLU (or positively homogeneous) activation functions, the range of neural network (in each layer) is contained in a union of afﬁne subspaces.
The following lemma, which is based on ideas and results in [9], gives a precise statement of this. [Equation]
Lemma 7.
If min j ∈ [ d ] { n j } ≥ n 0 , then for i ∈ [ d ] , range( G i ) is contained in a union of afﬁne subspaces.
Precisely,
[Equation] where
[Equation]
Here each S i,j is some n 0 -dimensional afﬁne subspace (which depends on { A l } l ∈ [ i ] ) in R n i .
Proof.
The theory on hyperplane arrangements [28, Chapter 6.1] tells us that n hyperplanes in R k (assume n ≥ k ) partition the space R k into at most (cid:80) kj =0 (cid:0) nj (cid:1) regions 4 .
Also for k ∈ [ n ] ,
[Equation]
So consider range( G 1 )
= { σ ( A 1 x ) : x ∈ R n 0 } .
Denote by a 1 j ( j ∈ [ n 1 ] ) the rows of A 1 and let H be the set of hyperplanes H := ∪ j ∈ [ n 1 ] { x : (cid:10) a 1 j , x (cid:11) = 0 } .
Then H partitions R n 0 into at most ( en 1 /n 0 ) n 0 regions.
Note that σ is linear in each of these regions (thus the mapping G 1 is linear in each region), so range( G 1 ) is contained in at most ( en 1 /n 0 ) n [-
-]0 many n 0 -dimensional (afﬁne) subspace.
The result then follows by induction.
The following lemma shows that the network G in our model is Lipschitz with high probability.
This may be an interesting result on its own.
Lemma 8.
For mappings G i , ˜
A i : R n 0 → R n i , let L G i and
L ˜
A i be their Lipschitz constants respectively.
Under assumptions A1 and A2, we have
[Equation]
Proof.
Denote ˜ R
= R
= R n 0 and
R j = range( G j ) − range( G j ) ,
[Equation]
∪ range
[Equation]
˜
A j ) , j ∈ [ d ] .
Note that ˜
A j is linear, so range(
˜ A j ) is a subspace in R n i with dimension at most n 0 .
Since σ is 1-Lipschitz, we have
[Equation]
Hence (cid:107)G i ( x ) − G i ( x (cid:48) )
(cid:107) ≤ (cid:32) i
(cid:89) l
= (cid:107)
A l (cid:107) ˜ R l −(cid:33) (cid:107) x − x (cid:48) (cid:107) , ∀ i ∈
[ d ] .
Similarly, (cid:107) ˜ A
i x − ˜
A i x (cid:48) (cid:107) ≤ (cid:32) i (cid:89) l = (cid:107) A l (cid:107) ˜ R l −(cid:33) (cid:107) x − x (cid:48) (cid:107) , ∀ i ∈
[ d ] .
By Lemma 7, range( G i ) is contained in a union of N G i many n 0 -dimensional afﬁne subspaces, so
R i is contained in a union of at most N 2 G i many 2 n 0 -dimensional afﬁne subspaces.
Since every
2 n 0 -dimensional afﬁne subspaces in R n i is also contained in a (2 n 0 + 1) -dimensional subspace, we can further write this as
[Equation]
where
[Equation]
and each S i,j is a (2 n 0 + 1) -dimensional subspace in R n i .
Thus by Lemma 3(b) and union bound we have, for i ∈
[ d − 1] ,
[Equation]
Choose t i = 26 log ψ i = 26 n 0 (cid:80) ij =1 log( en j n 0 ) > 2 n 0 +
we get
P
[Equation]
Under assumption A2 (with C 0 ≥ 2 2 · 26 ), this implies
P
[Equation]
+
[Equation]
A
i +1 (cid:107) ˜
R i ≥
[Equation] 2 i + (cid:19) ≤ e − 10log ψ
i , i ∈
[ d − 1] .
Also by Lemma 3(b) with t = 20 n 0 and assumption A2 (assume C 0 ≥ 2 2 · 26 ), we have
[Equation]
Therefore with probability at least 1 − e − 10 n 0 − (cid:80) d − 1 i =1 e − 10log ψ i , ∀ i ∈ [ d ] , i (cid:89) [Equation] = [Equation]
(cid:107)
[Equation]
≤ [Equation] (cid:89) [Equation]
=
[Equation]
(cid:18)
[Equation]
(cid:19) ≤
[Equation]
(cid:89) l = (cid:18) 1 + [Equation] (cid:19) < 3 . Finally, note that log ψ i ≥ in 0 , so we have (cid:80) d − 1 i =1 e − 10log ψ i ≤ (cid:80) ∞
i =1 e − 10 n 0 i < 2 e − 10 n 0 .
This completes the proof.
Lemma 9.
Let R =
range( G ) − range( G ) .
If min j ∈ [ d ] { n j } ≥ n 0 and assumption A3 holds, then
[Equation]
Proof.
Let ψ = (cid:81)
dj
=1 (cid:16) en j n 0 (cid:17) n 0 .
Similar to the proof in Lemma 8, we know that R is contained in a union of at most ψ 2 many (2 n 0 + 1) -dimensional subspaces in R n d .
From Lemma 3(b) and a union bound we get
[Equation]
By choosing t = 24 n 0 (cid:80) dj
=1 log( en j n 0 ) and noticing that (cid:107) I − Φ (cid:124)
Φ (cid:107) R = |(cid:107) Φ (cid:107) R − 1 | 2 , we have P
[Equation]
Φ [Equation] [Table] This completes the proof with c 0 ≥ 96 · 36 in assumption A3.
E An Example of n i
Here we show if n i = βC 0 5 d n 0 d (2 d − i ) where β is any ﬁxed number such that βC 0 ∈ N and β ≥ 4 + log C 0 , then n i satisfy (5).
In fact, note that 2 log d < d and log(2 β ) <
β , we have log
[Equation]
Since n i ≥ C 0 5 d n 0 ( βd 2 ) , it is easy to see that n i satisfy (5).
Remark: A similar argument as above can also show that n i = βC 0 5 i n
i 2 satisfy (5) .
F Code Link
Codes for numerical experiments are available at https://github.com/babhrujoshi/PLUGIn.