Beyond Independent Measurements: General Compressed Sensing with GNN Application
Abstract
We consider the problem of recovering a structured signal x ∈ R n from noisy linear observations y
=
Mx + w .
The measurement matrix is modeled as M = BA , where B ∈ R l × m is arbitrary and A ∈ R m × n has independent sub-gaussian rows.
By varying B , and the sub-gaussian distribution of A , this gives a family of measurement matrices which may have heavy tails, dependent rows and columns, and singular values with a large dynamic range.
When the structure is given as a possibly non-convex cone T ⊂ R n , an approximate empirical risk minimizer is proven to be a robust estimator if the effective number of measurements is sufﬁcient, even in the presence of a model mismatch.
In classical compressed sensing with independent (sub-)gaussian measurements, one asks how many measurements are needed to recover x ?
In our setting, however, the effective number of measurements depends on the properties of B .
We show that the effective rank of B may be used as a surrogate for the number of measurements, and if this exceeds the squared
Gaussian mean width of ( T − T )
∩ S n − 1 , then accurate recovery is guaranteed.
Furthermore, we examine the special case of generative priors in detail, that is when x lies close to T = ran( G ) and G : R k
→ R n is a Generative Neural
Network (GNN) with ReLU activation functions.
Our work relies on a recent result in random matrix theory by Jeong, Li, Plan, and Yılmaz [10].
Introduction
In compressed sensing [7], the goal is to reconstruct a high-dimensional signal x ∈ R n from a noisy low-dimensional linear transformation of it, y =
Mx + w ∈ R l , l < n .
Even in the absence of noise, the reconstruction would not be possible without a further assumption of signal structure , i.e., some restriction of the possible values of x .
In early works in compressed sensing, the signal structure was encoded via sparsity, or sparsity with respect to a dictionary.
A sufﬁcient condition on the measurement matrix M that enables robust recovery is the celebrated Restricted Isometry
Property (RIP) [4].
Intriguingly, all known algorithms for certifying RIP either are computationally intractable or only function in the parameter regimes which are highly suboptimal [6], but a subgaussian measurement matrix with independent rows satisﬁes the RIP with high probability in near optimal parameter regimes [2].
More recently, compressed sensing ideas have been generalized to allow the signal structure to be almost arbitrary, encoded as a subset T ⊂ R n , provided that the measurements are appropriately random.
Gaussian measurement matrices are rotationally invariant [23], which allows them to be universally effective for general compressed sensing.
This was proven using [5, 18] or generalizing [19, 15, 20] Gordon’s theorem [8], and also through a foundational result in conic geometry [1].
Sub-gaussian matrices, as deﬁned later, are essentially the largest class of matrices that approximately satisfy rotation invariance, and also enjoy universal guarantees, although with different proof techniques relying on chaining arguments [14, 11, 10].
We also note a few results outside of the Gaussian or sub-gaussian framework: [21, 16, 13, 12].
The recent results of [10] extend previous theory by allowing the measurement matrix to take the form M = BA with arbitrary B and sub-gaussian A with independent rows.
They prove a general restricted isometry property, thereby giving basic signal recovery guarantees for measurement matrices with dependent rows.
This random model is compelling because real-data measurement matrices often have highly variable singular values, but sub-gaussian matrices with independent rows typically have nearly uniform singular values.
In fact, under mild assumptions, the singular values of BA concentrate around the singular values of B (after rescaling), and so, one may target any singular value vector by adjusting the singular values of B .
We build upon the work of [10] to show accurate signal recovery by the generalized Lasso, even in the presence of inexact optimization and/or a mismatch in the signal structure.
We believe this provides the ﬁrst general compressed sensing guarantees allowing any two of the following three items simultaneously (we allow all three): (i) the measurement matrix need not have independent rows, (ii) the signal may be estimated by generalized
Lasso, but without exact optimization, and (iii) the signal is allowed to be only approximately structured, i.e. close to the structure set and not belonging to it.
The improved dependence on the sub-gaussian parameter of the measurement matrix in [10] is reﬂected in our work as well.
Items (ii) and (iii) above make our framework well-suited to the setting in which the signal structure is the range of a neural network.
Indeed, in recent years, it has been shown empirically that learning the appropriate signal structure by ﬁtting it to the range of a neural network can be much more effective than using a predetermined structure such as sparsity [3, 22, 9].
In this case, one assumes that T is the range of a generative model G : R k → R n which is already trained.
While training often approximates the signal structure well, any new signal outside of the training set will typically be close, but not in T ; thereby necessitating item (iii).
The standard method of estimating x is to ﬁnd the restricted least squares ﬁt, i.e., to let the estimate
ˆx be the solution to the program, sometimes called generalized Lasso:
[Equation]
Since T = ran( G ) is non-convex, and no convex relaxation of such a program is known in general, typical gradient-descent-based algorithms would not be guaranteed to approach a global minimum; thereby necessitating item (ii).
Our theory requires an upper bound of the Gaussian mean width of ( T − T ) ∩ S n − 1 , which we give in Proposition 3.2 below; we believe this is novel.
Main Results 2.1 Problem Setup
Recall that a random variable Z is called sub-gaussian , if its tail probability is dominated by that of a
Gaussian random variable.
An equivalent rigorous deﬁnition requires the sub-gaussian norm to be ﬁnite:
|| Z || ψ 2 := inf (cid:8) t > 0 (cid:12)(cid:12) E
( Z 2 /t 2 ) ≤ 2 (cid:9) < ∞ .
A random vector v ∈ R n is sub-gaussian , if all of its one-dimensional marginals are sub-gaussian random variables.
Mathematically, if ||(cid:104) θ, v (cid:105)|| ψis ﬁnite for all θ ∈ S n − 1 , we deﬁne || v || ψ 2 :=
sup θ ∈ S n − 1 ||(cid:104) θ, v (cid:105)|| ψ 2 .
We will consider a random matrix A ∈ R m × n whose rows { a (cid:62) 1 , · · · , a (cid:62) m } are statistically independent, mean-zero ( E a i = 0 ), isotropic ( E a i a (cid:62) i = I n ), and sub-gaussian with parameter K ( || a i || ψ 2 ≤ K ).
We let the measurement matrix be the product BA , where B ∈ R l × m is arbitrary.
In other words, we let every row of our measurement matrix be an arbitrary linear combination of a (cid:62) 1 , · · · , a (cid:62) m .
Our goal is to determine the criteria that B , A , and the structure set T must satisfy for accurate recovery to be possible.
Informally, one requires B to be far from low-rank, otherwise, the number of independent effective measurements would not be sufﬁcient.
An appropriate quantity is the stable rank deﬁned as
[Equation]
where σ i ’s are the singular values of B .
The main advantage of this deﬁnition over rank itself is that it is robust to the small non-zero singular values that increase the rank but do not contribute to an effective measurement, hence the name stable rank.
Furthermore, one requires T to be small in some sense, ideally not to intersect the null space of M = BA .
A useful notion of size is the Gaussian mean width deﬁned as
[Equation]
where the expectation is calculated with respect to g ∼ N ( 0 , I n ) .
The reader may consult [17] for some basic properties of Gaussian mean width.
The following theorem gives the desired estimation bound.
2.2 Main Theorem
Theorem 2.1.
Let x ∈ R n , B ∈
R
l
× m be an arbitrary ﬁxed matrix, and A ∈ R m × n be a matrix whose rows are independent, mean-zero, isotropic, and sub-gaussian vectors with sub-gaussian parameter K .
Let T ⊂ R n be a closed cone and deﬁne T (cid:48) :
= ( T − T )
∩ S n − 1 .
Let y = BAx + w for some ﬁxed unknown w ∈ R m .
Let ˆx ∈ T satisfy || y − BAˆx
|| 22 ≤ min x (cid:48) ∈ T
|| y − BAx (cid:48)
||
22 + (cid:15) 2 .
If sr( B ) (cid:29) K 2 log K · w 2 ( T (cid:48) ) , then with probability larger than 1 − 11 e − w 2 ( T (cid:48) ) ,
[Equation]
There are three sources of error present here: the additive noise w , the inaccuracy in optimization (cid:15) , and the model mismatch, i.e., x not exactly belonging to T .
Note that sr( B ) ≤ rank( B ) ≤ min( l, m ) , so basically the effective number of measurements is bounded by the number of underlying independent measurements m .
When B is a multiple of identity, the equality case sr( B ) = m occurs.
What happens if we have more effective measurements than needed?
The energy of the noise that appears in the total error in (3) is decreased by the oversampling factor sr( B ) /w 2 ( T (cid:48) ) .
In other words, if we have some control over the measurement matrix, then we are able to denoise the original signal by increasing sr( B ) .
In case of B = I m , this simply means increasing the number of measurements m .
This denoising effect is well known for stochastic noise, but we believe that it had previously been shown only for non-random noise under the assumption of a Gaussian measurement matrix, as in [20].
We note, it is important here that the noise is ﬁxed i.e., not chosen adversarially depending on the realization of A .
Application on Generative Neural Networks
Due to the success of deep generative neural networks to learn complex structures, recent works on compressed sensing have considered generative priors instead of the traditional sparsity [3].
While some works require the trained network G : R k → R n to be L -Lipschitz [3] and prove that O ( k log L ) number of measurements would sufﬁce for a recovery guarantee, a drawback of such analysis is that the Lipschitz constant of the network cannot be calculated solely based on its architecture.
Here we present a different approach: we give an upper bound of the Gaussian mean width of T (cid:48)
= (cid:0) ran( G ) − ran( G ) (cid:1)
∩ S n − 1 , which only depends on the hyperparameters of the model.
Then we apply Theorem 2.1 to achieve a recovery guarantee for compressed sensing with generative priors.
3.1 GNN and Guassian Mean Width
A d -layer GNN with ReLU activation function is a function G : R k
→ R n of the form
[Equation]
[Equation]
where σ ( · ) = max( · , 0) is applied entrywise and A i ∈ R p i × p i − 1 , p
= k , and p d = n .
The weight matrices A i are sometimes assumed to have iid Gaussian entries, however, our analysis puts no requirement on A i .
To bound the Gaussian mean width of ran( G ) , we ﬁrst show that the set is contained in a union of subspaces with minimal count.
Note that the choice of ReLU activation function is not essential to what follows, and any piecewise linear function with only two pieces (e.g. leaky ReLU) could be substituted.
Similar counting arguments have appeared in various theoretical works on expansive neural nets.
Lemma 3.1.
Let G be as in (4) and T = ran( G ) .
We have T ⊂ (cid:83) Ni =
1 E i , where each E i is a subspace of dimension at most k and
[Equation]
Consequently,
[Equation]
where each F i is a subspace of dimension at most 2 k .
Proposition 3.2.
Let G be as in (4) and T = ran( G )
⊂
R n .
Then the following holds:
[Equation]
where p (cid:48) = (cid:16) (cid:81) dj
=1 p j (cid:17) 1 /d is the geometric mean of p 1 , · · · , p d .
3.2 Compressed Sensing with Generative Priors
The effective number of measurements required for any recovery algorithm to be successful is
O (cid:0) w 2 ( T (cid:48) ) (cid:1) =
O (cid:0) kd log( p (cid:48) /k ) (cid:1) .
Typically, a GNN is assumed to be expansive, that is k = p [-
 ≤p-]0 ≤ p 1 ≤ · · · ≤ p d = n .
Thus, k ≤ p (cid:48) ≤ n , which slightly improves O ( kd log n ) in [3].
Moreover, by combining Theorem 2.1 and Proposition 3.2, we present a recovery guarantee for the compressed sensing with generative priors that not only allows for a much broader range of measurement matrices, but also demonstrates the dependence on their parameters.
Corollary 3.2.1.
Consider the settings of Theorem 2.1 for T = ran( G ) as in (4) .
If sr( B ) (cid:29)
K 2 log K · kd log( p (cid:48) /k ) , then
[Equation]
Our analysis improves upon the best known results in three ways: Firstly, our result suggests a denoising behaviour as the number of measurements (or the stable rank of B ) increases, even though the noise is not assumed to be random.
Intuitively, one expects a smaller error bound associated with less compression, and our theory highlights this dependence.
In contrast, the bound in [3] shows constant dependence on the noise level, the optimization margin, and the model mismatch, regardless of the number of measurements.
Secondly, we improve the logarithmic factor in the compression bound, i.e. we need O ( kd log( p (cid:48) /k ))
≤ O ( kd log( n/k )) effective measurements, rather than O ( kd log n ) .
This was also available to the authors of [3], had they used a tighter inequality in proof of Lemma 8.3.
Finally, we require milder conditions for the measurement matrix, allowing for a ﬁxed mixing matrix to create dependence among the rows.
This improvement is based on the geometry-preserving properties discussed in [10].
Summary
We achieve an estimation bound for the reconstruction of structured signals using noisy and dependent random measurements.
The generality of our model enables its application on the non-traditional structure sets such as the range of a ReLU generative neural network.
We believe that this is the ﬁrst general compressed sensing result that specializes well to the generative structure.
Whether a similar recovery guarantee is obtainable for GNNs with other activation functions (e.g. sigmoid or tanh), remains an open question.
A Appendix
Proof of Theorem 2.1.
Since T is closed, there exists a (possibly not unique) point x 0 = argmin x (cid:48) ∈ T || x − x (cid:48)
|| 2 .
Let r := x − x 0 and h :
= ˆx − x 0 ∈ T − T .
We have
[Equation]
Expanding the LHS and rearranging yields
[Equation]
The LHS is concentrated about || B || 2 F · ||
h
|| 22 .
In fact, by Theorem 1.1 of [10] we can write
[Equation]
with probability at least 1 − 3 e − α 2 .
Choosing α = w ( T (cid:48) ) and using the fact that sr( B ) (cid:29) K 2 log K · w 2 ( T (cid:48) ) , we get
[Equation]
14) with probability at least 1 − 3 e − w 2 ( T (cid:48) ) .
To bound the ﬁrst term on the RHS, deﬁne the random process X t := (cid:104) BAt , w (cid:105) , for t ∈ T (cid:48) .
Recall that || · || ψ 2 = sup ||
u
|| 2 =1 ||(cid:104) u , ·(cid:105)|| ψ 2 .
So, || X t − X s || ψ 2 = ||(cid:104) t − s , A (cid:62) B (cid:62) w (cid:105)|| ψ 2 ≤ || t − s
|| 2 ||
A (cid:62) B (cid:62) w || [Equation] [Equation] |
[Equation] w || [-
-][Equation] t
[Equation]
|| [Equation] ≤ K || B || · || w || 2 · || t − s || 2 .
Therefore, by
Talagrand’s comparison inequality (Exercise 8.6.5 of [23]),
[Equation]
(15) with probability at least 1 − 2 e − β 2 .
Again, by setting β = w ( T (cid:48) )
we get
[Equation]
(16) with probability at least 1 − 2 e − w 2 ( T (cid:48) ) .
Now let us bound the second term on the RHS.
By Cauchy-Schwarz inequality, (cid:104) BAh , BAr (cid:105) ≤ || BAh || 2 || BAr || 2 .
Theorem 1.1 of [10] implies that || BAh || 2 (cid:46)
||
B || F || h
|| 2 with probability at least 1 − 3 e − w 2 ( T (cid:48) ) , considering that sr( B ) (cid:29) K 2 log K · w 2 ( T (cid:48) ) .
The same theorem can be used on singleton { r } to bound || BAr
|| 2 .
Since w ( { r } ) = 0 and rad( { r } )
=
||
r || , we get
[Equation]
[Equation]
18) with probability at least 1 − 3 e − u 2 .
Choosing u
= w ( T (cid:48) )
and using sr( B ) (cid:29) K 2 log K · w 2 ( T (cid:48) ) yields || BAr
|| 2 (cid:46) ||
B ||
F || r
|| 2 , with probability at least 1 − 3 e − w 2 ( T (cid:48) ) .
Thus, [Equation] BAr
[Equation]
(19) with probability at least 1 − 3 e − w 2 ( T (cid:48) ) − 3 e − w 2 ( T (cid:48) ) ≥ 1 − 6 e − w 2 ( T (cid:48) ) .
Combining equations (12)-(19) gives us
[Equation]
which implies
[Equation]
with probability at least 1 − 3 e − w
[Equation]
T
[Equation]
−
[Equation]
Finally
[Equation]
with the aforesaid probability
[Equation]
Lemma A.1.
A k -dimensional subspace in R n intersects at most 2 k (cid:0) nk (cid:1) different orthants.
Lemma A.2.
Let D ⊂ R n be a k -dimensional subspace, and Q be an orthant with q number of positive (and n − q negative) coordinates.
Then σ ( D ∩ Q ) is contained in a subspace of dimension at most min( k, q ) .
Proof of Lemma 3.1.
By the rank-nullity theorem, the subspace im( A 1 ) ⊂ R p 1 is at most kdimensional.
By Lemma A.1, this particular subspace hits at most 2 k (cid:0) p 1 k (cid:1) different orthants.
Therefore, by Lemma A.2, σ (im( A 1 )) is contained in a union of at most 2 k (cid:0) p 1 k (cid:1) subspaces of dimension at most k .
Each of these subspaces, when multiplied by A 2 , is mapped to another subspace of dimension at most k in R p 2 .
Thus, the linear transformations do not increase the number nor the dimension of the subspaces.
Hence, at each layer i , every subspace breaks into 2 k (cid:0) p i k (cid:1) subspaces, at most.
All in all, after d layers, we end up having a union of N subspaces of dimension at most k , where
[Equation]
Thus,
T = ran( G ) ⊂
N (cid:91) i =
E i , (28) and
T − T ⊂
N (cid:91) i =
N (cid:91)
j = ( E i − E j ) =
(cid:91) 1 ≤ i<j ≤ N ( E i + E j ) , (29) where dim( E i ) ≤ k and dim( E i + E j ) ≤ 2 k .
Lemma A.3.
Let g ∼ N (0 , I n ) and f : R n → R be a Lipschitz function.
Then
[Equation]
Lemma A.4.
Let X 1 , X 2 , · · · , X N be sub-gaussian random variables with K = max
i
||
X
i || ψ 2 .
Then
[Equation]
Lemma A.5.
Let T 1 , · · · , T N ⊂ S n − 1 ⊂ R n .
Then
[Equation] [Equation]
Proof of Lemma A.5.
Let g be a standard Gaussian vector and deﬁne z
i := sup x ∈ T
i
(cid:104) x , g (cid:105) − w ( T i ) .
Equivalently, let f i ( · ) :
= sup x ∈ T i (cid:104) x , ·(cid:105) and check that z
i = f i ( g )
− E f i ( g ) .
Lemma [-A.
 implies||-]A.3 implies || z i || ψ 2 =
||
f i ( g ) − E f i ( g ) || ψ 2 (cid:46) ||
f i || Lip = 1 .
Then, by Lemma A.4, we have E max
i z
i
(cid:46) √
[Equation]
i |
[Equation]
z i || ψ 2 (cid:46) [Equation] log N .
Hence,
[Equation]
Proof of Proposition 3.2.
By Lemma 3.1, T − T ⊂ (cid:83) ( N 2 ) i =1 F i , where dim( F i ) ≤ 2 k .
Thus, w (cid:0) ( T − T ) ∩ S n − 1 (cid:1) ≤ w (cid:0) ( N 2 ) (cid:91)
[Equation]
Using Lemma A.5 and Lemma 3.1 respectively [Equation] then we have
[Equation]