Multi-layer State Evolution Under Random Convolutional Design
Anonymous Author(s)
AfﬁliationAddress email
Abstract
Signal recovery under generative neural network priors has emerged as a promisingdirection in statistical inference and computational imaging.
Theoretical analysisof reconstruction algorithms under generative priors is, however, challenging.
For generative priors with fully connected layers and Gaussian i.i d. weights,this was achieved by the multi-layer approximate message (ML-AMP) algorithmvia a rigorous state evolution.
However, practical generative priors are typicallyconvolutional, allowing for computational beneﬁts and inductive biases, and
so the
Gaussian i.i.d.
weight assumption is very limiting.
In this paper, we overcome thislimitation and establish the state evolution of ML-AMP for random convolutionallayers.
We prove in particular that random convolutional layers belong to the sameuniversality class as Gaussian matrices.
Our proof technique is of an independentinterest as it establishes a mapping between convolutional matrices and spatiallycoupled sensing matrices used in coding theory.
Introduction
In a typical signal recovery problem, one seeks to recover a data signal x 0 given access to measurements y
= G ✓ ( x 0 ) , where the parameters ✓ of the signal model are known.
In many problems, it isnatural to view the measurement generation process as a composition of simple forward operators, or‘layers.’
In this work, we are concerned with multi-layer signal models of the form
[Equation]
(1) where W ( l ) 2 R n l  1 ⇥ n l are linear sensing matrices and where ' ( l ) ( z ) are separable, possibly nonlinear channel functions.
In the L = 1 case, this signal model naturally generalizes problems suchas phase retrieval ' ( z ) =
| z | or compressive sensing [-y 
 = AG θ-]' ( z ) =
z , and for multi-layer models L > 1 , G ✓ ( h ) may be viewed as a deep neural network.
Recently, convolutional Generative Neural Networks (GNNs) have shown promise as generalizationsof sparsity priors for a variety of signal processing applications Bora et al.
Motivated bythis success, we take interest in a variant of the recovery problem (1) in which some of the sensingmatrices W ( l ) may be multi-channel convolutional (MCC) matrices, having a certain block-sparsecirculant structure which captures the convolutional layers used by many modern generative neuralnetwork architectures Karras et al.
In this work, we develop an asymptotic analysis of the performance of an Approximate Message
Passing (AMP)
algorithm Donoho
et al.
[2009] for recovery from multichannel convolutional signalmodels.
This family of algorithms originates in statistical physics Mézard and Montanari [2009],
Zdeborová and Krzakala [2016] and allows to compute the marginals of an elaborate posteriordistributions deﬁned by inference problems involving dense random matrices.
A number of AMP
[Figure]
iterations have been proposed for various inference problems, such as compressed sensing Donohoet al.
[2009], low-rank matrix recovery Rangan and Fletcher [2012] or generalized linear modeling
Rangan [2011].
More recently, composite AMP iterations (ML-AMP) have been proposed to studymultilayer inference problems Manoel et al.
[2017], Aubin et al.
Here we consider the
ML-AMP proposed in Manoel et al.
[2017] to compute marginals of a multilayer generalized linearmodel, however the usual dense Gaussian matrices will be replaced by random convolutional ones.
A major beneﬁt of AMP lies in the fact that the asymptotic distribution of their iterates can beexactly determined by a low-dimensional recursion: the state evolution equations.
This enablesto obtain precise theoretical results for the reconstruction performance of the proposed algorithm.
Another beneﬁt of such iterations is their low computational complexity, as they only involve matrixmultiplication and, in the separable case, pointwise non-linearities.
Previous works on AMP suggest that the state evolution is not readily applicable to our settingbecause its derivation requires strong independence assumptions on the coordinates of the { W ( l ) }which are violated by structured multi-channel convolution matrices.
Despite this, we use AMP forour setting and rigorously prove its state evolution.
Our main contributions are:
We rigorously prove state evolution equations for models of the form (1), where weights areallowed to be either i.i.d.
Gaussian or random structured MCC matrices, as in Deﬁnition 3.2.2.
For separable channel functions ' ( l ) and separable signal priors, we show that the original ML-
AMP of Manoel et al.
[2017] used with dense Gaussian matrices or random convolutional ones admitsthe same state evolution equations, up to a rescaling.
Multi-layer MCC signal models can therefore simulate dense signal models while making use of fast structured matrix operations for convolutions.
The core of our proof shows how an AMP iteration involving random convolutional matrices may
be reduced to another one with dense Gaussian matrices.
We ﬁrst show that random convolutional matrices are equivalent, through permutation matrices, to dense Gaussian ones with a (sparse) block- circulant structure.
We then show how the block-circulant structure can be embedded in a new,matrix-valued, multilayer AMP with dense Gaussian matrices, the state evolution equations of whichare proven using the results of Gerbelot and Berthier [2021], with techniques involving spatiallycoupled matrices Krzakala et al.
[2012], Javanmard and Montanari [2013].4.
We validate our theory numerically and observe close agreement between convolutional AMPiterations and its state evolution predictions, as shown in Figure 1 and in Section 5.
Our code canbe used as a general purpose library to build compositional models and evaluate AMP and its stateevolution.
We make this code available at <see supplemental> .
Related Work
AMP-type algorithms arose independently in the contexts of signal recovery and spin-glass theory.
Inthe former case, Donoho et al.
[2009] derives AMP for Gaussian compressive sensing.
This approachwas later generalized by Rangan [2011] to recovery problems with componentwise channel functionsthat may be stochastic and/or nonlinear, and generalized further by Manoel et al.
[2017] to multi-layeror compositional models.
Due to the versatility of this approach, a wide variety of general purposeframeworks for designing AMP variants have since been popularized Fletcher et al.
[2018], Bakeret al.
[2020], Gerbelot and Berthier [2021].
Proof techniques to show the concentration of AMPiterates to the state evolution prediction developed alongside new variants of the algorithm.
In thecontext of spin-glass theory, Bolthausen’s seminal work Bolthausen [2009] introduces a Gaussianconditioning technique used widely to prove AMP concentration.
Following this approach, Bayatiand Montanari [2011], Javanmard and Montanari [2013], Berthier et al.
[2020] treat signal modelswith dense couplings and generalized channel functions.
More recently, a proof framework adaptableto composite inference problems was proposed in Gerbelot and Berthier [2021], which we use in ourproof.
There has also been signiﬁcant interest in relaxing the strong independence assumptions required by
AMP algorithms on sensing matrix coordinates.
In one direction, Vector AMP (VAMP) algorithmstarget signal models whose sensing matrices are drawn from right orthogonally invariant distributions.
The development of VAMP algorithms followed a similar trajectory to that of vanilla AMP Schniteret al.
[2016], Fletcher et al.
[2018], Rangan et al.
[2019], Baker et al.
The MCC ensembleconsidered in this work is not right orthogonally invariant, but we observe strong empirical evidencethat an analogous version of Theorem 4.2 holds for VAMP as well, as described in Appendix C.2.
In asecond direction, there has been much interest in spatial coupling sensing matrices, which were usedto achieve the information-theoretically optimal performance in sparse compressive sensing Donohoet al.
[2013], Barbier et al.
[2015], Krzakala et al.
[2012], with complementary state evolutionguarantees Javanmard and Montanari [2013].
The concept of spatial coupling and proofs of itsperformance originated in the literature of error correcting codes Kudekar et al.
[2011, 2013], whereit developed from the so-called convolutional codes Felstrom and Zigangirov [1999].
The connectionbetween spatial coupling and convolution layers of neural networks, that we establish in this paper, isas far as we know novel.
Another direction of related work is the design of generative neural network architectures, andcorrespondingly, the design of signal recovery procedures that can make use of new generative priormodels.
There is a wide variety of architectures for feedforward convolutional generative priors,which are often hand-crafted to be stably trained on real world datasets. For instance, the DC-GANarchitecture, studied by Bora et al.
[2017] in compressive sensing and superresolution tasks, achievesstable training through the use of multi-channel strided convolutional layers and batch normalization this, the PG-GAN architecture Karras
] uses multichannel convolutional layers, upsampling layers, and a parameter free alternative to batch normalization. Recently, Style-GAN has emerged as a popular architecture for generating large, high-resolutionimages Karras et al. The StyleGAN generator uses residual and skip connections to encouragea hierarchical image generation process, contributing to stable training even when generating highresolution images. Style-GAN and PG-GAN, and further domain-speciﬁc modiﬁcations, have beenstudied as priors for a variety of signal recovery problems Daras et al.
[2021], Gu et al.
Forsimplicity and theoretical tractibility, we do not consider ﬁne-grained practical modiﬁcations likebatch normalization or strided convolution, focusing instead on the essential elements of simple convolutional networks.
Lastly, while our focus is on feedforward convolutional priors such as GAN/VAE networks, there is growing interest in alternative approaches to signal recovery underneural network priors, such as normalizing ﬂows Kingma and Dhariwal [2018], Asim et al.
[2020]and score-based generative models Song and Ermon [2020], Jalal et al.
These approaches falloutside the scope of our work and may be interesting directions for future investigation.
Deﬁnition of the problem
3.1 Multi-channel Convolutional Matrices
We focus our attention on multichannel convolution matrices that have localized convolutional ﬁlters .
In this section, we introduce our notation and deﬁne the random matrix ensembles which are relevant [Figure] to our result.
We consider block structured signal vectors x 2 R Pq of the form x =
[ x ( i ) ]
Pi =1 , and werefer to the blocks x ( i ) 2 R q as ‘channels.’
For any vector of dimension d , we denote by P d 2 R d ⇥ dthe cyclic coordinate permutation matrix of order d , whose coordinates are h e i , P d e j i =
[ i = j + 1] .
For a block-structured vector x 2 R Pq , we denote by P P,q 2 R Pq ⇥
Pq the block cyclic permutationmatrix satisfying ( P P,q x ) ( i ) =
x ( i +1) for 1  i < P , and ( P P,q x ) ( P ) = x (1) .
Similarly, we denoteby S i,j 2 R Pq ⇥
Pq the swap permutation matrix which exchanges blocks i, j : [ S i,j x ] ( i ) =
x ( j ) ,[ S i,j x ] ( j ) =
x ( i ) , and [ S i,j x ] ( k ) =
x ( k ) for k 6 = i, j .
Last, given a vector ! 2 R k for k  q , denoteby Zero-Pad q,k ( ! ) the vector whose ﬁrst k coordinates are ! , and whose other coordinates arezero.
[Equation]
We deﬁne the following ensemble for random multi-channel convolution matrices.
Deﬁnition 3.1 (Gaussian i.i.d.
Convolution) .
Let q  k be integers.
The convolutional ensemble C ( q, k ) contains random circulant matrices C 2 R q ⇥ q whose ﬁrst row is given by
C
= Zero-pad q,k [ ! ] where ! 2 R k has i.i.d.
Gaussian coordinates !
i ⇠ N (0 , 1 /k ) .
Theremaining rows C i are determined by circulant structure, ie.
C i
=
P
i  1 q
Zero-pad q,k [ ! ] .
Random multi-channel convolutions are block-dense matrices with independent C ( q, k ) blocks.
Deﬁnition 3.2 (Multi-channel Gaussian i.i.d.
Convolution) .
Let D, P  1 and q  k  1 be integers.
The random multi-channel convolution ensemble M ( D, P, k, q ) contains random block matrices
M 2 R Dq ⇥ Pq of the form M = 1 p P C 1 ,
C 1 ,. . . C 1 ,P C 2 ,... C D,. . .
C D,P  where each C i,j ⇠ C ( q, k ) is sampled independently.
Fig.
2 gives a graphical explanation of the link between these matrices and the convolutional layers.
The parameter P ( D ) is the number of input (output) channels, q is the dimension of the input and kthe ﬁlter size.
3.2 Thermodynamic-like Limit and Finite-size Regimes
We prove our main result in a thermodynamic-like limit D, P ! 1 while  = D/P is ﬁxed andq, k
= O (1) .
From a practical perspective, convolutional layers in deep neural networks often uselarge channel dimensions ( D, P  1 ), large signal dimensions ( q  1 ), and a small ﬁlter size( k = O (1) ).
As an example, we show in Figure 3 the sizes of convolutional layers used by the
DC-GAN architecture to generate LSUN images [Radford et al., 2015, Figure 1].
Interestingly, our theoretical predictions do not depend explicitly on the relative sizes of q and ( D, P ) .
We observe empirically that these predictions become accurate at ﬁnite sizes of ( D, P ) which may
Layer
D
P q k 1 !  4 2 !  8 3 !
64 seem small relative to q , and which are realistic from a practical neural network perspective.
Forexample, in Figure 1, we observe strong empirical agreement with predictions for q = P = 1024 as and D =  P vary.
3.3 Multi-layer AMP
In this section, we deﬁne a class of probabilistic graphical models (PGMs) that captures the inferenceproblems of interest, and we state the Multi-layer Approximate Message Passing (ML-AMP)
Manoelet al.
[2017] iterations, which can be used for inference on these PGMs.
We consider the followingsignal model.
Deﬁnition 3.3 (Multi-layer Signal Model) .
Let { W ( l ) } {+
 +} l  L be matrices of dimension W ( l ) {+
+}
R n l  1 ⇥ n l .
Let { ' ( l ) ⇣ ( z ) } {+
 +} l  L be scalar channel functions ' ( l ) ⇣ : R ! R for which z is theestimation quantity and ⇣ represents channel noise.
We write ' ( l ) ⇣ ( z ) for vectors z 2 R n l  1 toindicate the coordinatewise application of ' ( l ) .
The multi-layer GLM signal model is given by
[Equation]
[Equation]
[Equation]
L
[Equation]
( L ) x ))) . We assume x 2 R n L follows a known separable prior, x i ⇠ P X ( x ) i.i.d., and that ⇣ ⇠ N (0 , 1) .
The full estimation quantities of the model are the coordinates of the vectors { h ( l ) } 1  l  L ,{ z ( l ) } {+
 +} l  L , which are related by
[Equation]
and the corresponding conditional probabilities, which deﬁne the factor nodes of the underlying PGM, are given by
[Equation]
To compute the posterior marginals, ML-AMP iteratively updates the parameters of independent 1D
Gaussian approximations to each marginal.
Each coordinate h ( l ) i ( t ) has corresponding parameters{ A ( l ) i ( t ) , B ( l ) i ( t ) } and each z ( l ) µ ( t ) has corresponding { V ( l ) µ ( t ) , ! ( l ) µ ( t ) } , where t  1 indexes the
ML-AMP iterations.
The recursive relationship between these parameters is deﬁned in terms of scalardenoising functions , ˆ h ( l ) and g ( l ) , which compute posterior averages of the estimation quantitiesgiven their prior parameters.
In general, these denoising functions can be chosen (up to regularity assumptions) to adjust MLAMP’s performance in applied settings, such as in Metzler et al.
[2015], and in these cases the denoisers may be nonseparable vector valued functions.
However, in the separable, Bayes-optimalregime where P x ( x ) and P ( l ) ( h | z ) are known, the optimal denoisers are given by,
[Equation]
where 2  L  L  1 , t  2 and the prior parameters on the right hand side are taken at iteration 177 t  2 .
The corresponding ML-AMP iterations are given by,
[Equation]
For the boundary cases t = 1 , l = 1 , and l = L , the iterations (3), (4) are modiﬁed as follows.1.
At t = 1 , we initialize B ( l ) i ⇠ P ( l ) B 0 and ! ( l ) µ ⇠ P ( l ) ! 0 , where P ( l ) B 0 , P ( l ) {+! 
+}
are the distributionsof the signal model parameters (2) when x i ⇠ P X .
We take ( A ( l ) i )  1 =
Var ( B ( l ) i ) and
V ( l ) µ = Var ( !
( l ) µ ) .2. At l = 1 , the denoiser g (1) µ ( t ) = @ !
log Z (1) ( y, V (1) µ , ! (1) µ ) , where
[Equation]
[Equation]
At l = L , the denoiser ˆ h ( L ) ( t ) = @
B log Z ( L ) ( A ( L ) i , B ( L ) i ) , where
[Equation]
3.3.1 Computational Savings of MCC Matrices
As ML-AMP requires only matrix-vector products, its computational burden can be signiﬁcantlyreduced when using structured and/or sparse sensing matrices.
In our setting, multi-channel convolutions M ⇠ MCC ( D, P, q, k ) have DPk nonzero coordinates, compared to DPq 2 nonzerocoordinates of a Gaussian i.i.d.
matrix.
Typically, k represents the size of a localized ﬁlter appliedto a larger image, with k ⌧ q [Gonzalez and Woods, 2008, Section 3.4], leading to signiﬁcantspace savings by a factor k/q 2 .
This same is true in convolutional neural networks, where theuse of localized convolutional ﬁlters represents an inductive bias towards localized features that isconsidered a key aspect of their practical success Krizhevsky et al.
[2012], Zeiler and Fergus [2014].
In addition to space savings, specialized matrix-vector product implementations can reduce the timecomplexity of ML-AMP with MCC sensing matrices.
Simple routines for sparse matrix-vectorproducts run in time proportional to the number of nonzero entries, resulting in the same k/q 2constant factor speed up for MCC matrix-vector products.
Alternatively, if k  log q , then a simplealgorithm using a fast Fourier transform for convolution-vector products yields time complexity
O ( DPq log q ) .
Such an algorithm is sketched Appendix B.
Main result
We now state our main technical result, starting with the set of required assumptions.(A1) for any 1 6 l 6 L , the function ' l is continuous and there exists a polynomial b ( l ) of ﬁniteorder such that, for any x 2 R , | ' ( l ) ( x ) | 6 | b ( l ) ( x ) |
(A2) for any 1 6 l 6 L , the matrix W ( l ) is sampled from the ensemble M ( D l , P l , k l , q l ) where P
[Equation]
(A3) the iteration 4 is initialized with a random vector independent of the mixing matrices verifying
N k h 0 k 22 < + 1 almost surely (A4) for any 1 6 l 6 L , D l , P l ! 1 with constant ratio  l = D l /P l , with ﬁnite q l .
Under these assumptions, we may deﬁne the following state evolution recursion
Deﬁnition 4.1 (State Evolution) .
Consider the following recursion,
[Equation]
where ⌧ ( l ) is the second moment of P ( l ) B 0 , where the right hand side parameters are taken at time t ,and the expectations E ( l ) are taken with respect to
[Equation]
At t = 1 , the state evolution is initialized at  ( l ) =
and (ˆ  ( l ) )  1 = ⌧ ( l ) .
At the boundaries l = 1 , L ,the expectations are modiﬁed analogously to the ML-AMP iterations as described by Manoel et al.[2017].
We then have the following asymptotic characterization of the iterates from the convolutional
ML-AMP algorithm
Theorem 4.2.
Under the set of assumptions (A1)-(A4), for any sequences of uniformly pseudoLipschitz functions N 1 , N 2 of order k , for any 1 6 l 6 L and any t 2 N , the following holds
[Equation]
) where Z l ( t ) ⇠ N (0 ,  l ( t )) , ˆ Z l ( t ) ⇠ N (0 , ˆ  l ( t )) are independent random variables.
4.1 Proof Sketch
The proof of Theorem 4.2, which is given in Appendix A, has two key steps.
First, we constructpermutation matrices U, ˜
U such that for W ⇠ MCC ( D, P, q, k ) , the matrix ˜ W = UW ˜ U T is a blockmatrix whose blocks either have i.i.d.
Gaussian elements or are zero valued, and has a block-circulantstructure.
The effect of the permutation is that entries of ˜ W which are correlated due to circulantstructure of W are relocated to different blocks.
Once these permutation matrices are deﬁned,we deﬁne a new, matrix-valued AMP iteration involving the dense Gaussian matrices obtainedfrom the permutations, and whose non-linearities account for the block-circulant structures and thepermutation matrices.
The state evolution of this new iteration is proven using the results of Gerbelotand Berthier [2021].
This provides an explicit example of how the aforementioned results can be usedto obtain rigorous, non Bayes-optimal SE equations on a composite AMP iteration.
The separabilityassumption is key in showing that the AMP iterates obtained with the convolutional matrices can beexactly embedded in a larger one.
Note that this is a stronger result than proving SE equations for analgorithm that computes marginals of a random convolutional posterior: we show the SE equationsare the same as in the dense case.
We ﬁnally invoke the Nishimori conditions, see e.g. Krzakala et al.[2012], to simplify the generic, non Bayes-optimal SE equations to the Bayes-optimal ones.
The idea of embedding a non-separable effect such as a block-circulant structure or different variancesin a mixing matrix is the core idea in the proofs of SE equations for spatially coupled systems, notablyas done in Javanmard and Montanari [2013], Donoho et al.
We note that in the numericalexperiments shown at Figure 1, the parameter q , considered ﬁnite in the proof, is actually comparableto the number of channel, considered to be extensive.
Empirically we observe that this does nothinder the validity of the result, something that was also observed in the spatial coupling literature,e.g. Krzakala et al.
[2012], where large number of different blocks in spatially coupled matrices were
z 11 wz 12 wz 13 w 13 z 11 wz 12 wz 13 w 13 wz 11 wz 12 wz 13 z 21 wz 22 wz 23 w 23 z 21 wz 22 wz 23 w {+
+} wz 21 wz 22 wz 23 z 31 wz 32 wz 33 w 33 z 31 wz 32 wz 33 w 33 wz 31 wz 32 wz 33 z 41 wz 42 wz 43 w 43 z 41 wz 42 wz 43 w 43 wz 41 wz 42 {+wz    z 11 z 12 z 13 w 11 w 12 w 13 z 
+}
z
z
w 21 w 22 wz 31 z 32 z 33 w 31 w 32 w 33 z 41 z {+
+} z 43 w 41 w 42 wz 11 z 12 z 13 w 11 w 12 w 13 z 21 z 22 z 23 w 21 w 22 wz 31 z 32 z 33 w 31 w 32 w 33 z 41 z {+
+} z 43 w 41 w 42 ww 11 w 12 wz 11 z 12 z 13 w 21 w 22 wz 21 z 22 z 23 w 31 w 32 wz 31 z 32 z 33 w 41 w 42 wz {+
+} z 42 z  considered, with convincing
The existence of permutations matrices verifying the property described above is formalized in thefollowing lemma:
Lemma 4.3 (Permutation Lemma) .
Let W ⇠ M ( D, P, k, q ) be a multi-channel convolution matrix.
There exist row and column permutation matrices
U 2 R Dq ⇥ Dq , ˜
U 2 R Pq ⇥
Pq such that ˜ W =
UW ˜ U T is a block-convolutional matrix with dense, Gaussian i.i.d.
blocks.
That is,˜ W = 1 p k
A (1)
A (2) . . .
A ( k )
A (1)
A (2) . . .
A ( k ) ...
A (2) . . .
A ( k ) ...
A (2)
A (3) . . .
A ( k )
A (1)  where each A ( s ) 2 R D,P , 1  s  k has i.i.d.
N (0 , 1 /P ) coordinates.
Proof.
Consider the elements of the matrix M which are non-zero and sampled i.i.d.
as opposed toexact copies of other variables.
They are positioned on the ﬁrst line of each block of size q ⇥ q , andthus the indexing for their lines and columns can be written as M aq +1 ,bq + c where a, b, c are integerssuch that 0 6 a 6 D  1 , [-
 ⩽-]0 6 b 6 P  1 and 1 6 c 6 k .
The integers a, b describe the position ofthe q ⇥ q block the variable is in, and c describes, for each block, the position in the initial random
Gaussian vector of size k that is zero-padded and circulated to generate the block.
The goal is toﬁnd the mapping that groups these variables into k dense blocks of extensive size D ⇥ P .
To doso, one can use the following bijection ˜ M  , ↵ P +  =
M aq +1 ,bq + c where  = a + 1 , ↵ = c  1 and 257  = b + 1 .
By doing this, c becomes the block index and a, b become the position in the dense block.
This mapping can be represented by left and right permutation matrices which also prescribe thepermutation for the rest of the elements of M .
A graphical sketch of this coordinate permutation isshown in Figure 4.
Numerical Experiments
In this section, we compare state evolution predictions from Theorem 4.2 with a numerical implementation of the ML-AMP algorithm described in Section 3.3.
Our ﬁrst experiment, shown in Figure 1, is a noisy compressive sensing task under a sparsity prior
P X ( x ) = ⇢ N ( x ; 0 , 1) + (1  ⇢ )  ( x ) , where ⇢ is the expected fraction of nonzero componentsof x
Measuremements are generated y
=
Wx 0 + ⌘ for noise ⌘ ⇠ N (0 , 10  4 ) , where W ⇠ MCC ( D, P, q, k ) .
We show recovery performance at sparsity levels ⇢ 2 { 0 . 25 , 0 . 5 , 0 . 75 } as themeasurement ratio  = D/P varies, averaged over 10 independent AMP iterates.
Additionally, weshow convergence of the (averaged) AMP iterates for sparsity ⇢ = 0 .
25 at a range of  near therecovery threshold.
We observe strong agreement between AMP empirical performance and the stateevolution prediction.
The system sizes are P = 1024 , q = 1024 , with D =  P varying.
In Figure 5, we consider two examples of L = 2 , 3 , 4 layer models following Equation (2).
Inboth, the output channel l = 1 generates noisy, compressive linear measurements
y =
z (1) + ⇣ for⇣ i ⇠ N (0 ,  2 ) and for dense couplings W (1) ij ⇠ N (0 , 1 /n (1) ) .
Layers 2  l  4 use MCC couplings
W ( l ) ⇠ MCC ( D l , P l , q, k ) , where qP l = n l and D l =  P l = qn l  1 .
Channel functions { ' ( l ) } varyacross the two experiments.
The input prior is P X ( x ) =
N ( x ; 0 , 1) and the problem parametersare q = 10 channels, ﬁlter size k = 3 , noise level  2 = 10  4 , input dimension n ( L ) = 5000 , andlayerwise aspect ratios  ( L ) = 2 and  ( l ) = 1 for 2  l < L .
Finally, the channel aspect ratio  (1)varies in each experiment.
[Figure]
coordinatewise. This channel function is the popular ReLU activation function used by generative convolutional neural networks such as in Radford et al.
[2015], Bora et al.
We compare the state evolution equations to empirical AMP results in two cases.
In the left panel, weshow multilayer models with identity channel functions, and in the right panel, we show models with
ReLU channel functions.
The latter model captures a simple but accurate example of a convolutionalgenerative neural network.
Discussion and Future Work
We have proven state evolution recursions for the ML-AMP algorithm for signal recovery from multilayer convolutional networks.
We consider networks whose weight matrices are drawn either i.i.d.
Gaussian or from an ensemble of random multi-channel convolution matrices.
Interestingly, under aseparable prior and separable channel functions, these two matrix ensembles yield the same stateevolution (up to a rescaling).
These predictions closely match empirical observations in compressivesensing under a sparsity prior (Figure 1) and under multi-layer priors (Figure 5).
Lemma (4.3) allows to rewrite an MCC matrix M as a block circulant matrix ˜ M with randomextensive blocks, reminiscent of the block structure of spatially coupled sensing matrices. As aconsequence of separability, the nonzero blocks of ˜ M have identical statistics, which is key to ourequivalence theorem between MCC matrices and their dense i.i.d. counterparts. This is in contrast tospatial coupling, where extensive blocks may have different variances, or equivalently when denoisingfunctions may be non-separable.
We prove in Appendix A a more general result for non-separablechannel functions, of which Theorem 4.2 is a specialization to the separable case. In Appendix D, wediscuss a potential application to signal recovery with non-i.i.d. convolutional ﬁlters, in which thedynamics of ML-AMP is expected to differ from the analogous fully connected model. Ultimately,studying non-separable models is an interesting and potentially fruitful avenue for future work.
Another important direction for future work is to go beyond random convolutional layers and studyhow to account for trained layers in the ML-AMP algorithm and its state evolution.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’scontributions and scope?
[Yes](b) Did you describe the limitations of your work?
[Yes] See Section 3.2(c) Did you discuss any potential negative societal impacts of your work?
[No] We believethat because of the theoretical nature of this work, any potential negative societalimpacts would be highly indirect.
We are not aware of any potential direct negativesocietal impacts to discuss.(d) Have you read the ethics review guidelines and ensured that your paper conforms tothem?
[Yes]2.
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes] See Section 4(b) Did you include complete proofs of all theoretical results?
[Yes] See Section 4 and
Appendix A3.
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi-
mental results (either in the supplemental material or as a URL)?
[Yes] The code to
reproduce our results is included in supplemental material and will be made publically
available on Github.
(b) Did you specify all the training details (e
g., data splits, hyperparameters,
how they
were chosen)?
[N/A]
Our results do not require training
(c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[No] (d) Did you include the total amount of compute and the type of resources used (e
g., type
of GPUs, internal cluster, or cloud provider)?
[No]
If you are using existing assets (e
g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[N/A]
(b) Did you mention the license of the assets?
[N/A]
(c) Did you include any new assets either in the supplemental material or as a URL?
[N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[N/A]
(e) Did you discuss whether the data you are using/curating contains personally identiﬁable
information or offensive content?
[N/A]
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if
applicable?
[N/A]
[-with respect to the problem dimensions ( n l ) 
-]
[Table]
[Table]
[Table]