Multi-layer State Evolution Under Random Convolutional Design
Abstract
Signal recovery under generative neural network priors has emerged as a promising direction in statistical inference and computational imaging.
Theoretical analysis of reconstruction algorithms under generative priors is, however, challenging.
For generative priors with fully connected layers and Gaussian i.i.d.
weights, this was achieved by the multi-layer approximate message (ML-AMP) algorithm via a rigorous state evolution.
However, practical generative priors are typically convolutional, allowing for computational benefits and inductive biases, and so
the
Gaussian i.i.d.
weight assumption is very limiting.
In this paper, we overcome this limitation and establish the state evolution of ML-AMP for random convolutional layers.
We prove in particular that random convolutional layers belong to the same universality class as Gaussian matrices.
Our proof technique is of an independent interest as it establishes a mapping between convolutional matrices and spatially coupled sensing matrices used in coding theory.
Introduction
In a typical signal recovery problem one seeks to recover a data signal x 0 given access to measurements y
= G θ ( x 0 ) , where the parameters θ of the signal model are known.
In many problems, it is natural to view the measurement generation process as a composition of simple forward operators, or ‘layers.’
In this work, we are concerned with multi-layer signal models of the form
[Equation]
where W ( l ) ∈ R n l − 1 × n l are linear sensing matrices and where φ ( l ) ( z ; ζ ) are separable channel functions that may be non-linear and may depend on unknown channel noise ζ .
In the multi-layer case L > 1 , this signal model can be viewed as a simple instance of recovery under a Generative
Neural Network (GNN) prior, a technique which has recently shown promise as a generalization of sparsity priors for signal processing applications Bora et al.
For example, Gaussian compressive sensing y
= AG θ ( x 0 )
+ ζ under a prior with random Gaussian weights can be naturally viewed as an instance of the multilayer signal model, and given an estimate ˆ x , one can recover the target signal ˆ s =
G θ (ˆ x ) from the weights { W ( l ) } and noise-independent channel functions { φ ( l ) ( z ) } .
In practical settings GNNs often use structured convolutional weight matrices , preventing the direct application of estimators that require W ( l ) to have unstructured entries.
Motivated by this, we take interest in a variant of the recovery problem (1) in which some of the sensing matrices W ( l ) may be [Figure] than showing multi-channel convolutional (MCC) matrices, having a certain block-sparse circulant structure which captures the convolutional layers used by many modern generative neural network architectures
Karras et al.
In this work, we develop an asymptotic analysis of the performance of an Approximate Message
Passing (AMP)
algorithm Donoho
et al.
[2009] for recovery from multichannel convolutional signal models.
This family of algorithms originates in statistical physics Mézard and Montanari [2009],
Zdeborová and Krzakala [2016] and allows to compute the marginals of an elaborate posterior distribution defined by inference problems involving dense random matrices.
A number of AMP iterations have been proposed for various inference problems, such as compressed sensing Donoho et al.
[2009], low-rank matrix recovery Rangan and Fletcher [2012] or generalized linear modeling
Rangan [2011].
More recently, composite AMP iterations (ML-AMP) have been proposed to study multilayer inference problems Manoel et al.
[2017], Aubin et al.
Here we consider the
ML-AMP proposed in Manoel et al.
[2017] to compute marginals of a multilayer generalized linear model with unstructured random weights, but we replace these weights random convolutional weight matrices.
A major benefit of AMP lies in the fact that the asymptotic distribution of their iterates can be exactly determined by a low-dimensional recursion: the state evolution equations.
This enables to obtain precise theoretical results for the reconstruction performance of the proposed algorithm.
Another benefit of such iterations is their low computational complexity, as they only involve matrix-multiplication and, in the separable case, pointwise non-linearities.
Previous works on AMP suggest that the state evolution is not readily applicable to our setting because its derivation requires strong independence assumptions on the coordinates of the { W ( l ) } which are violated by structured multi-channel convolution matrices.
Despite this, we use AMP for our setting and rigorously prove its state evolution.
Our main contributions are:
We rigorously prove state evolution equations for models of the form (1), where weights are allowed to be either i.i.d.
Gaussian or random structured MCC matrices, as in Definition 3.2. 2.
For separable channel functions φ ( l ) and separable signal priors, we show that the original
ML-AMP of Manoel et al.
[2017] used with sensing matrices that are either dense Gaussian matrices or random convolutional ones admits the same state evolution equations, up to a rescaling.
Multi-layer MCC signal models can therefore simulate dense signal models while making use of fast structured matrix operations for convolutions.
The core of our proof shows how an AMP iteration involving random convolutional matrices
may be reduced to another one with dense Gaussian matrices
We first show that random convolutional matrices are equivalent, through permutation matrices, to dense Gaussian ones with a (sparse) block-circulant structure.
We then show how the block-circulant structure can be embedded in a new, matrix-valued, multilayer AMP with dense Gaussian matrices, the state evolution equations of which are proven using the results of Gerbelot and Berthier [2021], with techniques involving spatially coupled matrices Krzakala et al.
[2012], Javanmard and Montanari [2013].
We validate our theory numerically and observe close agreement between convolutional AMP iterations and its state evolution predictions, as shown in Figure 1 and in Section 5.
Our code can be used as a general purpose library to build compositional models and evaluate AMP and its state evolution.
We make this code available at https://github. com/mdnls/conv-ml-amp.git .
Related Work
AMP-type algorithms arose independently in the contexts of signal recovery and spin-glass theory.
In the former case, Donoho et al.
[2009] derives AMP for Gaussian compressive sensing.
This approach was later generalized by Rangan [2011] to recovery problems with componentwise channel functions that may be stochastic and/or nonlinear, and generalized further by Manoel et al.
[2017] to multi-layer or compositional models.
Due to the versatility of this approach, a wide variety of general purpose frameworks for designing AMP variants have since been popularized Fletcher et al.
[2018], Baker et al.
[2020], Gerbelot and Berthier [2021].
Proof techniques to show the concentration of AMP iterates to the state evolution prediction developed alongside new variants of the algorithm.
In the context of spin-glass theory, Bolthausen’s seminal work Bolthausen [2009] introduces a Gaussian conditioning technique used widely to prove AMP concentration.
Following this approach, Bayati and Montanari [2011], Javanmard and Montanari [2013], Berthier et al.
[2020] treat signal models with dense couplings and generalized channel functions.
More recently, a proof framework adaptable to composite inference problems was proposed in Gerbelot and Berthier [2021], which we use in our proof.
There has also been significant interest in relaxing the strong independence assumptions required by
AMP algorithms on sensing matrix coordinates.
In one direction, Vector AMP (VAMP) algorithms target signal models whose sensing matrices are drawn from right orthogonally invariant distributions.
The development of VAMP algorithms followed a similar trajectory to that of vanilla AMP Schniter et al.
[2016], Fletcher et al.
[2018], Rangan et al.
[2019], Baker et al.
The MCC ensemble considered in this work is not right orthogonally invariant, but we observe strong empirical evidence that an analogous version of Theorem 4.2 holds for VAMP as well, as described in Appendix C.2.
In a second direction, there has been much interest in spatial coupling sensing matrices, which were used to achieve the information-theoretically optimal performance in sparse compressive sensing Donoho et al.
[2013], Barbier et al.
[2015], Krzakala et al.
[2012], with complementary state evolution guarantees Javanmard and Montanari [2013].
The concept of spatial coupling and proofs of its performance originated in the literature of error correcting codes Kudekar et al.
[2011, 2013], where it developed from the so-called convolutional codes Felstrom and Zigangirov [1999].
The connection between spatial coupling and convolution layers of neural networks, that we establish in this paper, is as far as we know novel.
Another direction of related work is the design of generative neural network architectures, and correspondingly, the design of signal recovery procedures that can make use of new generative prior models.
Bora et al.
[2017], one of the original works on compressive sensing with generative models, studies signal recovery under a fully connected VAE prior and a convolutional DC-GAN prior.
More advanced convolutional architectures such as PG-GAN Karras et al.
[2018] and Style-GAN Karras et al.
[2019] have been studied in followup work as priors for a variety of signal recovery problems
Daras et al.
[2021], Gu et al.
For simplicity and theoretical tractibility, we do not consider some fine-grained practical modifications used by these architetures, like batch normalization Ioffe and Szegedy [2015] or strided convolution layers Radford et al.
[2015], focusing instead on the essential elements of simple convolutional networks.
Lastly, while our focus is on feedforward convolutional priors such as GAN/VAE networks, there is growing interest in alternative approaches to signal recovery under neural network priors, such as normalizing flows Rezende and Mohamed [Figure] [2015], Dinh et al.
[2015, 2017], Kingma and Dhariwal [2018], Asim et al.
[2020] and score-based generative models Song and Ermon [2020], Jalal et al.
These approaches fall outside the scope of our work and may be interesting directions for future investigation.
Definition of the problem 3.1 Multi-channel Convolutional Matrices
We focus our attention on multichannel convolution matrices that have localized convolutional filters .
In this section, we introduce our notation and define the random matrix ensembles which are relevant to our result.
We consider block structured signal vectors x ∈ R Pq of the form x =
[ x ( i ) ]
Pi =1 , and we refer to the blocks x
( i ) ∈
R q as ‘channels.’
For any vector of dimension d , we denote by P d ∈ R d × d the cyclic coordinate permutation matrix of order d , whose coordinates are ⟨ e i , P d e j ⟩ =
[ i = j + 1] .
For a block-structured vector x ∈ R Pq , we denote by P P,q ∈ R Pq
× Pq the block cyclic permutation matrix satisfying ( P P,q x ) ( i ) =
x ( i +1) for 1 ≤ i < P , and ( P P,q x ) ( P ) = x (1) .
Similarly, we denote by S i,j ∈ R Pq
× Pq the swap permutation matrix which exchanges blocks i, j : [ S i,j x ] ( i ) =
x ( j ) , [ S i,j x ] ( j ) =
x ( i ) , and [ S i,j x ] ( k ) =
x ( k ) for k ̸
= i, j .
Last, given a vector ω ∈ R k for k ≤ q , denote by Zero-Pad q,k ( ω ) the vector whose first k coordinates are ω , and whose other coordinates are zero.
[Equation]
We define the following ensemble for random multi-channel convolution matrices.
Definition 3.1 (Gaussian i.i.d.
Convolution) .
Let q ≥
k
be integers.
The convolutional ensemble C ( q, k ) contains random circulant matrices C ∈ R q × q whose first row is given by
C
= Zero-pad q,k [ ω ] where ω ∈ R k has i.i.d.
Gaussian coordinates ω
i ∼ N (0 , 1 /k ) .
The remaining rows C i are determined by circulant structure, ie.
C i
=
P
i − 1 q
Zero-pad q,k [ ω ] .
Random multi-channel convolutions are block-dense matrices with independent C ( q, k ) blocks.
Definition 3.2 (Multi-channel Gaussian i.i.d.
Convolution) .
Let D, P ≥ 1 and q ≥ k ≥ 1 be integers.
The random multi-channel convolution ensemble M ( D, P, k, q ) contains random block matrices
M ∈ R Dq × Pq of the form M
[Equation]
 
[Equation]
C
[Equation]
C
[Equation]
[Equation]
[Equation]
[Equation]
C D
[Equation]
[Equation] C D [Equation]   where each C i,j ∼ C ( q, k ) is sampled independently.
Fig.
2 gives a graphical explanation of the link between these matrices and the convolutional layers.
The parameter P ( D ) is the number of input (output) channels, q is the dimension of the input and k the filter size. [Figure]
3.2 Thermodynamic-like Limit and Finite-size Regimes
We prove our main result in a thermodynamic-like limit D, P → ∞ while β = D/P is fixed and q, k
= O (1) .
From a practical perspective, convolutional layers in deep neural networks often use large channel dimensions ( D, P ≫ 1 ), large signal dimensions ( q ≫ 1 ), and a small filter size ( k = O (1) ).
As an example, we show in Figure 3 the sizes of convolutional layers used by the
DC-GAN architecture to generate LSUN images [Radford et al., 2015, Figure 1].
Interestingly, our theoretical predictions do not depend explicitly on the relative sizes of q and ( D, P ) .
We observe empirically that these predictions become accurate at finite sizes of ( D, P ) which may seem small relative to q , and which are realistic from a practical neural network perspective.
For example, in Figure 1, we observe strong empirical agreement with predictions for q = P = 1024 as β and D = βP vary.
3.3 Multi-layer AMP
In this section, we define a class of probabilistic graphical models (PGMs) that captures the inference problems of interest, and we state the Multi-layer Approximate Message Passing (ML-AMP)
Manoel et al.
[2017] iterations, which can be used for inference on these PGMs.
We consider the following signal model.
Definition 3.3 (Multi-layer Signal Model) .
Let { W ( l ) } 1 ≤ l ≤ L be matrices of dimension W ( l ) ∈
R n l − 1 × n l .
Let { φ ( l ) ( z ; ζ ) } 1 ≤ l ≤ L be scalar channel functions for which z is the estimation quantity and ζ represents channel noise.
For vectors z, ζ ∈ R n l − 1 , we write φ ( l ) ( z ; ζ ) to indicate the coordinatewise application of φ ( l ) .
The multi-layer GLM signal model is given by
[Equation]
We assume x ∈ R n L follows a known separable prior, x i ∼ P X ( x ) i.i.d., and that the channel noise ζ ( l ) has i.i.d.
N (0 , ( σ ( l ) ) 2 ) coordinates of known variance ( σ ( l ) ) 2 ≥ 0 .
The full estimation quantities of the model are the coordinates of the vectors { h ( l ) } 1 ≤ l ≤ L , { z ( l ) } 1 ≤ l ≤ L , which are related by
[Equation]
where we take h ( L ) = x 0 .
The corresponding conditional probabilities, which define the factor nodes of the underlying PGM, are given by
[Equation]
To compute the posterior marginals, ML-AMP iteratively updates the parameters of independent 1D
Gaussian approximations to each marginal.
Each coordinate h ( l ) i ( t ) has corresponding parameters { A ( l ) i ( t ) , B ( l ) i ( t ) } and each z ( l ) µ ( t ) has corresponding { V ( l ) µ ( t ) , ω ( l ) µ ( t ) } , where t ≥ 1 indexes the
ML-AMP iterations.
The recursive relationship between these parameters is defined in terms of scalar denoising functions , ˆ h ( l ) and g ( l ) , which compute posterior averages of the estimation quantities given their prior parameters.
In general, these denoising functions can be chosen (up to regularity assumptions) to adjust MLAMP’s performance in applied settings, such as in Metzler et al.
[2015], and in these cases the denoisers may be nonseparable vector valued functions.
However, in the separable, Bayes-optimal regime where P x ( x ) and P ( l ) ( h | z ) are known, the optimal denoisers are given by,
[Equation]
where 2 ≤ L ≤ L − 1 , t ≥ 2 and the prior parameters on the right hand side are taken at iteration t ≥ 2 .
The corresponding ML-AMP iterations are given by,
[Equation]
For the boundary cases t = 1 , l = 1 , and l = L , the iterations (3), (4) are modified as follows.
At t = 1 , we initialize B ( l ) i ∼ P ( l ) B 0 and ω ( l ) µ ∼ P ( l ) ω 0 , where P ( l ) B 0 , P ( l ) ω 0 are the distributions of the signal model parameters (2) when x i ∼ P X .
We take ( A ( l ) i ) − 1 =
Var ( B ( l ) i ) and V ( l ) µ =
Var ( ω ( l ) µ ) .
At l = 1 , the denoiser g (1) µ ( t ) =
∂ ω log Z (1) ( y, V (1) µ , ω (1) µ ) , where
[Equation]
At l = L , the denoiser ˆ h ( L ) ( t ) =
∂ B log Z ( L ) ( A ( L ) i , B ( L ) i ) , where
[Equation]
3.3.1 Computational Savings of MCC Matrices
As ML-AMP requires only matrix-vector products, its computational burden can be significantly reduced when using structured and/or sparse sensing matrices.
In our setting, multi-channel convolutions M ∼ MCC ( D, P, q, k ) have DPk nonzero coordinates, compared to DPq 2 nonzero coordinates of a Gaussian i.i.d.
matrix.
Typically, k represents the size of a localized filter applied to a larger image, with k ≪
q
[Gonzalez and Woods, 2008, Section 3.4], leading to significant space savings by a factor k/q 2 .
This same is true in convolutional neural networks, where the use of localized convolutional filters represents an inductive bias towards localized features that is considered a key aspect of their practical success Krizhevsky et al.
[2012], Zeiler and Fergus [2014].
In addition to space savings, specialized matrix-vector product implementations can reduce the time complexity of ML-AMP with MCC sensing matrices.
Simple routines for sparse matrix-vector products run in time proportional to the number of nonzero entries, resulting in the same k/qconstant factor speed up for MCC matrix-vector products.
Alternatively, if k ≫ log q , then a simple algorithm using a fast Fourier transform for convolution-vector products yields time complexity
O ( DPq log q ) .
Such an algorithm is sketched Appendix B.
Main result
We now state our main technical result, starting with the set of required assumptions. (A1) for any 1 ⩽ l ⩽ L , the function φ ( l ) ( z ; ζ ) is continuous and there exists a polynomial b ( l ) of finite order such that, for any x ∈ R , | φ ( l ) ( x ; ζ ) | ⩽ | b ( l ) ( x, ζ ) | .
(A2) for any 1 ⩽ l ⩽ L , the matrix W ( l ) is sampled from the ensemble M ( D l , P l , k l , q l ) where P l q l = D l − 1 q l −(A3)
the iteration (4) is initialized with a random vector independent of the mixing matrices verifyingN ∥ h 0 ∥
22 < + ∞ almost surely, and the prior distribution P X is subGaussian (A4) for any 1 ⩽ l ⩽ L , D l , P l → ∞ with constant ratio β l = D l /P l , with finite q l .
Under these assumptions, we may define the following state evolution recursion
Definition 4.1 (State Evolution) .
Consider the following recursion,
[Equation]
where τ ( l ) is the second moment of P ( l ) B 0 , where the right hand side parameters are taken at time t , and the expectations E ( l ) are taken with respect to
[Equation]
At t = 1 , the state evolution is initialized at κ ( l ) =
and (ˆ κ ( l ) ) − 1 = τ ( l ) .
At the boundaries l = 1 , L , the expectations are modified analogously to the ML-AMP iterations as described by Manoel et al.
We then have the following asymptotic characterization of the iterates from the convolutional
ML-AMP algorithm
Theorem 4.2.
Under the set of assumptions (A1)-(A4), for any sequences of uniformly pseudoLipschitz functions ψ N 1 , ψ N 2 of order k , for any 1 ⩽ l ⩽ L and any t ∈ N , the following holds
[Equation]
(8) where Z l ( t ) ∼ N (0 , κ l ( t )) , ˆ Z l ( t ) ∼ N (0 , ˆ κ l ( t )) are independent random variables.
4.1 Proof Sketch
The proof of Theorem 4.2, which is given in Appendix A, has two key steps.
First, we construct permutation matrices U, ˜
U such that for W ∼ MCC ( D, P, q, k ) , the matrix ˜ W = UW ˜
U T is a block matrix whose blocks either have i.i.d.
Gaussian elements or are zero valued, and has a block-circulant structure.
The effect of the permutation is that entries of ˜ W which are correlated due to circulant structure of W are relocated to different blocks.
Once these permutation matrices are defined, we define a new, matrix-valued AMP iteration involving the dense Gaussian matrices obtained from the permutations, and whose non-linearities account for the block-circulant structures and the permutation matrices.
The state evolution of this new iteration is proven using the results of Gerbelot and Berthier [2021].
This provides an explicit example of how the aforementioned results can be used to obtain rigorous, non Bayes-optimal SE equations on a composite AMP iteration.
The separability assumption is key in showing that the AMP iterates obtained with the convolutional matrices can be exactly embedded in a larger one.
Note that this is a stronger result than proving SE equations for an algorithm that computes marginals of a random convolutional posterior: we show the SE equations are the same as in the dense case.
We finally invoke the Nishimori conditions, see e.g. Krzakala et al. [2012], to simplify the generic, non Bayes-optimal SE equations to the Bayes-optimal ones.
  z 11 wz 12 wz 13 w 13 z 11 wz 12 wz 13 w 13 wz 11 wz 12 wz 13 z 21 wz 22 wz 23 w 23 z 21 wz 22 wz 23 w 23 wz 21 wz 22 wz 23 z 31 wz 32 wz 33 w 33 z 31 wz 32 wz 33 w 33 wz 31 wz 32 wz 33 z 41 wz 42 wz 43 w 43 z 41 wz 42 wz 43 w 43 wz 41 wz 42 wz   
z 11 z 12 z 13 w 11 w 12 w 13 z 21 z 22 z 23 w 21 w 22 wz 31 z 32 z 33 w 31 w 32 w 33 z 41 z 42 z 43 w 41 w 42 wz 11 z 12 z 13 w 11 w 12 w 13 z 21 z 22 z 23 w 21 w 22 wz 31 z 32 z 33 w 31 w 32 w 33 z 41 z 42 z 43 w 41 w 42 ww 11 w 12 wz 11 z 12 z 13 w 21 w 22 wz 21 z 22 z 23 w 31 w 32 wz 31 z 32 z 33 w 41 w 42 wz 41 z 42 z 
The idea of embedding a non-separable effect such as a block-circulant structure or different variances in a mixing matrix is the core idea in the proofs of SE equations for spatially coupled systems, notably as done in Javanmard and Montanari [2013], Donoho et al.
We note that in the numerical experiments shown at Figure 1, the parameter q , considered finite in the proof, is actually comparable to the number of channel, considered to be extensive.
Empirically we observe that this does not hinder the validity of the result, something that was also observed in the spatial coupling literature, e.g. Krzakala et al.
[2012], where large number of different blocks in spatially coupled matrices were considered, with convincing numerical agreement.
The existence of permutations matrices verifying the property described above is formalized in the following lemma:
Lemma 4.3 (Permutation Lemma) .
Let W ∼ M
( D, P, k, q ) be a multi-channel convolution matrix.
There exist row and column permutation matrices U ∈
R
Dq × Dq , ˜ U
∈ R
Pq ×
Pq such that ˜ W =
UW ˜ U T is a block-convolutional matrix with dense, Gaussian i.i.d.
blocks.
That is,
[Equation] [Equation]
[Equation] [Equation]
[Equation]
A [Equation]
  where each A ( s ) ∈ R D,P , 1 ≤ s ≤ k has i.i.d.
N (0 , 1 /P ) coordinates.
Proof.
Consider the elements of the matrix M which are non-zero and sampled i.i.d.
as opposed to exact copies of other variables.
They are positioned on the first row of each block of size q × q , and thus the indexing for their rows and columns can be written as M aq +1 ,bq + c where a, b, c are integers such that 0 ⩽ a ⩽ D − 1 ,
⩽ b ⩽ P − 1 and 1 ⩽ c ⩽ k .
The integers a, b describe the position of the q × q block the variable is in, and c describes, for each block, the position in the initial random
Gaussian vector of size k that is zero-padded and circulated to generate the block.
The goal is to find the mapping that groups these variables into k dense blocks of extensive size D × P .
To do so, one can use the following bijection ˜
M γ,αP + β
=
M aq +1 ,bq + c where γ = a + 1 , α = c − 1 and β = b + 1 .
By doing this, c becomes the block index and a, b become the position in the dense block.
This mapping can be represented by left and right permutation matrices which also prescribe the permutation for the rest of the elements of M .
Figure 4 shows a sketch of this permutation.
The state evolution recursion given in Definition 4.1 is equivalent to the state evolution for unstructured matrices proposed by Manoel et al.
[2017], which may seem surprising relative to existing literature on spatial coupling, where spatially coupled sensing matrices are used to achieve improved performance over unstructured sensing matrices.
This equivalence is a consequence of the fact that ˜ W has dense
Gaussian blocks which each have the same variance, whereas in the literature, spatially coupled matrices typically have Gaussian blocks with inhomogeneous variances that can be tuned to improve recovery performance.
Numerical Experiments
In this section, we compare state evolution predictions from Theorem 4.2 with a numerical implementation of the ML-AMP algorithm described in Section 3.3.
Our first experiment, shown in Figure 1, is a noisy compressive sensing task under a sparsity prior P X ( x ) =
ρ N ( x ; 0 , 1) + (1 − ρ ) δ ( x ) , where ρ is the expected fraction of nonzero components of x 0 .
Measuremements are generated y
=
Wx 0 + η for noise η ∼ N (0 , 10 − 4 ) , where W ∼ MCC ( D, P, q, k ) .
We show recovery performance at sparsity levels
ρ ∈ { 0 . 25 , 0 . 5 , 0 .
75 } as the measurement ratio β
= D/P varies, averaged over 10 independent AMP iterates.
Additionally, we show convergence of the (averaged)
AMP iterates for sparsity ρ = 0 .
25 at a range of β near the recovery threshold.
We observe strong agreement between AMP empirical performance and the state evolution prediction.
The system sizes are P = 1024 , q = 1024 , with D = βP varying.
In Figure 5, we consider two examples of L = 2 , 3 , 4 layer models following Equation (2).
In both, the output channel l = 1 generates noisy, compressive linear measurements
y =
z (1) + ζ for ζ i ∼ N (0 , σ 2 ) and for dense couplings W (1) ij ∼ N (0 , 1 /n (1) ) .
Layers 2 ≤ l ≤ 4 use MCC couplings
W ( l ) ∼ MCC ( D l , P l , q, k ) , where qP l = n l and D l = βP l = qn l − 1 .
Channel functions { φ ( l ) } vary across the two experiments.
The input prior is P X ( x ) =
N ( x ; 0 , 1) and the problem parameters are q = 10 channels, filter size k = 3 , noise level σ 2 = 10 − 4 , input dimension n ( L ) = 5000 , and layerwise aspect ratios β ( L )
= 2 and β ( l ) = 1 for 2 ≤ l < L .
Finally, the channel aspect ratio β (1) varies in each experiment.
[Figure]
We compare the state evolution equations to empirical AMP results in two cases.
In the left panel, we show multilayer models with identity channel functions, and in the right panel, we show models with
ReLU channel functions.
The latter captures a simple but accurate example of a convolutional GNN.
Discussion and Future Work
We have proven state evolution recursions for the ML-AMP algorithm for signal recovery from multilayer convolutional networks.
We consider networks whose weight matrices are drawn either i.i.d.
Gaussian or from an ensemble of random multi-channel convolution matrices.
Interestingly, under a separable prior and separable channel functions, these two matrix ensembles yield the same state evolution (up to a rescaling).
These predictions closely match empirical observations in compressive sensing under a sparsity prior (Figure 1) and under multi-layer priors (Figure 5).
Based on the discussion in Section 4.1, the equivalence property is expected to break down if ˜ W is replaced by a block matrix ˜ W sp with dense Gaussian blocks organized in the same block-circulant structure, with inhomogeneous variances.
As we discuss in Appendix D, by inverting the permutation lemma, ˜ W sp corresponds to a multi-channel convolution W sp with inhomogeneous convolutional filters, which can be taken as a simple model for structured convolutions.
Inhomogeneous blocks can also easily be incorporated in the Graph-AMP framework using nonseparable denoisers, so the key elements of our proof also apply to the inhomogeneous case.
This generalized model for MCC matrices represents an interesting direction for future exploration, whose state evolution is expected to diverge from that of the dense Gaussian ensemble.
Acknowledgments and Disclosure of Funding
M.D. acknowledges funding from Northeastern University’s Undergraduate Research and Fellowships office and the Goldwater Award.
We also acknowledge funding from the ERC under the European
Union’s Horizon 2020 Research and Innovation Program Grant Agreement 714608- SMiLe, and by the Swiss National Science Foundation grant SNFS OperaGOST, 200021 _ 200390 .
A Proof of the main theorem
The proof of the main theorem is presented in this section.
We start with a generic result on a family of AMP iterations including the (non Bayes-optimal) MLAMP one, using the framework of Gerbelot and Berthier [2021], from which we remind the required notions.
A.1 Notations and definitions
If f : R N × q → R N × q is an function and i ∈ { 1 , . . .
N } , we write f i : R N × q → R q the component of f generating the i -th row of its image, i.e., if X ∈ R N × q ,
[Equation]
We write ∂f i ∂
X i the q × q Jacobian containing the derivatives of f i with respect to (w.r.t.)
the i -th row
X i ∈ R q :
[Equation]
For two sequences of random variables X n , Y n , we write X n P ≃ Y n when their difference converges in probability to 0 , i.e., X n − Y n P −→ 0 .
Oriented graphs with a set of vertices V and edges −→ E are denoted G =
( V, −→ E ) .
The set of edges may be split into right-pointing and left-pointing edges, i.e., −→ E = { [Equation] e [Equation]
[Equation] [Equation] e L } . Definition A.1 (pseudo-Lipschitz function) .
For k ∈ N ∗ and any N, m ∈ N ∗ , a function Φ : R N × q → R m × q is said to be pseudo-Lipschitz of order k
if there exists a constant L such that for any x , y ∈ R N × q ,
[Equation]
For a function φ : R → R , the property becomes
[Equation]
and a straightforward calculation shows that for any scalar pseudo-Lipshitz function of order 2 , the function
[Equation]
is pseudo-Lipschitz of order 2 according to the definition above.
This definition is handy for proofs involving non-separable functions and leads to Gaussian concentration using the Gauss-Poincaré inequality (see Lemma C.8.
from Berthier et al.
[2020]), while in the separable case, a strong law of large number is proven for a class of distributions including sub-Gaussian ones in Lemma 5 of Bayati and Montanari [2011].
A.2 State evolution for generic multilayer AMP iterations with matrix valued variables and dense Gaussian matrices
In the notations of Gerbelot and Berthier [2021], consider the AMP iteration indexed by the following directed graph G =
( V, −→ E ) , where the set of vertices is denoted V
= { v 0 , v 1 , ..., v L } , and the set of edges −→ E
= { −→ e 1 , ..., −→ e l , ←− e 1 , ..., ←− e L } .
For any edge −→ e l , the corresponding matrix A −→ e l has dimensions R n l × n l − 1 with A ←− e l =
A ⊤−→ e l , and the variables x −→ e l ∈ R n l × q , x ←− e l ∈ R n l − 1 × q for some finite q ∈ N , with N = (cid:80) Ll =1 n l .
Finally, we define the non-linearities of the iteration by specifying the variables they are acting on as follows:
[Equation]
any
[Equation]
for any
[Equation]
where w −→ e 1 , ..., w −→ e L are low-rank matrices respectively in R n 0 × q , ..., R n L − 1 × q , whose rows are sampled i.i.d.
from subgaussian probability distributions in R q .
The graph indexing the iteration then reads:
[Equation]
with the corresponding iteration:
[Equation]
and Onsager terms, for the right oriented edges
[Equation]
and left oriented edges
[Equation]
We now make the following assumptions
(A1)
The matrices ( A −→ e ) −→ e ∈−→
E are random and independent, up to the symmetry condition A ←− e = A ⊤−→ e .
Moreover A −→ e has independent centered Gaussian entries with variance 1 /N .
(A2)
For all 1 ⩽ l ⩽ L , n l → ∞ and n l /N converges to a well-defined limit δ l ∈ [0 , 1] .
We denote by n → ∞ the limit under this scaling.
(A3)
For all t ∈ N and −→ e ∈ −→ E , the non-linearity f t −→ e is pseudo-Lipschitz of finite order, uniformly with respect to the problem dimensions ( n l )
⩽ l ⩽ L (A4)
For all −→ e ∈ E , the rows of x 0 −→ e , w −→ e are sampled from subgaussian probability distributions in R q . (A5)
For all −→ e ∈ E , the following limit exists and is finite:
[Equation]
(A6) Let ( κ −→ e )
−→ e ∈ E be an array of bounded non-negative reals and Z −→ e ∼ N (0 , κ −→ e I n w ) independent random variables for all −→ e .
For
all −→ e ∈ E , for any t ∈ N > 0 , the following limit exists and is finite:
[Equation]
(A7) Consider any array of 2 × 2 positive definite matrices ( S −→ e ) −→ e ∈ E and the collection of random variables ( Z −→ e , Z ′ −→
e )
∼ N ( S −→ e ⊗ I n w )) defined independently for each edge −→ e . Then for any −→ e ∈ E and s, t > 0 , the following limit exists and is finite:
[Equation]
Under these assumptions, we define the following state evolution recursion: • for l = 1 :
[Equation]
[Equation] [Equation] [Table]
[Table]
[Equation]
[Table]
[Equation] [Equation]
L
[Equation]
where, for any 1 ⩽ l ⩽ L , the symbol ∂
z w −→ e ,i , ϕ −→ e denotes the partial derivative w.r.t.
the argument of ϕ −→ e , ( Z 1 −→ e , . .
, Z t −→ e ) is a centered Gaussian random vector with covariance ( κ r,s −→ e ) r,s ≤ t ⊗ I n w (and similarly for left-oriented edges), and z w −→ e is distributed according to N (0 , ρ w −→ e ) .
Theorem A.2.
Assume (A1)-(A7).
Define, as above, independently for each −→ e l , Z 0 −→ e l = x 0 −→ e l and ( Z 1 −→ e l , . . .
, Z t −→ e l ) a centered Gaussian random vector of covariance ( κ r,s −→ e l ) r,s ≤ t ⊗ I n l − 1 .
Then for any sequence of uniformly (in n ) pseudo-Lipschitz function Φ : ( R n l − 1 × ( t +1) q ) 2 → R , for any 1 ⩽ l ⩽ L
[Equation]
In summary, at each time step, the variables associated with right oriented edges x −→ e l asymptotically behave as the sum of the ground truth w −→ e l reweighted by a q × q matrix coefficient ˆ ν ←− e l and a n l − 1 × q random matrix with i.i.d.
rows Z −→ e l with q × q covariance κ ←− e l determined by the function associated to the corresponding left-oriented arrow f t ←− e l .
Similarly, the variables associated with left oriented edges x ←− e l asymptotically behave as the sum of the linear response to the ground truth z w −→ e l (asymptotic equivalent of A −→ e l w −→ e l ) reweighted by a q × q matrix coefficient ν ←− e l and a n l × q random matrix with i.i.d.
rows Z ←− e l with q × q covariance κ −→ e l determined by the function associated to the corresponding right-oriented arrow f t −→ e l .
Proof.
This result is a special case of Lemma 2 from Gerbelot and Berthier [2021], with a perturbation where only the left-oriented edges involve an additional dependence on A −→ e w −→ e .
The required conditions are the same as in Gerbelot and Berthier [2021], barring the subgaussian assumption (A3) which ensures the scaled norm of the x 0 −→ e , w −→ e are finite with high-probability as n → ∞ .
A.3 State evolution for multilayer AMP iterations with random convolutional matrices
The following lemma proves the state evolution equations for a multilayer AMP iteration where the dense Gaussian matrices are replaced with random convolutional ones (MCC from Def.3.2) with variance 1 N , with a vector valued variables, i.e. q=1, and separables non-linearities.
We choose the variance as 1 N to follow the notations of Gerbelot and Berthier [2021] for more convenience, recovering the variances of iteration Eq.(4) is a straightforward rescaling as done in Berthier et al. [2020] and will be discussed in the next section.
Assume q = 1 and that, for any t ∈ N and 1 ⩽ l ⩽ L , the functions f t −→ e l , f t ←− e l are separable in all their arguments, i.e there exists scalar valued, pseudo-Lipschitz functions σ t −→ e l : R 2 → R , σ t ←− e l : R 3 → R (where σ t −→ e 1 : R → R , σ t ←− e L : R 2 → R ) such that:
for l = 1 , for any 1 ⩽ i ⩽ n 0 :
[Equation]
for any 1 ⩽ l ⩽ L − 1 , for any 1 ⩽ i ⩽ n l :
[Equation]
for any 2 ⩽ l ⩽ L , 1 ⩽
i ⩽ n l − 1 :
[Equation]
for l=L, any 1 ⩽
i ⩽ n L :
[Equation]
Define the following scalar SE equations
[Equation]
any 2 ⩽ l ⩽ L −
[Equation] [Equation]
Lemma A.3.
Under the assumptions of section A [Equation] 3, define, as above, independently for each −→ e l ,
Z 0 −→ e l = x 0 −→ e and ( Z 1 −→ e l , . . .
, Z t −→ e l ) a centered Gaussian random vector of covariance ( κ r,s −→ e l ) r,s ≤ t (and similarly for left-oriented edges).
Then for any 1 ⩽ l ⩽ L , for any sequence of uniformly (in n ) pseudo-Lipschitz function Φ
[Equation]
Proof.
Consider the following iteration, corresponding to the algorithm presented in the previous section Eq.(56) with q = 1 indexed on the same graph as above, but where the matrices A −→ e l are replaced with random convolutional ones, denoted ˆ A −→ e l such that
[Equation]
where A −→ e l ∈ R D −→ e l q −→ e l × P −→ e l q −→ e l , and we remind that we chose variances of 1 /N .
Since we assume that q = 1 , thus the Onsager terms are scalars, which we denote with lowercase letters b t −→ e .
The corresponding iteration then reads:
[Equation]
Then
[Equation] according to Lemma 4. [Equation] for any 1 ⩽ l ⩽ L
[Equation]
there exists a pair of orthogonal matrices
U −→ e l ∈ R D −→ e l q −→ e l × D −→
e l q −→ e l , V −→ e l ∈ R P −→
e l q −→ e l × P −→ e l q −→ e l such that ˆ A −→ e l = U −→ e l ˜
A −→ e l V ⊤−→ e l and ˜
A −→ e l = (cid:20)(cid:16)
P P −→ e l ,q −→ e l (cid:17)
i − 1 Q −→ e l (cid:21) q −→ e l i = , where Q −→ e l ∈ R D −→ e l × P −→ e l q −→ e l is composed of q −→ e l blocks of size D −→ e l ×
P −→ e l , denoted Q j −→ e l , verifying
• for any 1 ⩽ j ⩽ k −→ e , Q j −→ e has i.i.d.
N (0 , 1 N ) elements • for any k −→ e < j ⩽ q −→ e , all elements of Q j −→ e are zero.
In the preceding definition of ˜ A −→ e l , Q −→ e l is understood as a vector of size R P −→ e q −→ e with elements in
R D −→ e , such that the permutation matrix P P −→ e ,q −→ e shifts blocks of size D −→ e × P −→ e , yielding
[Equation]

[Equation]
[Equation] [Equation] [Equation] Q
[Equation]
Q
[Equation]
Q
[Equation]

[Equation]
[Equation]
The iteration then reads
[Equation]
Since we will not be making any change of variable on the w −→ e l
[Equation]
we will keep the ˆ A −→ e l notation for the quantities related to the planted model.
Define, for any 1 ⩽ l ⩽ L and any t ∈ N :
[Equation]
Using the orthogonality of the permutation matrices U −→ e , V −→ e , the iteration may be rewritten
[Equation]
Recall, for any 1 ⩽ l ⩽ L , the dimensions ˜
A −→ e l ∈ R D −→ e l q −→ e l × P −→ e l q −→ e l and ˜ f t −→ e l ( ... )
∈ R P −→ e l q −→ e l .
Consider then
[Equation]
[Equation]
where, for any 1 ⩽ k ⩽ q −→ e l , ( ˜ f t −→ e l ) ( k )
( ... ) ∈ R P −→ e .
The product ˜
A −→ e l ˜ f t −→ e l ( ... )
∈
R D −→
e l q −→ e l then reads, using the circulant structure of ˜
A −→ e l
[Equation]

[Equation]
where the notation ⌊ .
⌋ q −→ e l denotes the modulo q −→ e l , i.e. the remainder of the euclidian division by q −→ e l .
Now define
[Equation]
) and the matrix ˜ Q −→ e l ∈ R D −→ e l q −→ e l × P −→ e l q −→ e l is a dense Gaussian matrix with i.i.d.
elements.
Then ˜ Q [Equation] ˜
[Equation]
where each . .
. is an identical copy of the first D −→ e l × q −→ e l block, for a total of k −→ e l blocks.
This means the D −→ e l q −→ e l output of the product ˜
A −→ e l f t −→ e l ( ... ) may be rewritten as a D −→ e l × q −→ e l matrix (copied k −→ e l times) resulting from the product of a dense Gaussian matrix with i.i.d.
elements and a matrix valued function ˜ F t −→ e l which verifies the same regularity conditions as f t −→ e l .
Note that, owing to the separability assumption, we may use any permutation of the ( ˜ f t −→ e l ) ( i ) , 1 ⩽
i ⩽ q −→ e l and will thus drop the permutations to write
[Equation]
Similarly, for products of the form (cid:16) ˜
A −→ e l (cid:17) ⊤ ˜ f t ←− e l ( ... )
∈ R P −→ e l q −→ e l , we may write:
[Equation]

[Equation]
Then, using once again the separability assumption, we may define:
[Equation]
such that the term ˜ Q ⊤−→ e l ˜ F t ←− e l ( ... ) also contains k −→ e l copies of a P −→ e l × q −→ e l block containing the q −→ e l blocks of size P −→ e l of the original P −→ e l q −→ e l vector ˜
A ⊤−→ e l ˜ f ←− e l ( ... ) .
The iterates of the sequences defined by Eq.(59) may then be rewritten as a subset of the rows of the following matrix valued iteration, i.e.:
[Equation]
69) where each W −→ e l contains k −→ e l copies of the initial w −→ e l reorganised into matrices as described above.
The dimensions of the variables are Note that at this point we have almost reached an iteration verifying the structure of that appearing in Theorem A.2, except the Onsager term isn’t, a priori, the correct one.
Consider the following iteration, where we replaced the original, scalar Onsager terms with the correct, matrix-valued ones:
[Equation]
[Equation]
where, for any −→ e ∈ −→ E and any t ∈ N for the right oriented edges
[Equation]
and left oriented edges
[Equation]
Using the separability assumption
[Equation]
we can simplify this expression.
To take a concrete example, consider ˜ F t −→ e 2 (cid:16)
˜
X t −→ e 1 , ˜ X t ←− e 2 (cid:17) .
Let’s start with the dimensions.
Recall
[Equation]
where
[Equation]
using the separability assumption
[Equation]
we may write
[Equation] [Equation]
And
[Equation]
76) where
[Equation]
where each ˜ x t, ( i ) ←− e 2 ∈ R P −→ e 2 q −→ e 2 .
Recall that, for any 1 ⩽ i ⩽
Pk , ˜ F t −→ e 2 ,i : R q −→ e 2 → R q −→ e 2 .
Then, for any 1 ⩽ k, l ⩽ q −→ e
[Equation]
where we wrote J ( g t −→ e 2 )
[Equation]
k ) the N × N Jacobian matrix of the function ( g t −→ e 2 ) ( k ) :
R N → R N .
Using
Berthier et al.
[2020] corollary 2, the Onsager term can be replaced by any estimator based on the asymptotically Gaussian iterates converging, in the high-dimensional limit, to the correct expectation.
The adaptation to the graph framework of Gerbelot and Berthier [2021] is immediate (see the proof in Berthier et al.
[2020] and corresponding comment in Gerbelot and Berthier [2021]).
Using the permutation invariance of the Gaussian distribution, we can therefore replace each element of the matrix the Onsager term with
[Equation]
which amounts to
[Equation]
[Equation] We therefore obtain an exact reformulation of the initial MLAMP iteration with convolutional matrices in terms of a subset (first row of size P −→ e l × q −→ e l for right oriented edges and D −→ e l × q −→ e l for left-oriented variables) of the variables of a matrix-valued iteration with dense Gaussian matrices verifying the SE equations.
Isolating the aforementioned first lines, recalling that the SE equations prescribes i.i.d.
rows in the asymptotically Gaussian fields, we recover that, for any 1 ⩽ l ⩽ L , the variable x −→ e l ∈ R P −→ e l q −→ e l is composed of q −→ e l copies of block of size P −→ e l with i.i.d.
Gaussian elements distributed according to the SE equations (A.3).
The distribution of the variables associated to left-oriented edges is obtained similarly.
Note that, from a finite size point of view, the effect of D −→ e l , P −→ e l is different from that of q −→ e l : the former results in subGaussian concentration i.e. exponential in the dimension, while the latter only represents copies (and not i.i.d.
samples), and thus only has an averaging effect.
This is observed in simulations.
A.4 Bayes-optimal MLAMP with random convolutional matrices
In this section, we specialize the equations obtained in the previous section to the Bayes-optimal
MLAMP iteration of the main body of the paper.
Several functions are reminded for convenience.
Consider the MLAMP iteration outlined in section 3.3.
The scalar updates described in Eq.(4) can be rewritten as vector-valued updates as follows, for any t ∈ N , and any 0 ⩽ l ⩽ L :
[Equation]
[Equation]
To define the update functions and terms V ( l ) , ˆ V ( l ) , the following partition functions were introduced.
• for l =
[Equation]
• for any 2 ⩽ l ⩽ L −
[Equation]
• for l = L
[Equation]
We then define the layer-dependent, time-dependent, scalar update functions
[Equation]
and their corresponding separable, vector valued counterparts f ( l ) , ˜ f
( l ) , which leads to the following iteration
[Equation]
where the Onsager terms V ( l ) ,t and ˆ V ( l ) ,t reduce to, using the separability of the update functions,
[Equation]
We now show that the update functions defined above are Lipschitz continuous and increasing, thus ensuring that the integrals are well defined through positivity of the parameters V, ˆ V .
Lemma A.4.
For any 1 ⩽ l ⩽ L , and any t ∈ N , the functions f ( l ) ,t , ˜ f ( l ) ,t are Lipschitz continuous in B, ω .
Furthermore, the functions f ( l ) ,t , ˜ f ( l ) ,t are respectively decreasing in ω and increasing in B .
As a consequence, the variance terms A ( l ) ,t and V ( l ) ,t are strictly positive.
Proof.
Recall the partition function, omitting the layer index since all regularity assumptions are the same for all layers and time indices,
[Equation]
recalling p ( h | z )
= (cid:82) p ( ξ ) δ ( h − f ξ ( z ))
dξ , integrating in h yields
[Equation]
Starting with ˜ f [Equation] we can straightforwardly verify the conditions to apply the dominated convergence theorem and differentiate under the integral to obtain
[Equation]
where the positivity comes from the Cauchy-Schwarz inequality and positivity of the term
P
[Equation]
(cid:17) .
Turning to f
[Equation]
we complete the square in the variable h to obtain
[Equation]
and differentiating under the integral yields
[Equation]
where the term
[Equation]
is the conditional mean of the distribution with density
[Equation]
The Lipschitz property is straightforward to verify using the polynomial bound assumption on the activation functions and the inverse exponential factors.
In the Bayes-optimal MLAMP, see Manoel et al.
[2017], the planted vectors w −→ e l are chosen as independently distributed as the asymptotic SE representation of the output of the previous layer, and are therefore Lipschitz transforms of subGaussian random variables, and thus are also subgaussian.
Using the permuation invariance of the Gaussian distribution, the quantities z −→ e l =
ˆ
A −→ e l remain Gaussian.
We can therefore apply the result of Lemma A.3 to this iteration and obtain that iterates of Eq.(4) verify the SE equations from Lemma A.3 with the corresponding update functions.
Furthermore, in the Bayes optimal case, the Nishimori conditions, see e.g. Krzakala et al.
[2012], allow to only keep the parameters ν −→ e l , ˆ ν ←− e l to describe the distribution of of the iterates, recovering the equations of Theorem 4.2.
Finally, the rescaling of the variances to go from the factors δ l to the β l of the main can be done by rescaling each non-linearity f t −→ e l by (cid:112) N/n l − 1 (and similary for the f t ←− e l with (cid:112) N/n l ) as done in Javanmard and Montanari [2013], Berthier et al.
B Fast MCC-vector Products
Here is a simple sketch of an algorithm for multiplying M ∼ MCC ( D, P, q, k ) with a vector v ∈ R Pq that runs in time O ( DPq log q ) .
If k ≫ log q , this improves on the runtime required by a simple sparse matrix-vector product.
We use Matlab index notation for matrix and vector coordinates, for example M [ i : j, k ] =
[ M rk : r = i . . .
j ] , and we write shorthand M ij for M [ i, j ] .
Data: matrix M ∼ MCC ( D, P, q, k ) , vector v ∈ R Pq
Initialize s ∈ R Dq the zero vector; for i = 1 . . .
D do for j = 1 . . .
P do
C ij ← M [ qi : q ( i + 1) , qj : q ( j + 1)] ; ω ij =
C ij [0 , 0 : k ] ; ˆ
ω ij =
FFT ( w ij ) ; ˆ
v j = FFT ( v [ qj, q
( j + 1)] ; ˆ s i
= ˆ ω ij ∗ ˆ v j ; s [ qi : q ( i + 1)] =
IFFT (ˆ s i ) ; end end
[Figure]
C Additional Experiments C.1 Sparse Compressive Sensing
We observe empirically that in the sparse compressive sensing task of Figure 1, the relative sizes of ( D, P ) and q have little impact on the performance of the corresponding AMP iteration.
In Figure 6, we show a replica of this figure with q = 10 and P = 10000 .
Despite a significant difference between the relative sizes of these parameters, the AMP iterations behave largely the same.
C.2 Empirical Results for Vector-AMP Algorithms
We observe that a similar equivalence property as Theorem 4.2 holds for algorithms based on the
VAMP framework Schniter et al.
[2016], Fletcher et al.
[2018], Baker et al.
Previously, state evolution has been proven for such algorithms when their sensing matrices are drawn from a rightorthogonally-invariant ensemble.
While the random MCC ensemble does not satisfy this property, we show in Figure 7 a comparison between empirical VAMP performance and the corresponding SE predictions for dense matrices, which are almost identical.
[Figure]
Sparse
[Figure]
D Structured Convolutions and Non-separable Denoising
Our proof uses a relatively simple version of spatial coupling, leaving avenues for potential generalizations.
Spatially coupled sensing matrices typically consist of a block structured matrix whose blocks are i.i d. Gaussian with different variances, as in (for instance)
Krzakala et al.
[2012], Barbier et al.
As a model, consider ˜ M sp of the following form, with variances κ ∈ R q × q + , {+˜ M sp =   κ 
+}
Aκ 12 A. . .
κ 1 q A 1 q κ 21 A. . .
κ 1 q A 1 q . . .
κ qq A qq   .
As a result of Lemma 4.3, a given MCC matrix M is equivalent to ˜ M corresponding to the case where κ is a convolutional matrix according to Definition 3.1, with filter ω
=
1] ∈ R k .
One avenue to extend our results is to consider general ˜ M where κ is any circulant matrix.
Inverting the permutation lemma, this corresponds to MCC matrices whose convolutional blocks have filters with independent non-isotropic coordinates, as in the following definition, which may be viewed as a simple model for structured convolutional filters.
Definition D.1 (Independent Gaussian Random Convolutions) .
Let ⃗κ =
[ κ 1 , . .
, κ k ] ∈ R k + and let Σ = diag( ⃗κ ) .
Let q ≥ k > 0 be integers.
The Gaussian convolutional ensemble C ( q, k ) contains random circulant matrices C ∈ R q × q whose first rows are given by C {+
+} = Zero-pad q,k [ ω ] where ω ∼ N (0 , Σ) .
This model is a natural extension of our current setting, which is also amenable to proof techniques designed for spatial coupling.
However, because the nonzero coordinates of the sensing matrix are no longer i.i.d., the Bayes-optimal denoising functions corresponding to this problem are non-separable.
So, an equivalence theorem analogous to Theorem 4.2 is not expected to hold – in other words, state evolution in this convolutional model is not expected to reduce to that of a signal model with dense i.i.d.
couplings.
More generally, multilayer AMP iterations with non-separable non-linearities may be written to compute marginals of posterior distributions involving such functions, and will verify
SE equations.
However there will be no direct correspondance with the iteration and SE equations of the fully separable case.