Original: A D ATA -D RIVEN AND D ISTRIBUTED A PPROACH TO S PARSE S IGNAL R EPRESENTATION AND R ECOVERY
Revised: A D ATA -D RIVEN AND D ISTRIBUTED A PPROACH TO S PARSE S IGNAL R EPRESENTATION AND R ECOVERY

Original: Anonymous authors

Original: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: In this paper, we focus on two challenges which offset the promise of sparse signal representation, sensing, and recovery.
Revised: In this paper, we focus on two challenges which offset the promise of sparse signal representation, sensing, and recovery.

Original: First, real-world signals can seldom be described as perfectly sparse vectors in a known basis, and traditionally used random measurement schemes are seldom optimal for sensing them.
Revised: First, real-world signals can seldom be described as perfectly sparse vectors in a known basis, and traditionally used random measurement schemes are seldom optimal for sensing them.

Original: Second, existing signal recovery algorithms are usually not fast enough to make them applicable to real-time problems.
Revised: Second, existing signal recovery algorithms are usually not fast enough to make them applicable to real-time problems.

Original: In this paper, we address these two challenges by presenting a novel framework based on deep learning.
Revised: In this paper, we address these two challenges by presenting a novel framework based on deep learning.

Original: For the ﬁrst challenge, we cast the problem of ﬁnding informative measurements by using a maximum likelihood (ML) formulation and show how we can build a data-driven dimensionality reduction protocol for sensing signals using convolutional architectures.
Revised: For the ﬁrst challenge, we cast the problem of ﬁnding informative measurements by using a maximum likelihood (ML) formulation and show how we can build a data-driven dimensionality reduction protocol for sensing signals using convolutional architectures.

Original: For the second challenge, we discuss and analyze a novel parallelization scheme and show it signiﬁcantly speeds-up the signal recovery process.
Revised: For the second challenge, we discuss and analyze a novel parallelization scheme and show it signiﬁcantly speeds-up the signal recovery process.

Original: We demonstrate the signiﬁcant improvement our method obtains over competing methods through a series of experiments.
Revised: We demonstrate the signiﬁcant improvement our method obtains over competing methods through a series of experiments.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: High-dimensional inverse problems and low-dimensional embeddings play a key role in a wide range of applications in machine learning and signal processing.
Revised: High-dimensional inverse problems and low-dimensional embeddings play a key role in a wide range of applications in machine learning and signal processing.

Original: In inverse problems, the goal is to recover a signal X ∈ R N from a set of measurements Y =
Revised: In inverse problems, the goal is to recover a signal X ∈ R N from a set of measurements Y =

Original: Φ( X ) ∈ R M , where Φ is a linear or non-linear sensing operator.
Revised: Φ( X ) ∈ R M , where Φ is a linear or non-linear sensing operator.

Original: A special case of this problem is compressive sensing (CS) which is a technique for efﬁciently acquiring and reconstructing a sparse signal (Donoho, 2006; Candès et al., 2006; Baraniuk, 2007).
Revised: A special case of this problem is compressive sensing (CS) which is a technique for efﬁciently acquiring and reconstructing a sparse signal (Donoho, 2006; Candès et al., 2006; Baraniuk, 2007).

Original: In CS Φ ∈ R M × N ( M (cid:28) N ) is typically chosen to be a random matrix resulting in a random low-dimensional embedding of signals.
Revised: In CS Φ ∈ R M × N ( M (cid:28) N ) is typically chosen to be a random matrix resulting in a random low-dimensional embedding of signals.

Original: In addition, X is assumed to be sparse in some basis Γ , i.e.,
Revised: In addition, X is assumed to be sparse in some basis Γ , i.e.,

Original: X = Γ S , where (cid:107) S (cid:107) 

Revised: X = Γ S , where (cid:107) S (cid:107)

Original: = K (cid:28) N .
Revised: = K (cid:28) N .

Original: While sparse signal representation and recovery have made signiﬁcant real-world impact in various ﬁelds over the past decade (Siemens, 2017), arguably their promise has not been fully realized.
Revised: While sparse signal representation and recovery have made signiﬁcant real-world impact in various ﬁelds over the past decade (Siemens, 2017), arguably their promise has not been fully realized.

Original: The reasons for this can be boiled down to two major challenges: First, real-world signals are only approximately sparse and hence, random/universal sensing matrices are sub-optimal measurement operators.
Revised: The reasons for this can be boiled down to two major challenges: First, real-world signals are only approximately sparse and hence, random/universal sensing matrices are sub-optimal measurement operators.

Original: Second, many existing recovery algorithms, while provably statistically optimal, are slow to converge.
Revised: Second, many existing recovery algorithms, while provably statistically optimal, are slow to converge.

Original: In this paper, we propose a new framework that simultaneously takes on both these challenges.
Revised: In this paper, we propose a new framework that simultaneously takes on both these challenges.

Original: To tackle the ﬁrst challenge, we formulate the learning of the dimensionality reduction (i.e., signal sensing operator) as a likelihood maximization problem; this problem is related to the Infomax principle (Linsker, 1989) asymptotically.
Revised: To tackle the ﬁrst challenge, we formulate the learning of the dimensionality reduction (i.e., signal sensing operator) as a likelihood maximization problem; this problem is related to the Infomax principle (Linsker, 1989) asymptotically.

Original: We then show that the simultaneous learning of dimensionality reduction and reconstruction function using this formulation gives a lower-bound of the objective functions that needs to be optimized in learning the dimensionality reduction.
Revised: We then show that the simultaneous learning of dimensionality reduction and reconstruction function using this formulation gives a lower-bound of the objective functions that needs to be optimized in learning the dimensionality reduction.

Original: This is similar in spirit to what Vincent et al.
Revised: This is similar in spirit to what Vincent et al.

Original: show for denoising autoencoders in the non-asymptotic setting (Vincent et al., 2010).
Revised: show for denoising autoencoders in the non-asymptotic setting (Vincent et al., 2010).

Original: Furthermore, we show that our framework can learn dimensionality reductions that preserve speciﬁc geometric properties.
Revised: Furthermore, we show that our framework can learn dimensionality reductions that preserve speciﬁc geometric properties.

Original: As an example, we demonstrate how we can construct a data-driven near-isometric low-dimensional embedding that outperforms competing embedding algorithms like
Revised: As an example, we demonstrate how we can construct a data-driven near-isometric low-dimensional embedding that outperforms competing embedding algorithms like

Original: NuMax (Hegde et al., 2015).
Revised: NuMax (Hegde et al., 2015).

Original: Towards tackling the second challenge, we introduce a parallelization (i.e., rearrangement) scheme that signiﬁcantly speeds up the signal sensing and recovery process.
Revised: Towards tackling the second challenge, we introduce a parallelization (i.e., rearrangement) scheme that signiﬁcantly speeds up the signal sensing and recovery process.

Original: We show that our framework can outperform state-of-the-art signal recovery methods such as DAMP
Revised: We show that our framework can outperform state-of-the-art signal recovery methods such as DAMP

Original: We now present a brief overview of prior work on embedding and signal recovery.
Revised: We now present a brief overview of prior work on embedding and signal recovery.

Original: Beyond random matrices, there are other frameworks developed for deterministic construction of linear (or nonlinear) near-isometric embeddings (Hegde et al., 2015; Grant et al., 2013;
Revised: Beyond random matrices, there are other frameworks developed for deterministic construction of linear (or nonlinear) near-isometric embeddings (Hegde et al., 2015; Grant et al., 2013;

Original: Bah et al., 2013; Tenenbaum et al., 2000; Weinberger & Saul, 2006; Broomhead & Kirby, 2001; 2005; Verma, 2013; Shaw & Jebara, 2007).
Revised: Bah et al., 2013; Tenenbaum et al., 2000; Weinberger & Saul, 2006; Broomhead & Kirby, 2001; 2005; Verma, 2013; Shaw & Jebara, 2007).

Original: However, these approaches are either computationally expensive, not generalizable to outof-sample data points, or perform poorly in terms of isometry.
Revised: However, these approaches are either computationally expensive, not generalizable to outof-sample data points, or perform poorly in terms of isometry.

Original: Our framework for low-dimensional embedding shows outstanding performance on all these aspects with real datasets.
Revised: Our framework for low-dimensional embedding shows outstanding performance on all these aspects with real datasets.

Original: Algorithms for recovering signals from undersampled measurements can be categorized based on how they exploit prior knowledge of a signal distribution.
Revised: Algorithms for recovering signals from undersampled measurements can be categorized based on how they exploit prior knowledge of a signal distribution.

Original: They could use hand-designed priors (Candès & Tao, 2005;
Revised: They could use hand-designed priors (Candès & Tao, 2005;

Original: Donoho et al., 2009; Daubechies et al., 2004; Needell & Tropp, 2009), combine hand-designed algorithms with data-driven priors (Metzler et al., 2017; Borgerding & Schniter, 2016; Kamilov &
Revised: Donoho et al., 2009; Daubechies et al., 2004; Needell & Tropp, 2009), combine hand-designed algorithms with data-driven priors (Metzler et al., 2017; Borgerding & Schniter, 2016; Kamilov &

Original: Mansour, 2016; Chang et al., 2017; Gregor & LeCun, 2010), or take a purely data-driven approach (Mousavi et al., 2015; Kulkarni et al., 2016; Mousavi & Baraniuk, 2017; Yao et al., 2017).
Revised: Mansour, 2016; Chang et al., 2017; Gregor & LeCun, 2010), or take a purely data-driven approach (Mousavi et al., 2015; Kulkarni et al., 2016; Mousavi & Baraniuk, 2017; Yao et al., 2017).

Original: As one moves from hand-designed approaches to data-driven approaches, models lose simplicity and generalizability while becoming more complex and more speciﬁcally tailored for a particular class of signals of interest.
Revised: As one moves from hand-designed approaches to data-driven approaches, models lose simplicity and generalizability while becoming more complex and more speciﬁcally tailored for a particular class of signals of interest.

Original: Our framework for sensing and recovering sparse signals can be considered as a variant of a convolutional autoencoder where the encoder is linear and the decoder is nonlinear and speciﬁcally designed for CS application.
Revised: Our framework for sensing and recovering sparse signals can be considered as a variant of a convolutional autoencoder where the encoder is linear and the decoder is nonlinear and speciﬁcally designed for CS application.

Original: In addition, both encoder and decoder contain rearrangement layers which signiﬁcantly speed up the signal sensing and recovery process, as we discuss later.
Revised: In addition, both encoder and decoder contain rearrangement layers which signiﬁcantly speed up the signal sensing and recovery process, as we discuss later.

Original: Convolutional autoencoder has been previously used for image compression (Jiang et al., 2017); however, our work is mainly focused on the CS application rather than image compression.
Revised: Convolutional autoencoder has been previously used for image compression (Jiang et al., 2017); however, our work is mainly focused on the CS application rather than image compression.

Original: In CS, measurements are abstract and linear whereas in the image compression application measurements are a compressed version of the original image and are nonlinear.
Revised: In CS, measurements are abstract and linear whereas in the image compression application measurements are a compressed version of the original image and are nonlinear.

Original: Authors in Jiang et al.
Revised: Authors in Jiang et al.

Original: (2017) have used bicubic interpolation for upscaling images; however, our framework uses a data-driven approach for upscaling measurements.
Revised: (2017) have used bicubic interpolation for upscaling images; however, our framework uses a data-driven approach for upscaling measurements.

Original: Finally, unlike the image compression application, when we deploy our framework for CS and during the test phase, we do not have high-resolution images beforehand.
Revised: Finally, unlike the image compression application, when we deploy our framework for CS and during the test phase, we do not have high-resolution images beforehand.

Original: In addition to image compression, there have been previous works (Shi et al., 2017; Kulkarni et al., 2016) to jointly learn the signal sensing and reconstruction algorithm in CS using convolutional networks.
Revised: In addition to image compression, there have been previous works (Shi et al., 2017; Kulkarni et al., 2016) to jointly learn the signal sensing and reconstruction algorithm in CS using convolutional networks.

Original: However, the problem with these works is that they divide images into small blocks and recover each block separately.
Revised: However, the problem with these works is that they divide images into small blocks and recover each block separately.

Original: This blocky reconstruction approach is unrealistic in applications such as medical imaging (e.g. MRI) where the measurement operator is a Fourier matrix and hence we cannot have blocky reconstruction.
Revised: This blocky reconstruction approach is unrealistic in applications such as medical imaging (e.g. MRI) where the measurement operator is a Fourier matrix and hence we cannot have blocky reconstruction.

Original: Since both papers are designed for block-based recovery whereas our method senses/recovers images without subdivision, we have not compared against them.
Revised: Since both papers are designed for block-based recovery whereas our method senses/recovers images without subdivision, we have not compared against them.

Original: Note that our method could be easily modiﬁed to learn near-optimal frequency bands for medical imaging applications.
Revised: Note that our method could be easily modiﬁed to learn near-optimal frequency bands for medical imaging applications.

Original: In addition, Shi et al.
Revised: In addition, Shi et al.

Original: (2017) and Kulkarni et al.
Revised: (2017) and Kulkarni et al.

Original: (2016) use an extra denoiser (e.g.
Revised: (2016) use an extra denoiser (e.g.

Original: BM3D, DCN) for denoising the ﬁnal reconstruction while our framework does not use any extra denoiser and yet outperforms state-of-the-art results as we show later.
Revised: BM3D, DCN) for denoising the ﬁnal reconstruction while our framework does not use any extra denoiser and yet outperforms state-of-the-art results as we show later.

Original: Beside using convolutional autoencoders, authors in Wu et al.
Revised: Beside using convolutional autoencoders, authors in Wu et al.

Original: (2018) have introduced the sparse recovery autoencoder (SRA).
Revised: (2018) have introduced the sparse recovery autoencoder (SRA).

Original: In SRA, the encoder is a fully-connected layer while in this work, the encoder has a convolutional structure and is basically a circulant matrix.
Revised: In SRA, the encoder is a fully-connected layer while in this work, the encoder has a convolutional structure and is basically a circulant matrix.

Original: For large-scale problems, learning a fully-connected layer (as in the SRA encoder) is signiﬁcantly more challenging than learning convolutional layers (as in our encoder).
Revised: For large-scale problems, learning a fully-connected layer (as in the SRA encoder) is signiﬁcantly more challenging than learning convolutional layers (as in our encoder).

Original: In SRA, the decoder is a T -step projected subgradient.
Revised: In SRA, the decoder is a T -step projected subgradient.

Original: However, in this work, the decoder is several convolutional layers plus a rearranging layer.
Revised: However, in this work, the decoder is several convolutional layers plus a rearranging layer.

Original: It should also be noted that the optimization in SRA is solely over the measurement matrix and T (which is the number of layers in the decoder) scalar values.
Revised: It should also be noted that the optimization in SRA is solely over the measurement matrix and T (which is the number of layers in the decoder) scalar values.

Original: However, here, the optimization is performed over convolution weights and biases that we have across different layers of our network.
Revised: However, here, the optimization is performed over convolution weights and biases that we have across different layers of our network.

Original:  A RCHITECTURE
Revised:  A RCHITECTURE

Original: In this section, we describe our framework for sparse signal representation and recovery and demonstrate how we can learn (near-)optimal projections and speed up signal recovery using parallelization along with convolutional layers.
Revised: In this section, we describe our framework for sparse signal representation and recovery and demonstrate how we can learn (near-)optimal projections and speed up signal recovery using parallelization along with convolutional layers.

Original: We call our framework by DeepSSRR , which stands for Deep Sparse
Revised: We call our framework by DeepSSRR , which stands for Deep Sparse

Original: Signal Representation and Recovery .
Revised: Signal Representation and Recovery .

Original: × × × × =
Revised: × × × × =

Original: = = =
Revised: = = =

Original: Input Signal
Revised: Input Signal

Original: Parallel Convolutions (First Convolutional Layer)
Revised: Parallel Convolutions (First Convolutional Layer)

Original: Convolutional Layers
Revised: Convolutional Layers

Original: Measurements
Revised: Measurements

Original: X
Revised: X

Original: Y =
Revised: Y =

Original: Φ X
Revised: Φ X

Original: or
Revised: or

Original: Y =
Revised: Y =

Original: Φ( X ) (Linear) (Nonlinear)
Revised: Φ( X ) (Linear) (Nonlinear)

Original: Convolutional Layers
Revised: Convolutional Layers

Original: Reconstructed 
Revised: Reconstructed Signal

Original: [Figure]
Revised: [Figure]

Original: 2. S ENSING AND R ECOVERY
Revised: 2. S ENSING AND R ECOVERY

Original: DeepSSRR consists of two parts: A linear dimensionality reduction Φ : R N → R M for taking undersampled measurements and a nonlinear inverse mapping f Λ ( . ) :
Revised: DeepSSRR consists of two parts: A linear dimensionality reduction Φ : R N → R M for taking undersampled measurements and a nonlinear inverse mapping f Λ ( . ) :

Original: R M → R N for recovering signals from their undersampled measurements.
Revised: R M → R N for recovering signals from their undersampled measurements.

Original: We learn both Φ and f Λ ( . ) from training data.
Revised: We learn both Φ and f Λ ( . ) from training data.

Original: DeepSSRR (Figure 1) is based primarily on deep convolutional networks (DCN) as this gives us two advantages: (a) sparse connectivity of neurons, and (b) having shared weights which increases learning speed compared to fully-connected networks.
Revised: DeepSSRR (Figure 1) is based primarily on deep convolutional networks (DCN) as this gives us two advantages: (a) sparse connectivity of neurons, and (b) having shared weights which increases learning speed compared to fully-connected networks.

Original: Therefore, we impose a convolutional network architecture on both Φ and f Λ ( . ) while learning them.
Revised: Therefore, we impose a convolutional network architecture on both Φ and f Λ ( . ) while learning them.

Original: Please note that we assume that measurements are linear; however, it is easy to extend DeepSSRR to adopt nonlinear measurements, i.e., allowing for Φ( . ) to be nonlinear by adding nonlinear units to convolutional layers.
Revised: Please note that we assume that measurements are linear; however, it is easy to extend DeepSSRR to adopt nonlinear measurements, i.e., allowing for Φ( . ) to be nonlinear by adding nonlinear units to convolutional layers.

Original: Given that the intervening layers are linear, one might argue that one convolutional layer (i.e., a single circulant matrix) is enough since we can merge kernel matrices into a single matrix.
Revised: Given that the intervening layers are linear, one might argue that one convolutional layer (i.e., a single circulant matrix) is enough since we can merge kernel matrices into a single matrix.

Original: However, we consider a multi-layer architecture for learning Φ for two reasons.
Revised: However, we consider a multi-layer architecture for learning Φ for two reasons.

Original: First, computationally it is cheaper to have separate and smaller kernels and second, it makes the implementation suitable for adding the aforementioned nonlinearities.
Revised: First, computationally it is cheaper to have separate and smaller kernels and second, it makes the implementation suitable for adding the aforementioned nonlinearities.

Original: We previously mentioned that in order to speed up the sensing and recovery process, we add a parallelization scheme in learning both Φ and f Λ ( . ) that we describe in the following.
Revised: We previously mentioned that in order to speed up the sensing and recovery process, we add a parallelization scheme in learning both Φ and f Λ ( . ) that we describe in the following.

Original: Our original sensing model was Y =
Revised: Our original sensing model was Y =

Original: Φ X where X ∈ R N and Y ∈ R M .
Revised: Φ X where X ∈ R N and Y ∈ R M .

Original: Assume that the undersampling ratio, i.e., MN is equal to 1 r .
Revised: Assume that the undersampling ratio, i.e., MN is equal to 1 r .

Original: The left vector-matrix multiplication in Figure 2(a) denotes a convolution of zero-padded input signal with size N (cid:48)
Revised: The left vector-matrix multiplication in Figure 2(a) denotes a convolution of zero-padded input signal with size N (cid:48)

Original: =
Revised: =

Original: rM (cid:48)
Revised: rM (cid:48)

Original: =
Revised: =

Original: r ( M + q − 1) , ﬁlter size rq , stride (i.e., ﬁlter shift at every step) of size r , and output size of M .
Revised: r ( M + q − 1) , ﬁlter size rq , stride (i.e., ﬁlter shift at every step) of size r , and output size of M .

Original: If we denote the input signal by X (in) and output by
Revised: If we denote the input signal by X (in) and output by

Original: X (out) and ﬁlter by W we can write
Revised: X (out) and ﬁlter by W we can write

Original: [Equation]
Revised: [Equation]

Original: If we concatenate the sub-ﬁlters and sub-signals denoted in orange in the left vector-matrix multiplication of Figure 2(a), we derive a new vector-matrix multiplication shown on the right side of Figure 2(a).
Revised: If we concatenate the sub-ﬁlters and sub-signals denoted in orange in the left vector-matrix multiplication of Figure 2(a), we derive a new vector-matrix multiplication shown on the right side of Figure 2(a).

Original: There the input size is M (cid:48) =
Revised: There the input size is M (cid:48) =

Original: ( M + q − 1) , ﬁlter size is q , stride size is 1, and output size is
Revised: ( M + q − 1) , ﬁlter size is q , stride size is 1, and output size is

Original: M .
Revised: M .

Original: Equation (1) states that the left convolution in Figure 2(a) can be written as the summation of r separate and parallel convolutions shown on the right side.
Revised: Equation (1) states that the left convolution in Figure 2(a) can be written as the summation of r separate and parallel convolutions shown on the right side.

Original: Much like in the sensing part (i.e., learning Φ ), as shown in Figure 2(b), a large strided deconvolution can be chopped into several parallel smaller deconvolutions for the recovery part (i.e., learning f Λ ( . ) ).
Revised: Much like in the sensing part (i.e., learning Φ ), as shown in Figure 2(b), a large strided deconvolution can be chopped into several parallel smaller deconvolutions for the recovery part (i.e., learning f Λ ( . ) ).

Original: Because of these parallelizations, the computational complexity of calculating the outputs of layers in DeepSSRR is O ( M ) which is much less than the one for typical iterative and unrolled algorithms O ( MN )
Revised: Because of these parallelizations, the computational complexity of calculating the outputs of layers in DeepSSRR is O ( M ) which is much less than the one for typical iterative and unrolled algorithms O ( MN )

Original: (e.g. DAMP and LDAMP (Metzler et al., 2016; 2017)) or previous recovery algorithms based on deep learning O 
Revised: (e.g. DAMP and LDAMP (Metzler et al., 2016; 2017)) or previous recovery algorithms based on deep learning O (

Original: × = M × =
Revised: × = M × =

Original: M
Revised: M

Original: M ′ =
Revised: M ′ =

Original: M + q − r r r rq r 1 1 q N ′ =
Revised: M + q − r r r rq r 1 1 q N ′ =

Original: r M ′ =
Revised: r M ′ =

Original: r ( M + q − 1 ) (a) Convolution parallelization × =
Revised: r ( M + q − 1 ) (a) Convolution parallelization × =

Original: M
Revised: M

Original: N ′ =
Revised: N ′ =

Original: N + r ( q − 1 ) =
Revised: N + r ( q − 1 ) =

Original: r ( M + q − 1 ) r r r r q r × × =
Revised: r ( M + q − 1 ) r r r r q r × × =

Original: M ×
Revised: M ×

Original: M ′ =
Revised: M ′ =

Original: M + q −q 

Revised: M + q −q

Original: [Figure]
Revised: [Figure]

Original: [Equation]
Revised: [Equation]

Original: As DeepSSRR architecture is shown in Figure 1, For learning Φ , we ﬁrst divide the input signal (of size N ) into r ( r = NM ) sub-signals (of size M ) such that all the congruent entries (modulo r ) are in the same sub-signal.
Revised: As DeepSSRR architecture is shown in Figure 1, For learning Φ , we ﬁrst divide the input signal (of size N ) into r ( r = NM ) sub-signals (of size M ) such that all the congruent entries (modulo r ) are in the same sub-signal.

Original: Then we run parallel convolutions on r sub-signals and stack the outputs (of size
Revised: Then we run parallel convolutions on r sub-signals and stack the outputs (of size

Original: M ), deriving a tensor of length M and depth r .
Revised: M ), deriving a tensor of length M and depth r .

Original: Through several convolutional layers, we turn this tensor into a vector of size M which is the measurements vector Y and this completes construction of Φ .
Revised: Through several convolutional layers, we turn this tensor into a vector of size M which is the measurements vector Y and this completes construction of Φ .

Original: Similarly and for learning f Λ ( . ) , through several convolutional layers, we turn vector Y into a tensor of length M and depth r .
Revised: Similarly and for learning f Λ ( . ) , through several convolutional layers, we turn vector Y into a tensor of length M and depth r .

Original: We then unstack channels similar to the sub-pixel layer architecture (Shi et al., 2016) and derive the ﬁnal reconstruction
Revised: We then unstack channels similar to the sub-pixel layer architecture (Shi et al., 2016) and derive the ﬁnal reconstruction

Original: (cid:98) X
Revised: (cid:98) X

Original: = f Λ ( Y ) =
Revised: = f Λ ( Y ) =

Original: f Λ (Φ X ) .
Revised: f Λ (Φ X ) .

Original: We use MSE as a loss function and ADAM (Kingma & Ba, 2014) to learn the convolution kernels and biases.
Revised: We use MSE as a loss function and ADAM (Kingma & Ba, 2014) to learn the convolution kernels and biases.

Original: Theoretical Insights.
Revised: Theoretical Insights.

Original: Notice that CS is the problem of recovering X ∈ R N from Y =
Revised: Notice that CS is the problem of recovering X ∈ R N from Y =

Original: Φ X ∈ R M where M (cid:28) N .
Revised: Φ X ∈ R M where M (cid:28) N .

Original: Therefore, an important question is how does one design Φ ?
Revised: Therefore, an important question is how does one design Φ ?

Original: Conventional CS is based on random projections of a signal which means that Φ is a random matrix in conventional CS.
Revised: Conventional CS is based on random projections of a signal which means that Φ is a random matrix in conventional CS.

Original: However, since signals are usually structured, random projections are not optimal for successfully recovering the corresponding signals.
Revised: However, since signals are usually structured, random projections are not optimal for successfully recovering the corresponding signals.

Original: In many applications (e.g. medical imaging), we know a lot about the signals we are acquiring.
Revised: In many applications (e.g. medical imaging), we know a lot about the signals we are acquiring.

Original: Hence, given a large-scale dataset of the same type of signals of interest, we can learn (near-)optimal measurement matrices.
Revised: Hence, given a large-scale dataset of the same type of signals of interest, we can learn (near-)optimal measurement matrices.

Original: As in the usual CS paradigm, if we assume that the measurement matrix Φ is ﬁxed, each Y i ( 1 ≤ i ≤ M ) is a linear combination of X j s ( 1 ≤ j ≤ N ).
Revised: As in the usual CS paradigm, if we assume that the measurement matrix Φ is ﬁxed, each Y i ( 1 ≤ i ≤ M ) is a linear combination of X j s ( 1 ≤ j ≤ N ).

Original: We assume the training set D train = { ( X (1) , Y (1) ) , ( X (2) , Y (2) ) , . .
Revised: We assume the training set D train = { ( X (1) , Y (1) ) , ( X (2) , Y (2) ) , . .

Original: , ( X ( (cid:96) ) , Y ( (cid:96) ) ) } consists of (cid:96) pairs of signals and their corresponding measurements.
Revised: , ( X ( (cid:96) ) , Y ( (cid:96) ) ) } consists of (cid:96) pairs of signals and their corresponding measurements.

Original: Accordingly, we deﬁne the optimal measurement operator (cid:98) Φ as the one which maximizes the probability of training data given the undersampled projections, (cid:98) Φ = arg max Φ (cid:81) (cid:96)i =1 P ( X ( i )
Revised: Accordingly, we deﬁne the optimal measurement operator (cid:98) Φ as the one which maximizes the probability of training data given the undersampled projections, (cid:98) Φ = arg max Φ (cid:81) (cid:96)i =1 P ( X ( i )

Original: | Y ( i ) ) .
Revised: | Y ( i ) ) .

Original: According to the law of large numbers, notice that we can write
Revised: According to the law of large numbers, notice that we can write

Original: [Equation]
Revised: [Equation]

Original: where in (a) I denotes the mutual information, and the equality follows since the Shannon entropy H ( X ) is constant for every Φ .
Revised: where in (a) I denotes the mutual information, and the equality follows since the Shannon entropy H ( X ) is constant for every Φ .

Original: According to (2), in the asymptotic setting, the measurement matrix which maximizes the probability of training data given its measurements, maximizes the mutual information between the input signal and undersampled measurements as well.
Revised: According to (2), in the asymptotic setting, the measurement matrix which maximizes the probability of training data given its measurements, maximizes the mutual information between the input signal and undersampled measurements as well.

Original: Equation (2) is the same as infomax principle ﬁrst introduced in Linsker (1989).
Revised: Equation (2) is the same as infomax principle ﬁrst introduced in Linsker (1989).

Original: Now, suppose that we have a function f ( . ) :
Revised: Now, suppose that we have a function f ( . ) :

Original: R M → R N parametrized by Λ that receives undersampled measurements Y ( i ) (1 ≤ i ≤ (cid:96) ) and reconstructs input signals as (cid:98) X ( i ) =
Revised: R M → R N parametrized by Λ that receives undersampled measurements Y ( i ) (1 ≤ i ≤ (cid:96) ) and reconstructs input signals as (cid:98) X ( i ) =

Original: f Λ ( Y ( i ) )
Revised: f Λ ( Y ( i ) )

Original: (1 ≤ i ≤ (cid:96) ) .
Revised: (1 ≤ i ≤ (cid:96) ) .

Original: We deﬁne the best reconstruction as the one which generates training data with the highest probability.
Revised: We deﬁne the best reconstruction as the one which generates training data with the highest probability.

Original: In other words, we deﬁne
Revised: In other words, we deﬁne

Original: [Equation]
Revised: [Equation]

Original: Therefore, in the asymptotic setting and similar to (2) we can write
Revised: Therefore, in the asymptotic setting and similar to (2) we can write

Original: [Equation]
Revised: [Equation]

Original: In practice and since we do not know the true underlying probability distribution of P ( X | (cid:98) X ) , we maximize a parametric distribution q
Revised: In practice and since we do not know the true underlying probability distribution of P ( X | (cid:98) X ) , we maximize a parametric distribution q

Original: ( X | (cid:98) X ) instead.
Revised: ( X | (cid:98) X ) instead.

Original: In this case, in the asymptotic setting we can write
Revised: In this case, in the asymptotic setting we can write

Original: [Equation]
Revised: [Equation]

Original: Therefore, since Kullback–Leibler divergence is bounded above zero we have
Revised: Therefore, since Kullback–Leibler divergence is bounded above zero we have

Original: [Equation]
Revised: [Equation]

Original: meaning that learning a parametric distribution for reconstructing X from Y is equivalent to maximizing a lower-bound of true conditional entropy and accordingly, mutual information between the input signal X and undersampled measurements Y .
Revised: meaning that learning a parametric distribution for reconstructing X from Y is equivalent to maximizing a lower-bound of true conditional entropy and accordingly, mutual information between the input signal X and undersampled measurements Y .

Original: Hence, although we are not maximizing the mutual information between X and Y , we are maximizing a lower-bound of it through learning Φ and Λ .
Revised: Hence, although we are not maximizing the mutual information between X and Y , we are maximizing a lower-bound of it through learning Φ and Λ .

Original: If we assume X = (cid:98) X + (cid:15) , where (cid:15) and has an isotropic Gaussian distribution, then, since q ( X | (cid:98) X = (cid:98) x ) =
Revised: If we assume X = (cid:98) X + (cid:15) , where (cid:15) and has an isotropic Gaussian distribution, then, since q ( X | (cid:98) X = (cid:98) x ) =

Original: N ( (cid:98) x , λ I ) , the above maximization may be performed by minimizing the mean squared error (MSE).
Revised: N ( (cid:98) x , λ I ) , the above maximization may be performed by minimizing the mean squared error (MSE).

Original: [Table]
Revised: [Table]

Original: 2. A PPLICATIONS OF L OW -D IMENSIONAL E MBEDDING
Revised: 2. A PPLICATIONS OF L OW -D IMENSIONAL E MBEDDING

Original: DeepSSRR is mainly designed for jointly sensing and recovering sparse signals for CS applications.
Revised: DeepSSRR is mainly designed for jointly sensing and recovering sparse signals for CS applications.

Original: However, we can speciﬁcally train the sensing part of DeepSSRR (without using the recovery part) for several important dimensionality reduction tasks.
Revised: However, we can speciﬁcally train the sensing part of DeepSSRR (without using the recovery part) for several important dimensionality reduction tasks.

Original: The sensing part of DeepSSRR (i.e., the encoder or matrix Φ ) is a linear low-dimensional embedding that we can apply it to learn a mapping from a subset of R N to R M ( M < N ) that is a near-isometry, i.e., a mapping that nearly preserves all inter-point distances.
Revised: The sensing part of DeepSSRR (i.e., the encoder or matrix Φ ) is a linear low-dimensional embedding that we can apply it to learn a mapping from a subset of R N to R M ( M < N ) that is a near-isometry, i.e., a mapping that nearly preserves all inter-point distances.

Original: This problem has a range of applications, from approximate nearest neighbor search to the design of sensing matrices for CS.
Revised: This problem has a range of applications, from approximate nearest neighbor search to the design of sensing matrices for CS.

Original: Recall that, for a set Q ⊂ R N and (cid:15) > 0 , the (linear or nonlinear) mapping Φ : Q → R M is an (cid:15) -isometry w.r.t the (cid:96) 2 -norm if for every x and x (cid:48) in Q we have (1 − (cid:15) )
Revised: Recall that, for a set Q ⊂ R N and (cid:15) > 0 , the (linear or nonlinear) mapping Φ : Q → R M is an (cid:15) -isometry w.r.t the (cid:96) 2 -norm if for every x and x (cid:48) in Q we have (1 − (cid:15) )

Original: (cid:107) X [Equation] X (cid:48) (cid:107) 2 ≤ (cid:107) Φ( X ) − Φ( X (cid:48) )
Revised: (cid:107) X − X (cid:48) (cid:107) 2 ≤ (cid:107) Φ( X ) − Φ( X (cid:48) )

Original: (cid:107) 2 ≤ (1 + (cid:15) )
Revised: (cid:107) 2 ≤ (1 + (cid:15) )

Original: (cid:107) X − X (cid:48) (cid:107) 2 .
Revised: (cid:107) X − X (cid:48) (cid:107) 2 .

Original: Algorithm 1 demonstrates the use of the low-dimensional embedding matrix Φ of DeepSSRR to construct a near-isometric embedding.
Revised: Algorithm 1 demonstrates the use of the low-dimensional embedding matrix Φ of DeepSSRR to construct a near-isometric embedding.

Original: We achieve this by penalizing the maximum deviation from isometry in several batches of data that are created by permuting the original training data in every training epoch.
Revised: We achieve this by penalizing the maximum deviation from isometry in several batches of data that are created by permuting the original training data in every training epoch.

Original: In Section 3 we will show how our proposed algorithm works compared to competing methods.
Revised: In Section 3 we will show how our proposed algorithm works compared to competing methods.

Original:  E XPERIMENTAL R ESULTS
Revised:  E XPERIMENTAL R ESULTS

Original: We now illustrate the performance of DeepSSRR against competing methods in several problems.
Revised: We now illustrate the performance of DeepSSRR against competing methods in several problems.

Original: 3. L INEAR L OW -D IMENSIONAL E MBEDDING
Revised: 3. L INEAR L OW -D IMENSIONAL E MBEDDING

Original: We ﬁrst study the quality of the linear embeddings produced by DeepSSRR and its comparison with two other linear algorithms – NuMax (Hegde et al., 2015) and random Gaussian projections.
Revised: We ﬁrst study the quality of the linear embeddings produced by DeepSSRR and its comparison with two other linear algorithms – NuMax (Hegde et al., 2015) and random Gaussian projections.

Original: To show the price of linearity, we also pit these against the nonlinear version of DeepSSRR and a DCN (eight nonlinear convolutional layers + a max-pooling layer).
Revised: To show the price of linearity, we also pit these against the nonlinear version of DeepSSRR and a DCN (eight nonlinear convolutional layers + a max-pooling layer).

Original: We use the grayscale version of CIFARdataset (50,000 training + 10,000 test 32 × 32 images).
Revised: We use the grayscale version of CIFARdataset (50,000 training + 10,000 test 32 × 32 images).

Original: We train DeepSSRR and DCN according to
Revised: We train DeepSSRR and DCN according to

Original: Algorithm 1 by using ﬁlters of size 5 × 5 .
Revised: Algorithm 1 by using ﬁlters of size 5 × 5 .

Original: For DeepSSRR, depending on the size of the embedding we use ﬁve to seven layers to learn Φ in Algorithm 1.
Revised: For DeepSSRR, depending on the size of the embedding we use ﬁve to seven layers to learn Φ in Algorithm 1.

Original: Figure 3(a) shows the size of embedding M as a function of the isometry constant (cid:15) for different methods.
Revised: Figure 3(a) shows the size of embedding M as a function of the isometry constant (cid:15) for different methods.

Original: For the random Gaussian projections we have considered 100 trials and the horizontal error bars represent the deviation from average value.
Revised: For the random Gaussian projections we have considered 100 trials and the horizontal error bars represent the deviation from average value.

Original: As we can see, the nonlinear version of DeepSSRR low-dimensional embedding outperforms almost all the other methods by achieving a given isometry constant with fewer measurements.
Revised: As we can see, the nonlinear version of DeepSSRR low-dimensional embedding outperforms almost all the other methods by achieving a given isometry constant with fewer measurements.

Original: The only exception is when (cid:15) > 0 .
Revised: The only exception is when (cid:15) > 0 .

Original: 6 (i.e., a regime where we are not demanding a good isometry), where the DCN outperforms the nonlinear version of DeepSSRR; though, with more number of parameters.
Revised: 6 (i.e., a regime where we are not demanding a good isometry), where the DCN outperforms the nonlinear version of DeepSSRR; though, with more number of parameters.

Original: Effect of Number of Layers.
Revised: Effect of Number of Layers.

Original: A convolutional layer is equivalent to the product of a circulant matrix and the vectorized input.
Revised: A convolutional layer is equivalent to the product of a circulant matrix and the vectorized input.

Original: The number of nonzero elements in a circulant matrix depends on the size of the convolution ﬁlter.
Revised: The number of nonzero elements in a circulant matrix depends on the size of the convolution ﬁlter.

Original: As the number of such layers grows, so does the number of nonzero elements in the ﬁnal embedding matrix.
Revised: As the number of such layers grows, so does the number of nonzero elements in the ﬁnal embedding matrix.

Original: There are lower bounds (Nelson & Nguyen, 2013) on the number of nonzero elements in a matrix to ensure it is near-isometric.
Revised: There are lower bounds (Nelson & Nguyen, 2013) on the number of nonzero elements in a matrix to ensure it is near-isometric.

Original: Table 1 shows the isometry constant value (cid:15) of DeepSSRR’s low-dimensional embedding with different number of layers and different ﬁlter sizes.
Revised: Table 1 shows the isometry constant value (cid:15) of DeepSSRR’s low-dimensional embedding with different number of layers and different ﬁlter sizes.

Original: As we can see, (cid:15) gets smaller as the ﬁnal embedding matrix has more nonzero elements (more layers, larger ﬁlters).
Revised: As we can see, (cid:15) gets smaller as the ﬁnal embedding matrix has more nonzero elements (more layers, larger ﬁlters).

Original: Approximate Nearest Neighbors.
Revised: Approximate Nearest Neighbors.

Original: Finding the closest k points to a given query datapoint is challenging for high-dimensional datasets.
Revised: Finding the closest k points to a given query datapoint is challenging for high-dimensional datasets.

Original: One solution is to create a near-isometric embedding that
Revised: One solution is to create a near-isometric embedding that

Original: [Figure]
Revised: [Figure]

Original: maps datapoints from R N to R M ( M < N ) and solving the approximate nearest neighbors (ANN) problem in the embedded space.
Revised: maps datapoints from R N to R M ( M < N ) and solving the approximate nearest neighbors (ANN) problem in the embedded space.

Original: Fig.
Revised: Fig.

Original: 3(b) compares the performance of different methods in the ANN problem.
Revised: 3(b) compares the performance of different methods in the ANN problem.

Original: It shows the fraction of k -nearest neighbors that are retained when embedding datapoints in a low-dimensional space.
Revised: It shows the fraction of k -nearest neighbors that are retained when embedding datapoints in a low-dimensional space.

Original: We have considered two separate embedding problems: First M = 65 for random embedding and NuMax and M = 64 for DCN and DeepSSRR’s low-dimensional embedding.
Revised: We have considered two separate embedding problems: First M = 65 for random embedding and NuMax and M = 64 for DCN and DeepSSRR’s low-dimensional embedding.

Original: Second, M = 289 for random embedding and NuMax and M = 256 for DCN and DeepSSRR’s lowdimensional embedding.
Revised: Second, M = 289 for random embedding and NuMax and M = 256 for DCN and DeepSSRR’s lowdimensional embedding.

Original: Since the size of the embedding for DCN and DeepSSRR’s low-dimensional embedding is smaller in both settings, they have a more challenging task to ﬁnd the nearest neighbors.
Revised: Since the size of the embedding for DCN and DeepSSRR’s low-dimensional embedding is smaller in both settings, they have a more challenging task to ﬁnd the nearest neighbors.

Original: As shown in Figure 3(b) DeepSSRR’s low-dimensional embedding outperforms other approaches.
Revised: As shown in Figure 3(b) DeepSSRR’s low-dimensional embedding outperforms other approaches.

Original: 3. S IGNAL R ECOVERY
Revised: 3. S IGNAL R ECOVERY

Original: We divide the discussion of this section into two parts.
Revised: We divide the discussion of this section into two parts.

Original: In the ﬁrst part, we study the performance of
Revised: In the ﬁrst part, we study the performance of

Original: DeepSSRR in the sparse signal recovery problem.
Revised: DeepSSRR in the sparse signal recovery problem.

Original: The discussion of this part along with experimental results showing the effect of learning a sparse representation and parallelization on different criteria (e.g. phase transition, recovery accuracy and speed) are provided in Appendix A .
Revised: The discussion of this part along with experimental results showing the effect of learning a sparse representation and parallelization on different criteria (e.g. phase transition, recovery accuracy and speed) are provided in Appendix A .

Original: In the second part that we provide in the following, we study the performance of DeepSSRR for the compressive image recovery problem.
Revised: In the second part that we provide in the following, we study the performance of DeepSSRR for the compressive image recovery problem.

Original: Compressive Image Recovery.
Revised: Compressive Image Recovery.

Original: In this part, we study the compressive image recovery problem by comparing DeepSSRR with two state-of-the-art algorithms DAMP (Metzler et al., 2016) and
Revised: In this part, we study the compressive image recovery problem by comparing DeepSSRR with two state-of-the-art algorithms DAMP (Metzler et al., 2016) and

Original: LDAMP (Metzler et al., 2017).
Revised: LDAMP (Metzler et al., 2017).

Original: Both DAMP and LDAMP use random Gaussian Φ while DeepSSRR learns a Φ .
Revised: Both DAMP and LDAMP use random Gaussian Φ while DeepSSRR learns a Φ .

Original: Here we run DAMP for 10 iterations and use a BM3D denoiser at every iteration.
Revised: Here we run DAMP for 10 iterations and use a BM3D denoiser at every iteration.

Original: We also run LDAMP for 10 layers and use a 20-layer DCN in every layer as a denoiser.
Revised: We also run LDAMP for 10 layers and use a 20-layer DCN in every layer as a denoiser.

Original: For DeepSSRR, we use 7 layers to learn the Φ and 7 layers to learn the f Λ ( . ) .
Revised: For DeepSSRR, we use 7 layers to learn the Φ and 7 layers to learn the f Λ ( . ) .

Original: DeepSSRR is trained with an initial learning rate of 0.001 that is changed to 0.0001 when the validation error stops decreasing.
Revised: DeepSSRR is trained with an initial learning rate of 0.001 that is changed to 0.0001 when the validation error stops decreasing.

Original: For training, we have used batches of 128 images of size 64 × 64 from ImageNet (Russakovsky et al., 2015).
Revised: For training, we have used batches of 128 images of size 64 × 64 from ImageNet (Russakovsky et al., 2015).

Original: Our training and validation sets include 10,000 and 500 images, respectively.
Revised: Our training and validation sets include 10,000 and 500 images, respectively.

Original: Figure 4(a)
Revised: Figure 4(a)

Original: [Figure]
Revised: [Figure]

Original: shows the reconstructions of the Bridge image ( MN = 0 . 25 ).
Revised: shows the reconstructions of the Bridge image ( MN = 0 . 25 ).

Original: DeepSSRR outperforms both DAMP and
Revised: DeepSSRR outperforms both DAMP and

Original: LDAMP in terms of accuracy (PSNR 1 ).
Revised: LDAMP in terms of accuracy (PSNR 1 ).

Original: In particular, DeepSSRR does a better job at recovering the ﬁne textures inside an image.
Revised: In particular, DeepSSRR does a better job at recovering the ﬁne textures inside an image.

Original: Figure 4(b) presents the reconstructions of the Man image ( MN = 0 . 25 ).
Revised: Figure 4(b) presents the reconstructions of the Man image ( MN = 0 . 25 ).

Original: Although in this example LDAMP outperforms DeepSSRR in terms of PSNR, DeepSSRR does a better job at recovering ﬁne textures.
Revised: Although in this example LDAMP outperforms DeepSSRR in terms of PSNR, DeepSSRR does a better job at recovering ﬁne textures.

Original: In this example LDAMP contains 10 unrolled iterations where each iteration contains a 20-layer DCN.
Revised: In this example LDAMP contains 10 unrolled iterations where each iteration contains a 20-layer DCN.

Original: In other words, LDAMP uses 200 convolutional layers in total.
Revised: In other words, LDAMP uses 200 convolutional layers in total.

Original: On the other hand, DeepSSRR uses only 7 convolutional layers to recover the Man image which is signiﬁcantly smaller compared to LDAMP’s number of layers.
Revised: On the other hand, DeepSSRR uses only 7 convolutional layers to recover the Man image which is signiﬁcantly smaller compared to LDAMP’s number of layers.

Original: Iterative recovery algorithms and their unrolled versions such as DAMP and LDAMP typically involve a matrix vector multiplication in every iteration or layer, and hence their computational complexity is O ( MN ) .
Revised: Iterative recovery algorithms and their unrolled versions such as DAMP and LDAMP typically involve a matrix vector multiplication in every iteration or layer, and hence their computational complexity is O ( MN ) .

Original: In DeepSSRR, the length of feature maps in every convolutional layer is equal to the size of embedding M .
Revised: In DeepSSRR, the length of feature maps in every convolutional layer is equal to the size of embedding M .

Original: Therefore, computing the output of typical middle layers will cost O ( M ) that is signiﬁcantly cheaper than the one for iterative or unrolled methods such as DAMP and LDAMP.
Revised: Therefore, computing the output of typical middle layers will cost O ( M ) that is signiﬁcantly cheaper than the one for iterative or unrolled methods such as DAMP and LDAMP.

Original: Effect of the Number of Layers.
Revised: Effect of the Number of Layers.

Original: Our experiments indicate that having more number of layers does not necessarily result in a better signal recovery performance.
Revised: Our experiments indicate that having more number of layers does not necessarily result in a better signal recovery performance.

Original: This phenomenon is also observed in Dong et al.
Revised: This phenomenon is also observed in Dong et al.

Original: (2016) for the image super-resolution problem.
Revised: (2016) for the image super-resolution problem.

Original: The reason for this problem is the increased non-convexity and non-smoothness of loss function as we add more layers.
Revised: The reason for this problem is the increased non-convexity and non-smoothness of loss function as we add more layers.

Original: One way to mitigate this problem is to add skip connections between layers.
Revised: One way to mitigate this problem is to add skip connections between layers.

Original: As shown in Li et al.
Revised: As shown in Li et al.

Original: (2017), skip connections smooth the loss surface of deep networks and make the optimization problem simpler.
Revised: (2017), skip connections smooth the loss surface of deep networks and make the optimization problem simpler.

Original:  C ONCLUSIONS
Revised:  C ONCLUSIONS

Original: In this paper we introduced DeepSSRR, a framework that can learn both near-optimal sensing schemes, and fast signal recovery procedures.
Revised: In this paper we introduced DeepSSRR, a framework that can learn both near-optimal sensing schemes, and fast signal recovery procedures.

Original: Our ﬁndings set the stage for several directions for future exploration including the incorporation of adversarial training and its comparison with other methods (Bora et al., 2017; Dumoulin et al., 2016; Donahue et al., 2016).
Revised: Our ﬁndings set the stage for several directions for future exploration including the incorporation of adversarial training and its comparison with other methods (Bora et al., 2017; Dumoulin et al., 2016; Donahue et al., 2016).

Original: Furthermore, a major question arising from our work is quantifying the generalizability of a DeepSSRR-learned model based on the richness of training data.
Revised: Furthermore, a major question arising from our work is quantifying the generalizability of a DeepSSRR-learned model based on the richness of training data.

Original: We leave the exploration of this for future research.
Revised: We leave the exploration of this for future research.

Original: A PPENDIX A S PARSE S IGNAL R ECOVERY
Revised: A PPENDIX A S PARSE S IGNAL R ECOVERY

Original: In this section we study the problem of sparse signal recovery by comparing DeepSSRR to another
Revised: In this section we study the problem of sparse signal recovery by comparing DeepSSRR to another

Original: DCN called DeepInverse (Mousavi & Baraniuk, 2017) and to the LASSO (Tibshirani, 1996) (cid:96) 1 -solver implemented using the coordinate descent algorithm of Friedman et al.
Revised: DCN called DeepInverse (Mousavi & Baraniuk, 2017) and to the LASSO (Tibshirani, 1996) (cid:96) 1 -solver implemented using the coordinate descent algorithm of Friedman et al.

Original: We assume that the optimal regularization parameter of the LASSO is given by an oracle in order to obtain its best possible performance.
Revised: We assume that the optimal regularization parameter of the LASSO is given by an oracle in order to obtain its best possible performance.

Original: Also, both training and test sets are wavelet-sparsiﬁed versions of 1D signals of size N = 512 extracted from rows of CIFAR-10 images and contain 100,000 and 20,000 signals, respectively.
Revised: Also, both training and test sets are wavelet-sparsiﬁed versions of 1D signals of size N = 512 extracted from rows of CIFAR-10 images and contain 100,000 and 20,000 signals, respectively.

Original: While DeepSSRR learns how to take undersampled measurements of data through its low-dimensional embedding Φ , DeepInverse uses random undersampling (i.e., a random Φ ).
Revised: While DeepSSRR learns how to take undersampled measurements of data through its low-dimensional embedding Φ , DeepInverse uses random undersampling (i.e., a random Φ ).

Original: DeepSSRR in this section has 3 layers for learning Φ and 3 layers for learning f Λ ( . ) with ﬁlter size 25 × 1 while DeepInverse has ﬁve layers for learning the inverse mapping with ﬁlter size 125 × 1 .
Revised: DeepSSRR in this section has 3 layers for learning Φ and 3 layers for learning f Λ ( . ) with ﬁlter size 25 × 1 while DeepInverse has ﬁve layers for learning the inverse mapping with ﬁlter size 125 × 1 .

Original: Figure 5(a) shows the (cid:96) 1 phase transition plot (Donoho et al., 2009).
Revised: Figure 5(a) shows the (cid:96) 1 phase transition plot (Donoho et al., 2009).

Original: This plot associates each grid point to an ordered pair ( δ, ρ ) ∈ [0 , 1] 2 , where
Revised: This plot associates each grid point to an ordered pair ( δ, ρ ) ∈ [0 , 1] 2 , where

Original: δ =
Revised: δ =

Original: MN denotes the undersampling ratio and ρ =
Revised: MN denotes the undersampling ratio and ρ =

Original: KM denotes the normalized sparsity level.
Revised: KM denotes the normalized sparsity level.

Original: Each grid point ( δ, ρ ) represents the probability of an algorithm’s success in signal recovery for that particular problem conﬁguration.
Revised: Each grid point ( δ, ρ ) represents the probability of an algorithm’s success in signal recovery for that particular problem conﬁguration.

Original: As the name suggests, there is a sharp phase transition between values of ( δ, ρ ) where recovery fails with high probability to when it succeeds with high probability.
Revised: As the name suggests, there is a sharp phase transition between values of ( δ, ρ ) where recovery fails with high probability to when it succeeds with high probability.

Original: In Figure 5(a), the blue curve is the (cid:96) 1 phase transition curve.
Revised: In Figure 5(a), the blue curve is the (cid:96) 1 phase transition curve.

Original: The circular points denote the problem instances on which we study the performance of DeepInverse and the LASSO.
Revised: The circular points denote the problem instances on which we study the performance of DeepInverse and the LASSO.

Original: The square points denote the problem instances on which we have trained and tested DeepSSRR.
Revised: The square points denote the problem instances on which we have trained and tested DeepSSRR.

Original: By design, all these problem instances are on the “failure” side of the (cid:96) 1 phase transition.
Revised: By design, all these problem instances are on the “failure” side of the (cid:96) 1 phase transition.

Original: For DeepSSRR (square points), we have made recovery problems harder by reducing δ and increasing ρ .
Revised: For DeepSSRR (square points), we have made recovery problems harder by reducing δ and increasing ρ .

Original: The arrows between the square points and circular points in Figure 5(a) denote correspondence between problem instances in DeepSSRR and DeepInverse.
Revised: The arrows between the square points and circular points in Figure 5(a) denote correspondence between problem instances in DeepSSRR and DeepInverse.

Original: Tableshows the average normalized MSE (NMSE) for the test signals.
Revised: Tableshows the average normalized MSE (NMSE) for the test signals.

Original: While DeepSSRR recovers the same signals from fewer measurements, it outperforms DeepInverse and the LASSO.
Revised: While DeepSSRR recovers the same signals from fewer measurements, it outperforms DeepInverse and the LASSO.

Original: DeepSSRR outperforms DeepInverse while having signiﬁcantly fewer number of parameters (less than 70,000 vs. approximately 200,000 parameters).
Revised: DeepSSRR outperforms DeepInverse while having signiﬁcantly fewer number of parameters (less than 70,000 vs. approximately 200,000 parameters).

Original: This is mainly due to the fact that DeepSSRR learns Φ instead of using a random Φ as is the case in DeepInverse and conventional CS.
Revised: This is mainly due to the fact that DeepSSRR learns Φ instead of using a random Φ as is the case in DeepInverse and conventional CS.

Original: Fig. 5(b) compares training speed of DeepInverse and DeepSSRR.
Revised: Fig. 5(b) compares training speed of DeepInverse and DeepSSRR.

Original: It shows the MSE of recovering test signals by DeepInverse and DeepSSRR in different training epochs.
Revised: It shows the MSE of recovering test signals by DeepInverse and DeepSSRR in different training epochs.

Original: While training and test sets are the same, the conﬁguration for DeepInverse (and LASSO) is ( δ, ρ ) =
Revised: While training and test sets are the same, the conﬁguration for DeepInverse (and LASSO) is ( δ, ρ ) =

Original: (0 . 7 , 0 . 72) and for DeepSSRR is ( δ, ρ ) =
Revised: (0 . 7 , 0 . 72) and for DeepSSRR is ( δ, ρ ) =

Original: (0 . 5 , 1 . 003) which means we have given DeepSSRR a more challenging problem.
Revised: (0 . 5 , 1 . 003) which means we have given DeepSSRR a more challenging problem.

Original: As shown in Fig.
Revised: As shown in Fig.

Original: 5(b), due to the extra parallelization scheme (i.e., rearrangement layer) convergence is signiﬁcantly faster for DeepSSRR compared to DeepInverse.
Revised: 5(b), due to the extra parallelization scheme (i.e., rearrangement layer) convergence is signiﬁcantly faster for DeepSSRR compared to DeepInverse.

Original: DeepSSRR outperforms the LASSO after only 4 training epochs while DeepInverse takes 138 epochs.
Revised: DeepSSRR outperforms the LASSO after only 4 training epochs while DeepInverse takes 138 epochs.

Original: This fast convergence has two major reasons: First, DeepSSRR has fewer number of parameters to learn.
Revised: This fast convergence has two major reasons: First, DeepSSRR has fewer number of parameters to learn.

Original: Second, DeepSSRR learns adaptive measurements (i.e., low-dimensional embedding) instead of using random measurements (i.e., random embedding).
Revised: Second, DeepSSRR learns adaptive measurements (i.e., low-dimensional embedding) instead of using random measurements (i.e., random embedding).

Original: Fig. 5(c) compares the probability of successful recovery by DeepSSRR and LASSO as measured by 20,000 Monte Carlo test samples.
Revised: Fig. 5(c) compares the probability of successful recovery by DeepSSRR and LASSO as measured by 20,000 Monte Carlo test samples.

Original: For each undersampling ratio δ and for the j -th Monte Carlo sample, we deﬁne the success variable ϕ
Revised: For each undersampling ratio δ and for the j -th Monte Carlo sample, we deﬁne the success variable ϕ

Original: δ,j = I (cid:16) (cid:107)
Revised: δ,j = I (cid:16) (cid:107)

Original: ˆ X ( j ) − X ( j ) (cid:107)(cid:107) X
Revised: ˆ X ( j ) − X ( j ) (cid:107)(cid:107) X

Original: ( j ) (cid:107)≤ 0 .
Revised: ( j ) (cid:107)≤ 0 .

Original: 01 (cid:17) , where X ( j ) is the j -th sample, ˆ X ( j ) is the recovered signal from measurements of j -th sample, and I ( . )
Revised: 01 (cid:17) , where X ( j ) is the j -th sample, ˆ X ( j ) is the recovered signal from measurements of j -th sample, and I ( . )

Original: is the indicator function.
Revised: is the indicator function.

Original: We denote empirical successful recovery probability by P δ
Revised: We denote empirical successful recovery probability by P δ

Original: = 1 q (cid:80) qj =1 ϕ δ,j .
Revised: = 1 q (cid:80) qj =1 ϕ δ,j .

Original: In Fig. 5(c), our test samples are k -sparse where k = 34 , and we have considered three different conﬁgurations:
Revised: In Fig. 5(c), our test samples are k -sparse where k = 34 , and we have considered three different conﬁgurations:

Original: M = 64 , 128 , 256 that correspond to above, on, and below the (cid:96) 1 phase transition, respectively.
Revised: M = 64 , 128 , 256 that correspond to above, on, and below the (cid:96) 1 phase transition, respectively.

Original: As we can see in Fig. 5(c), DeepSSRR signiﬁcantly outperforms LASSO when the problem conﬁguration lies above (failure phase) or on the (cid:96) 1 phase transition and LASSO slightly outperforms when the problem conﬁguration lies below the (cid:96) 1 phase transition (success phase).
Revised: As we can see in Fig. 5(c), DeepSSRR signiﬁcantly outperforms LASSO when the problem conﬁguration lies above (failure phase) or on the (cid:96) 1 phase transition and LASSO slightly outperforms when the problem conﬁguration lies below the (cid:96) 1 phase transition (success phase).

Original: For a setting below the (cid:96) 1 phase transition, we expect (cid:96) 1 minimization to behave the same as (cid:96) 0 minimization.
Revised: For a setting below the (cid:96) 1 phase transition, we expect (cid:96) 1 minimization to behave the same as (cid:96) 0 minimization.

Original: However,
Revised: However,

Original: DeepSSRR should learn a transformation for transforming measurements back to the original signals.
Revised: DeepSSRR should learn a transformation for transforming measurements back to the original signals.

Original: Furthermore, Fig. 5(c) shows the price we pay for using a linear low-dimensional embedding Φ X
Revised: Furthermore, Fig. 5(c) shows the price we pay for using a linear low-dimensional embedding Φ X

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: δ, ρ ) lies on the failure side of (cid:96) 1 phase transition.
Revised: , ρ ) lies on the failure side of (cid:96) 1 phase transition.

Original: instead of a nonlinear one Φ( X ) .
Revised: instead of a nonlinear one Φ( X ) .

Original: The main message of Figure 5(c) is that by using DeepSSRR we can have a signiﬁcantly better phase transition compared to (cid:96) 1 -minimization.
Revised: The main message of Figure 5(c) is that by using DeepSSRR we can have a signiﬁcantly better phase transition compared to (cid:96) 1 -minimization.

Original: Fig. 5(d) shows examples of signal recovery using DeepSSRR and LASSO.
Revised: Fig. 5(d) shows examples of signal recovery using DeepSSRR and LASSO.

Original: The test signal is k -sparse where k
Revised: The test signal is k -sparse where k

Original: = 64 .
Revised: = 64 .

Original: DeepSSRR and LASSO recover this signal from M = 64 and M =measurements, respectively.
Revised: DeepSSRR and LASSO recover this signal from M = 64 and M =measurements, respectively.

Original: DeepSSRR has solved a more challenging recovery problem better than the LASSO with optimal regularization parameter.
Revised: DeepSSRR has solved a more challenging recovery problem better than the LASSO with optimal regularization parameter.

Original: B C OMPRESSIVE I MAGE R ECOVERY
Revised: B C OMPRESSIVE I MAGE R ECOVERY

Original: In this section we study another example of the compressive image recovery problem.
Revised: In this section we study another example of the compressive image recovery problem.

Original: The settings we have used in here is exactly the same as Section 3.2.
Revised: The settings we have used in here is exactly the same as Section 3.2.

Original: Fig.
Revised: Fig.

Original: 6 shows the reconstruction of the mandrill image ( MN = 0 . 25 ).
Revised: 6 shows the reconstruction of the mandrill image ( MN = 0 . 25 ).

Original: Fig.
Revised: Fig.

Original: 6(a) shows the reconstruction of whole face and Fig.
Revised: 6(a) shows the reconstruction of whole face and Fig.

Original: 6(b) shows the reconstruction of nose and cheeks.
Revised: 6(b) shows the reconstruction of nose and cheeks.

Original: As we can see, although LDAMP slightly outperforms our method in Fig. 6(a), our method does a signiﬁcantly better job in recovering the texture of nose and cheeks in Fig. 6(b).
Revised: As we can see, although LDAMP slightly outperforms our method in Fig. 6(a), our method does a signiﬁcantly better job in recovering the texture of nose and cheeks in Fig. 6(b).

Original: Not only our method outperforms LDAMP by 0.9 dB, but also it has a better visual quality and fewer artifacts (e.g. less over-smoothing).
Revised: Not only our method outperforms LDAMP by 0.9 dB, but also it has a better visual quality and fewer artifacts (e.g. less over-smoothing).

Original: [Figure]
Revised: [Figure]

Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: C R UNNING T IME
Revised: C R UNNING T IME

Original: In this section we compare the running time of different algorithms.
Revised: In this section we compare the running time of different algorithms.

Original: We consider the reconstruction of a 512 × 512 image with an undersampling ratio of MN = 0 .
Revised: We consider the reconstruction of a 512 × 512 image with an undersampling ratio of MN = 0 .

Original: Table 3 shows the comparison between different algorithms.
Revised: Table 3 shows the comparison between different algorithms.

Original: We should note that authors in Metzler et al.
Revised: We should note that authors in Metzler et al.

Original: (2017) have used coded diffraction pattern in DAMP and LDAMP which simpliﬁes the computational complexity of vector-matrix multiplications in DAMP and LDAMP to O ( N log( N )) instead of O ( MN ) .
Revised: (2017) have used coded diffraction pattern in DAMP and LDAMP which simpliﬁes the computational complexity of vector-matrix multiplications in DAMP and LDAMP to O ( N log( N )) instead of O ( MN ) .

Original: In addition, we should note that LDAMP uses ﬁlters of size 3 × 3 in its convolutional layers while we use ﬁlters of size 5 × 5 in the convolutional layers of our architecture.
Revised: In addition, we should note that LDAMP uses ﬁlters of size 3 × 3 in its convolutional layers while we use ﬁlters of size 5 × 5 in the convolutional layers of our architecture.

Original: Table 3 shows that our method is almost 4 times faster than the LDAMP method.
Revised: Table 3 shows that our method is almost 4 times faster than the LDAMP method.